andygrove commented on PR #2638: URL: https://github.com/apache/datafusion-ballista/pull/2638#issuecomment-6881892254
> > Heads up that [#2718](https://github.com/apache/datafusion-ballista/pull/2718) just landed on `main`, pinning DataFusion to `56.0.0-rc1`, and the release will be 56.0.0. Could you merge the latest `apache/main` into this branch? I tried it locally and it merges cleanly. The crate still builds against its current Ballista pin, so the separate workspace is doing its job here. > > Moving the Ballista pin to current `main` is the harder part. iceberg-datafusion is still on DataFusion 55, so the build ends up with two copies of DataFusion and fails to compile. [apache/datafusion-iceberg#99](https://github.com/apache/datafusion-iceberg/pull/99) adds DataFusion 56 support, but it pins DataFusion commit `1635ed94` rather than the `56.0.0-rc1` tag Ballista uses. Those would still be two copies. > > To see what the bump would take, I checked out [apache/datafusion-iceberg#99](https://github.com/apache/datafusion-iceberg/pull/99), pointed it at the `56.0.0-rc1` tag, and used it as a path dependency. With that, the crate needed only these changes: > > ``` > > * `src/catalog.rs`: `TableProvider::scan` now takes `projection: Option<&[usize]>` instead of `Option<&Vec<usize>>`. > > > > * `Cargo.toml`: bump `arrow-schema` and `arrow-flight` from `59.2.0` to `60.0.0`. > > > > * `Cargo.toml`: Arrow 60 renamed the `flight-sql-experimental` feature to `flight-sql`. > > > > * `Cargo.toml`: move `iceberg`, `iceberg-storage-opendal` and `iceberg-catalog-rest` to `73e772c7dbc18f4a3f9f92187b6c56cafbc2f145` to match [chore: prepare for DataFusion 56 datafusion-iceberg#99](https://github.com/apache/datafusion-iceberg/pull/99). > > > > * `Cargo.toml`: point `datafusion` and `datafusion-proto` at `{ git = "https://github.com/apache/datafusion";, tag = "56.0.0-rc1" }`, and move the Ballista crates to a `main` commit that includes [chore: pin DataFusion to 56.0.0-rc1 ahead of DataFusion 56 #2718](https://github.com/apache/datafusion-ballista/pull/2718). > > ``` > > > > > > > > > > > > > > > > > > > > > > > > The codec, standalone, Flight SQL and doc tests all passed. I didn't run the testcontainers tests. > > I'm happy to merge this as experimental on the current pin and do the DataFusion 56 bump as a follow-up once [apache/datafusion-iceberg#99](https://github.com/apache/datafusion-iceberg/pull/99) lands. Would you be up for asking there whether it could pin the `56.0.0-rc1` tag? Then the two would line up and we could make the bump before the release is tagged. > > I just noticed this comment 👁️ Please ignore tha lst part about asking datafusion-iceberg to pin to 56.0.0-rc1. I didn't ask for that part to be posted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
