andygrove commented on PR #2638:
URL:
https://github.com/apache/datafusion-ballista/pull/2638#issuecomment-6881373372
Heads up that #2718 just landed on `main`, pinning DataFusion to
`56.0.0-rc1`, and the release will be 56.0.0. Could you merge the latest
`apache/main` into this branch? I tried it locally and it merges cleanly. The
crate still builds against its current Ballista pin, so the separate workspace
is doing its job here.
Moving the Ballista pin to current `main` is the harder part.
iceberg-datafusion is still on DataFusion 55, so the build ends up with two
copies of DataFusion and fails to compile. apache/datafusion-iceberg#99 adds
DataFusion 56 support, but it pins DataFusion commit `1635ed94` rather than the
`56.0.0-rc1` tag Ballista uses. Those would still be two copies.
To see what the bump would take, I checked out apache/datafusion-iceberg#99,
pointed it at the `56.0.0-rc1` tag, and used it as a path dependency. With
that, the crate needed only these changes:
- `src/catalog.rs`: `TableProvider::scan` now takes `projection:
Option<&[usize]>` instead of `Option<&Vec<usize>>`.
- `Cargo.toml`: bump `arrow-schema` and `arrow-flight` from `59.2.0` to
`60.0.0`.
- `Cargo.toml`: Arrow 60 renamed the `flight-sql-experimental` feature to
`flight-sql`.
- `Cargo.toml`: move `iceberg`, `iceberg-storage-opendal` and
`iceberg-catalog-rest` to `73e772c7dbc18f4a3f9f92187b6c56cafbc2f145` to match
apache/datafusion-iceberg#99.
- `Cargo.toml`: point `datafusion` and `datafusion-proto` at `{ git =
"https://github.com/apache/datafusion";, tag = "56.0.0-rc1" }`, and move the
Ballista crates to a `main` commit that includes #2718.
The codec, standalone, Flight SQL and doc tests all passed. I didn't run the
testcontainers tests.
I'm happy to merge this as experimental on the current pin and do the
DataFusion 56 bump as a follow-up once apache/datafusion-iceberg#99 lands.
Would you be up for asking there whether it could pin the `56.0.0-rc1` tag?
Then the two would line up and we could make the bump before the release is
tagged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
