comphead opened a new pull request, #7618: URL: https://github.com/apache/datafusion-comet/pull/7618
## Which issue does this PR close? None. This is a draft that tracks DataFusion `main` and is not meant to merge. It builds on #7249, the DataFusion 56.0.0 upgrade. Related: #6410. ## Rationale for this change DataFusion `main` is now `57.0.0-dev`, 128 commits past the `branch-56` cut (`1635ed9`). Building Comet against it surfaces breaks and behavior changes before 57.0.0, the way #7249 does for 56.0.0. #7249 currently conflicts with `main`, so this branch also carries those conflicts resolved. ## What changes are included in this PR? One commit: #7249 at `4c0479b` with `main` at `c09d8e8` merged in, then the pin moved from `56.0.0-rc1` to DataFusion `main` at `6c42917` (2026-11-13). **Conflicts with `main`** - The iceberg-rust pin. `main` moved to apache/iceberg-rust `de0a6ac` and added the `opendal-hdfs-native` feature. #7249 uses `62f8624` from the andygrove/iceberg-rust fork for Arrow 60. apache/iceberg-rust#3257 (Arrow 60) has merged since, so both `iceberg` crates now use its merge commit `6c97391`. That commit contains `de0a6ac` and `46e51d8`, the iceberg-rust main revision the fork branch merged, and the feature list keeps `opendal-hdfs-native`. - `native/native-executors`. `main` moved it to Ballista 55.0.0 with the `rest-api` and `tracing` features. #7249 has a Ballista fork on DataFusion `1635ed9`. Neither is on the DataFusion revision used here, so the workspace stays exactly as on `main`, and its `linux-test-native-executors` job stays off as in #7249. - `native/Cargo.lock` is regenerated from `main`'s lockfile, and `contrib/delta/native/Cargo.lock` from #7249's. **From `56.0.0-rc1` to DataFusion `main`** - Every DataFusion dependency (the workspace, the core dev-dependency and both contrib crates) uses `rev = "6c42917..."`. - apache/datafusion#27337 moved `datafusion-spark` out of DataFusion and into apache/datafusion-spark. Comet now takes it from there at `4cd969d` (2026-11-13). - That repository depends on DataFusion `branch-56`. A `[patch."https://github.com/apache/datafusion"]` section points the 13 DataFusion crates that Comet, the contrib crates and `datafusion-spark` name directly at `6c42917`. All 31 DataFusion crates in the graph resolve to that one source. - Cargo rejects a patch that points at the URL it patches, so the patch uses `https://github.com/apache/arrow-datafusion`, the repository's former name, which GitHub redirects to apache/datafusion. - Remove the section once `datafusion-spark` builds against the DataFusion revision Comet uses. - No source changes beyond #7249's were needed. The other items in DataFusion's 57.0.0 upgrade guide (the Substrait consumer context, `Any` as a supertrait of `ParquetFileReaderFactory`, and `GroupClusteringMode::Full` no longer implying sorted input) do not break the build. Comet reads `group_clustering_mode()` only in test assertions. - apache/datafusion-spark has changes that are not on DataFusion `branch-56`. One of them reaches Comet: apache/datafusion-spark#83 changes how `array_contains` compares floats, and Comet registers `SparkArrayContains`. **Carried over from #7249 unchanged** Arrow and Parquet 60.1.0, object_store 0.14.2, object_store_opendal 0.59.0 and parking_lot 0.13, the source changes for their API changes, the removal of the spill replay pool, the `ClusteredFinalAggregateStream` consumer name, and the ignored nested BINARY UTF-8 tests (#7469, apache/datafusion#27065 is still open). #7249's description has the details. ## How are these changes tested? Run locally on macOS aarch64 at `6c42917`. No Rust or JVM tests were run. - In `native/`, `cargo check --workspace --all-targets`, `cargo clippy --all-targets --workspace -- -D warnings` and `cargo fmt --all -- --check` are clean. - `cargo clippy -p datafusion-comet --all-targets --features contrib-delta,contrib-lance,python-udf,jemalloc -- -D warnings` is clean. - `cargo metadata --locked --offline` succeeds for `native/` and `contrib/delta/native`, so both lockfiles are complete. - CI runs the Comet suites on the default profile. The other Spark profiles, Spark's SQL tests and the Iceberg Spark tests need the `run-all-spark-profiles`, `run-spark-*-tests` and `run-iceberg-tests` labels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
