mbutrovich opened a new pull request, #4840: URL: https://github.com/apache/datafusion-comet/pull/4840
Draft while I test. ## Which issue does this PR close? Closes #4381. ## Rationale for this change Iceberg 1.11.0 is the first Iceberg release to support Spark 4.1, and it publishes a real `iceberg-spark-runtime-4.1` artifact. Until now we tested Iceberg only against Spark 3.4/3.5, so the Spark 4.1 path went uncovered. This adds an Iceberg 1.11 / Spark 4.1 target so we can catch compatibility issues in that configuration. Iceberg 1.11 also removed the legacy upstream Comet reader classes (the ones the older diffs had to delete), so the diff needed to enable Comet in Iceberg's own tests shrinks dramatically to just enabling Comet in the test session builders. ## What changes are included in this PR? - `spark/pom.xml`: the spark-4.1 profile now depends on the real `iceberg-spark-runtime-4.1_2.13:1.11.0` instead of reusing the 4.0 runtime, so `CometIcebergNativeSuite` runs against genuine Spark 4.1 Iceberg. - `dev/diffs/iceberg/1.11.0.diff`: new diff generated against the `apache-iceberg-1.11.0` tag. Adds the `comet` version and `build.gradle` dependency and injects the Comet plugin / shuffle manager config into the `spark/v4.1` test bases. No class deletions or reader edits are needed since 1.11 removed the legacy Comet integration. Includes the new `TestVariantShredding` base (variant shredding is new in 1.11 and builds its own session). - CI: `iceberg_1_11` runs on every PR (our only Spark 4.1 Iceberg coverage) and `iceberg_1_10` is demoted to main-only. A new `run-iceberg-tests` PR label enables the main-only Iceberg jobs (1.8/1.9/1.10) on a PR. - Dropped dead/removed configs from all Iceberg test-enable blocks (1.8.1/1.9.1/1.10.0/1.11.0): `spark.comet.schemaEvolution.enabled` and `spark.comet.exec.broadcastExchange.enabled` (both gone from `CometConf`), and `spark.comet.use.lazyMaterialization` (no-op for the Iceberg native scan). - Docs updated for the new version, pairing, and label. ## How are these changes tested? Existing Iceberg Spark test suites (`iceberg-spark`, `iceberg-spark-extensions`, `iceberg-spark-runtime`) run with Comet enabled via the new `iceberg_1_11` CI job against Spark 4.1. `CometIcebergNativeSuite` exercises the native Iceberg scan against the real 4.1 runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
