andygrove opened a new pull request, #7520: URL: https://github.com/apache/datafusion-comet/pull/7520
## Which issue does this PR close? Closes #7240. Part of #7493. The first item there (pointing the `spark-4.2` profile at `iceberg-spark-runtime-4.2_2.13:1.12.0`) is not done, see below. ## Rationale for this change Iceberg 1.12 removed the public two-argument `ManifestFiles.read(ManifestFile, FileIO)`, so every native Iceberg write silently fell back to iceberg-java. No CI job covered 1.12, so nothing caught it. The Spark 4.2 runtime that #7493 wants the profile to use does not exist: Iceberg 1.12.0 has a `spark/v4.2` module in source, but `dev/stage-binaries.sh` at the release tag publishes `SPARK_VERSIONS=3.5,4.0,4.1` only, and `iceberg-spark-runtime-4.2_2.13:1.12.0` does not resolve from Maven Central. The `spark-4.2` profile therefore stays on the 4.0 runtime, and the `!isSpark42Plus` guards and the #4969 test stay as they are, until a release publishes it. ## What changes are included in this PR? One commit per part: 1. `ManifestFiles.read(manifest, io, specsById)`, public on every supported version (1.5.2 through 1.12), replaces the two-argument form in the executor probe and in the manifest decode. The decode passes the table's `PartitionSpec` for the stamped spec id. `DetectionDelegatingFileIO` moves to Java because `HadoopConfigurable.getConf` is abstract on some supported versions and a default method on 1.12, which Scala cannot spell for both. 2. `dev/diffs/iceberg/1.12.0.diff`, ported from the 1.11.0 diff and regenerated against the `apache-iceberg-1.12.0` tag (`v4.1` becomes `v4.2`; the session builder in the compaction benchmark moved to `IcebergCompactionBenchmark`), and an `iceberg_1_12` job running Iceberg 1.12.0 on Spark 4.2.0. It is nightly plus the `run-iceberg-tests` label, because Spark 4.2 support is still experimental. `ci.yml`, `compute-changes.py`, `check-ci-config.py` and the workflows README are updated. 3. Docs: `iceberg-spark-tests.md`, `iceberg-writes.md`, `ci.md`, the Iceberg user guide, and a `spark/pom.xml` comment recording why the 4.2 runtime is not used. ## How are these changes tested? - `CometIcebergRewriteActionSuite`, `CometIcebergWriteActionSuite` and `CometIcebergWriteDetectionSuite` pass on the default Iceberg 1.11 (206 of 206). - The same suites against the Iceberg 1.12.0 jar on Spark 4.1 (a temporary local pom edit): 204 of 206 pass, with no "reflection did not resolve" warnings. The two failures are in Comet's tests, not the fix: one needs `GenericAppenderFactory`, which the 1.12 runtime jar does not bundle, and `fall-back: format-version=4` now needs an absolute table location. `RESTCatalogAdapter.java` also does not compile against 1.12 (`RESTUtil.NAMESPACE_SPLITTER`, `CatalogHandlers.loadTable` and `RESTUtil.decodeNamespace` changed). These need fixing when a profile moves to 1.12. - The new diff applies cleanly to the `apache-iceberg-1.12.0` tag. With Comet installed for Spark 4.2, `TestFilterPushDown` (42 tests) and `TestRewriteDataFilesAction` (195 tests, 8 skipped) pass on Iceberg 1.12.0 and Spark 4.2.0 with no failures. The full `iceberg_1_12` job has not run yet. - `python3 dev/ci/check-ci-config.py` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
