slachiewicz opened a new pull request, #908: URL: https://github.com/apache/incubator-xtable/pull/908
Part of #902, which folds the skipped Spark 3.5 step into the Spark 4.0 work. This is that step on its own: Spark 3.5 and Delta 3.3 with Scala still on 2.12 and the compiler target still Java 8, so the Delta 2.4 to 3.x jump can be reviewed apart from the Scala 2.13 and Java 17 flip that follows. ## What is the purpose of the pull request Moves the build to `spark.version` 3.5.9 and `delta.version` 3.3.3, and ports the Delta call sites that 3.x moved. Hudi 1.2.0, Iceberg 1.9.2 and Paimon 1.3.1 all publish their 3.5 variants, so no engine version changes. `delta-spark_2.12:3.3.3` is still class-file 52, so Java 8 remains a valid target and CI stays on JDK 11. ## Brief change log - Renamed `io.delta:delta-core_` to `io.delta:delta-spark_`, which Delta did in 3.0, across the four consuming modules, the `xtable-utilities` shade include, the demo dependency scripts and two docs snippets that still named Delta 2.x. - `AddFile` gained `baseRowId`, `defaultRowCommitVersion` and `clusteringProvider`, so its constructor takes 11 arguments; the sync leaves all three unset. - `DeltaLog.getSnapshotAt` now takes a catalog table, and `ensureLogDirectoryExist` is `createLogDirectoriesIfNotExists`. - `reuseMetadataDisabledReconstructsSnapshotPerCommit` no longer asserts that `getChangeLogFiles` goes uncalled: as of Delta 3.x `DeltaLog.getChanges` delegates to it, so the spy cannot tell that call apart from one `DeltaIncrementalChangesState` makes. The snapshot-per-commit assertion it shares the test with is unchanged. - `ITDeltaDeleteVectorConvert` reloads its table handle after enabling deletion vectors. Delta 3.x exposes the `row_index` metadata column only when deletion vectors are readable on the snapshot the scan was planned against (`DeltaParquetFileFormat.scala:255-262`), and the handle resolved at construction still carries the pre-`ALTER TABLE` protocol. ## Verify this pull request Built and run locally on JDK 11: 494 unit tests pass. Two integration failures are outstanding, which is why this is a draft: - `ITDeltaDeleteVectorConvert` reaches `validateDeletedRecordCount` and finds one file carrying a deletion vector where the test expects none, because Delta 3.x persists deletion vectors for `MERGE`. The expectations in that test are cumulative, so they need re-deriving against the new behaviour, or the merge DV persistence needs turning off for that phase. Worth a maintainer opinion on which. - `ITConversionController#testColumnMappingEnabledDeltaToIceberg` fails in Iceberg's Parquet reader: the map key path it asks for carries a different column-mapping physical name than the data file holds. That points at the Delta to Iceberg schema conversion picking up a physical name from a different schema version, not at a version pin. Not yet diagnosed. Nine further errors seen locally, all `NoSuchElementException: No value present in Option` from `scheduleCompaction` on MERGE_ON_READ tables in `ITHudiConversionSource` and `ITConversionController`, reproduce on unmodified `main` and are green in CI, so they are an artefact of this machine rather than of this change. *This change was created with AI assistance.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
