j1wonpark opened a new pull request, #4378: URL: https://github.com/apache/amoro/pull/4378
## Why are the changes needed? Close #4377. This follows up on the Iceberg upgrade discussion: let the Iceberg version used by the self-optimizing / maintenance path evolve independently from the mixed-format connectors, so that a maintenance-side upgrade no longer forces engine drops on connector users. It keeps every module and class where it is. AMS, the optimizers and `dist` select Iceberg 1.10.2; `amoro-format-iceberg`, `amoro-mixed-hive` and all engine connectors stay on 1.8.1. The code is ready for review. I am opening it as a draft because one point needs a community decision (Hadoop 2 support for AMS, see below); once that is settled I will mark it ready. ## Brief change log - Root `pom.xml`: new `maintenance-iceberg.version` (1.10.2) and `maintenance-parquet.version` (1.16.0) properties. `iceberg.version` stays 1.8.1. - Root `pom.xml`: the connector-side Parquet version is now explicit as well. `parquet-avro.version` becomes `parquet.version` (still 1.15.2) and the root manages all six Parquet artifacts with it, so the two sides read as `iceberg.version` / `parquet.version` vs. `maintenance-iceberg.version` / `maintenance-parquet.version`. On the connector side `parquet-jackson` keeps its separate 1.13.1 pin; on the maintenance side it is managed at 1.16.0 like the rest of the family. Verified with `dependency:list` before and after: shipped artifacts (connector runtime jars, Trino) resolve exactly as before; the only change is that `amoro-format-paimon` and `amoro-mixed-spark-3-common` previously resolved a mix of 1.13.1 / 1.15.0 / 1.15.2 and now resolve 1.15.2 consistently. - `amoro-ams`, `amoro-optimizer`, `dist`: set `iceberg.version` to the maintenance version and manage the Iceberg artifacts the root does not pin (`iceberg-api`, `iceberg-bundled-guava`, Spark binding and extensions) plus the Parquet family. The Parquet alignment is required: Iceberg 1.10.x needs a newer Parquet than the one resolved through 1.8.1, and this shows up as `NoSuchMethodError` at runtime, not at build time. - Two Parquet reader calls that Iceberg removed in 1.9 (`ParquetValueReaders.StructReader(types, readers)` and `ParquetValueReader.setPageSource(pages, rowPosition)`) are switched to the form that exists in both 1.8.1 and 1.10.x, in `amoro-mixed-hive` and the Spark 3.4 / 3.5 readers that AMS loads for its local terminal. The shared modules are still compiled against 1.8.1; the same bytecode runs on both versions. - New `TestMaintenanceIcebergCompatibility` in `amoro-ams` runs the existing `TestTaskReader`, `TestTaskWriter`, `TestHiveTaskReader` and `TestHiveTaskWriter` fixtures on the AMS classpath, i.e. against Iceberg 1.10.2. Without the reader change above this fails with the `NoSuchMethodError`; with it, all pass. - CI: the Spark 3.3 matrix entry and the Hadoop 2 job now build connectors and optimizers only (`-pl '!amoro-ams,!dist'`), see open questions below. The Hadoop 2 docker image builds `dist` with `-Phadoop2,spark-3.5`. - `dev/deps`: dependency lists regenerated with `dev/dependencies.sh --replace`. Besides Iceberg 1.10.2 and Parquet 1.16.0, the maintenance classpath picks up `parquet-variant`, `jts-core`, and transitive bumps of aircompressor (0.27 -> 2.0.3), avro (1.12.0 -> 1.12.1) and httpclient5 / httpcore5 (5.4.1 / 5.3.1 -> 5.5 / 5.3.4). The Hadoop 2 list is now `dependencies-hadoop-2-spark-3.5` because the Hadoop 2 dist build uses the Spark 3.5 profile. - `LICENSE-binary`: `jts-core` is a new transitive dependency of Parquet 1.16 (EDL 1.0). The other new or bumped artifacts are already covered. ## How was this patch tested? - Same shared bytecode (`amoro-common`, `amoro-format-iceberg`, `amoro-mixed-hive`, compiled with JDK 11 against Iceberg 1.8.1) run with the existing I/O tests on both an Iceberg 1.8.1 and an Iceberg 1.10.2 classpath: 85 passed / 15 assumption-skipped / 0 failed on each side. Class hashes verified identical between the two runs. - On the AMS classpath (Iceberg 1.10.2, JDK 11): `TestKeyedContinuousOptimizing` and `TestUnKeyedContinuousOptimizing` (12/12, native Iceberg, mixed Iceberg and mixed Hive), `TestSnapshotExpire*` and `TestOrphanFileClean*` (116 passed / 28 assumption-skipped / 0 failed). - Local Spark 3.5 with the mixed, Iceberg and Paimon extensions registered together (the terminal's combination): create native Iceberg table, insert, aggregate. - Packaged artifacts inspected with a small probe on JDK 11: `amoro-ams-dependency/lib`, Flink and Spark optimizer jars load Iceberg 1.10.2 (no duplicate `org/apache/iceberg` classes); mixed Flink 1.18 / 1.19 / 1.20 and Spark 3.5 runtime jars load relocated Iceberg 1.8.1. - Reactor builds: default (Spark 3.5), `-Pspark-3.4`, `-Pspark-3.3` (connectors and optimizers only), `-Phadoop2` (connectors and optimizers only). Not covered: end-to-end run with an external optimizer process, real terminal API / mixed catalog SQL, Trino, full dist tarball run. ## Decision needed **Hadoop 2 support for AMS.** Parquet 1.16 calls `FileSystem.openFile(Path)`, which does not exist in Hadoop 2.10, so with `-Phadoop2` the AMS I/O tests fail with `NoSuchMethodError`. There is no workaround on our side short of keeping AMS on the old Parquet, which defeats the upgrade. My proposal: AMS requires Hadoop 3 from the next release; the mixed-format connectors keep their Hadoop 2 build. This PR already implements that on the CI side (the Hadoop 2 job builds connectors and optimizers only). What still needs to follow if we agree: drop the Hadoop 2 dist / docker image (it currently still compiles with `-Phadoop2,spark-3.5` but AMS cannot read Parquet in it) and note the change in the release notes. If anyone still runs AMS on Hadoop 2, please speak up. ## Notes and limitations 1. **Spark 3.3.** `iceberg-spark-3.3` was last published for Iceberg 1.8.1, so AMS can no longer be built with `-Pspark-3.3`. The default AMS build is Spark 3.5 and is unaffected; the Spark 3.3 mixed-format connector is still built and published, just not in the same reactor as AMS. 2. **AMS and optimizers must share one Iceberg version.** Task descriptors cross the AMS / optimizer boundary through Java serialization, and `DataFile` serialization is not compatible between 1.8.1 and 1.9+ (`serialVersionUID`). This PR sets both from the same property. 3. **Compile boundary.** The shared modules are compiled against 1.8.1, so code that needs 1.9+ API (row lineage, `MetadataColumns.ROW_ID`, ...) cannot live in `amoro-format-iceberg` as it is today. New V3-specific maintenance code needs a location that compiles against the maintenance version. This PR does not decide that; it only shows that the version split itself does not require moving modules first. 4. Parquet 1.16.0 is the version Iceberg 1.10.2 declares. I have not tried to find the minimum Parquet version that works. ## Documentation - Does this pull request introduce a new feature? no - README build section updated: Hadoop 2 and Spark 3.3 connector build commands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
