xxubai opened a new pull request, #4109:
URL: https://github.com/apache/amoro/pull/4109
## Why are the changes needed?
Amoro is upgrading Iceberg, but different engines (Flink/Spark/Hive) have
different compatibility windows for Iceberg artifacts and APIs.
To avoid a one-version-for-all conflict, this PR introduces per-module
version overrides and engine-specific compatibility modules, so newer engines
can use newer Iceberg while older engines keep compatible versions.
Close #xxx.
## Brief change log
- Introduce per-module `iceberg.version` override strategy in mixed-format
modules.
- Upgrade Iceberg for newer engines while keeping compatibility for older
engines.
- Add Flink 1.17 dedicated compatibility modules:
- `amoro-mixed-flink-common-1.17`
- `amoro-mixed-flink-common-iceberg-bridge-1.17`
- Adapt Flink-related Iceberg bridge code for new Iceberg APIs:
- parquet/flink reader path adjustments
- `MiniClusterResource` test API compatibility updates
- Adapt mixed-hive code to new Iceberg parquet API changes.
- Adapt Spark parquet readers (`v3.3/v3.4/v3.5`) for Iceberg API updates.
- Update related tests (including `IcebergRewriteExecutorTest`) and module
wiring in POMs.
- **This change set was generated with Codex assistance.**
## How was this patch tested?
- [x] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
Local verification (examples):
- `mvn -pl amoro-format-mixed/amoro-mixed-hive -am -DskipTests compile`
- `mvn -pl
amoro-format-mixed/amoro-mixed-flink/amoro-mixed-flink-common-iceberg-bridge
-am -DskipTests compile`
- `mvn -pl
amoro-format-mixed/amoro-mixed-flink/v1.17/amoro-mixed-flink-1.17,amoro-format-mixed/amoro-mixed-flink/v1.18/amoro-mixed-flink-1.18
-am -DskipTests compile`
- project package validation (including dist packaging) has been executed in
local environment
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
