peterphitran opened a new pull request, #39559:
URL: https://github.com/apache/beam/pull/39559
## What Changes
Upgrades Apache Iceberg from 1.10.0 to 1.11.0, including its core,
file-format, and cloud-storage components (iceberg-core/api/parquet/orc/data
and the GCP/AWS/Azure runtime modules).
This PR is based on #39064, which raised IcebergIO’s Java requirement to
Java 17. Tracks #38925.
## What Does Not Change
Iceberg 1.11.0 would normally bring in newer Parquet libraries (1.17.1).
This PR keeps Beam on its existing Parquet version (1.16.0) so the Iceberg
upgrade does not change Parquet behavior. We can upgrade Parquet separately
after testing it across Beam.
The existing Iceberg read/write APIs used by Beam are still available in
Iceberg 1.11.0. They are marked as deprecated, but have not been removed, so no
source-code changes are needed in this PR. A follow-up PR will migrate Beam to
Iceberg’s newer API.
## What was tested
- Ran the full IcebergIO unit-test build: 309 tests passed with no failures
or errors.
- Confirmed Beam resolves Iceberg 1.11.0 and keeps parquet-avro and
parquet-hadoop at 1.16.0.
- Built the Beam modules that depend on IcebergIO:
- Beam SQL Iceberg extension
- Iceberg examples
- Java expansion-service jar
- Ran Iceberg’s Hadoop-version test matrix against the supported Hadoop
versions (3.3.6 and 3.4.1).
- Confirmed the Java 17 build path works, including compilation from a Java
11 host that forks a Java 17 compiler.
## Not run locally
Cloud-backed Iceberg integration tests require Google Cloud credentials.
Full linkage-checker output could not be completed locally because of
build-infrastructure dependency resolution issues.
## Reviewer decision requested (Dependency Questions)
1. **ORC artifact selection:** Iceberg uses the `orc-core:nohive` artifact.
Beam currently declares unclassified `orc-core`, which can conflict with
Iceberg’s ORC classes. The focused Beam ORC read test passes with `nohive` on
both ORC 1.9.6 and 1.9.8. Should this PR include the small dependency
correction, or should it remain strictly scoped to the Iceberg bump and track
that correction separately?
2. **Avro version:** Beam deliberately forces Avro 1.12.0 even though
Iceberg 1.11.0 requests 1.12.1. The current test suite passes with Beam’s
resolved 1.12.0, so this PR does not change Avro behavior. However, the issue
that motivated avoiding 1.12.1 looks to have been resolved
[AVRO-4209](https://issues.apache.org/jira/browse/AVRO-4209). Should we keep
the existing pin in this PR and open a separate Beam-wide Avro 1.12.2
upgrade/validation PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
