This is an automated email from the ASF dual-hosted git repository.
ahmedabu98 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 58bac320ebd [IcebergIO] Raise Java 17 floor for IcebergIO's Java 11
dependents (#39064)
58bac320ebd is described below
commit 58bac320ebd2601e6b66261b5e40a72d59161cff
Author: Peter Tran <[email protected]>
AuthorDate: Wed Jul 29 16:12:12 2026 -0500
[IcebergIO] Raise Java 17 floor for IcebergIO's Java 11 dependents (#39064)
* [IcebergIO] Bump Java floor to 17 (prep for Iceberg 1.11.0)
Iceberg 1.11.0 is published as Java 17 bytecode (class file version 61);
Gradle module metadata declares "org.gradle.jvm.version": 17. The current
Java 11 floor on sdks/java/io/iceberg causes Iceberg 1.11.0 artifacts
to fail resolution. Raise the floor independently so the version bump
is a clean diff.
Iceberg 1.10.0 still resolves under Java 17 (10.0 only requires Java 11+).
Tracks #38925.
* [IcebergIO] Run IcebergIO CI workflows on Java 17
Iceberg 1.11.0 dropped Java 11 support and this branch raised the iceberg
module floor to Java 17 (requireJavaVersion VERSION_17). The IO_Iceberg_*
workflows call setup-environment-action without a java-version, which
defaults to Java 11, so the module would fail to build or be disabled on
CI. Pin these workflows to Java 17, matching the Delta and Debezium IO
workflows.
For #38925
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* [IcebergIO] Raise Java 17 floor for IcebergIO's Java 11 dependents
Bumping the iceberg module to Java 17 (Iceberg 1.11.0 dropped Java 11)
breaks the Java 11 modules that depend on it, because Gradle's
JVM-version variant resolution refuses to let a Java 11 consumer depend
on a Java 17 library. Raise requireJavaVersion 11 -> 17 in:
- sdks/java/extensions/sql/iceberg
- examples/java/iceberg
The Java IO expansion service (sdks/java/io/expansion-service) also
depends on IcebergIO but was already raised to Java 17 on master in
For #38925
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* [IcebergIO] Fix Checker skipUses regex dropped on forked Java 17 compile
Raising the Java 17 floor routes IcebergIO through a forked compile: on an
older host JDK (e.g. Java 11 CI) the module is compiled by a separate Java 17
javac launched via java17Home, and Gradle passes its arguments through an
@argfile. The backslash-escaped '\.' in Beam's Checker Framework -AskipUses and
-AskipDefs regexes does not survive that @argfile round-trip, so
'^org\.slf4j\.Logger.*' becomes a literal-backslash regex that matches nothing
and the org.slf4j.Logger nullness supp [...]
Use the backslash-free character class '[.]' (semantically identical to
'\.') so the regexes survive the @argfile round-trip. This fixes the
suppression for every module that forks to a newer JDK, with no behavior change
on the in-process compile path.
* [IcebergIO] Run xlang wrapper validation expansion service on Java 17
The cross-language wrapper validation launches the io expansion-service jar
from Python via JAVA_HOME. That jar bundles IcebergIO, now Java 17 bytecode, so
launching it on the default Java 11 fails with UnsupportedClassVersionError.
createCrossLanguageUsingJavaExpansionTask was missing the JAVA_HOME
redirect that #38974 added to the other cross-language task factories. Add it,
driven by -PtestJavaVersion (resolving java\Home), and pass
-PtestJavaVersion=17 in the Xlang_Generated_Transforms precommit so the
expansion service runs on a JDK that can load the bundled Java 17 IOs.
* Trigger Python Xlang IO postcommits
* [IcebergIO] Run Python Xlang IO postcommits on Java 17
* [IcebergIO] Pin Java 17 on IO expansion service test suites
* [IcebergIO] Address Java 17 review comments
* [IcebergIO] Raise Iceberg integration tests to Java 17
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
---
.../beam_PostCommit_Python_Xlang_IO_Dataflow.json | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.json | 2 +-
.../workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml | 2 +-
.../workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Xlang_IO_Direct.yml | 2 +-
CHANGES.md | 4 ++++
.../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 12 +++++++++---
examples/java/iceberg/build.gradle | 4 ++--
it/iceberg/build.gradle | 4 ++--
sdks/java/extensions/sql/iceberg/build.gradle | 4 ++--
sdks/java/io/iceberg/build.gradle | 4 ++--
11 files changed, 26 insertions(+), 16 deletions(-)
diff --git
a/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Dataflow.json
b/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Dataflow.json
index e3d6056a5de..b2683333323 100644
--- a/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Dataflow.json
+++ b/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Dataflow.json
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run",
- "modification": 1
+ "modification": 2
}
diff --git a/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Direct.json
b/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Direct.json
index 455144f02a3..d6a91b7e2e8 100644
--- a/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Direct.json
+++ b/.github/trigger_files/beam_PostCommit_Python_Xlang_IO_Direct.json
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run",
- "modification": 6
+ "modification": 7
}
diff --git a/.github/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml
b/.github/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml
index 7a7dcb72f0d..5adb09f2462 100644
--- a/.github/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml
+++ b/.github/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml
@@ -75,7 +75,7 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
- java-version: default
+ java-version: '17'
python-version: default
- name: Set k8s access
uses: ./.github/actions/setup-k8s-access
diff --git a/.github/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml
b/.github/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml
index b848c55d767..20d93902328 100644
--- a/.github/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml
+++ b/.github/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml
@@ -75,7 +75,7 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
- java-version: default
+ java-version: '17'
python-version: |
3.10
3.14
diff --git a/.github/workflows/beam_PostCommit_Python_Xlang_IO_Direct.yml
b/.github/workflows/beam_PostCommit_Python_Xlang_IO_Direct.yml
index b254774312c..0767796b1ee 100644
--- a/.github/workflows/beam_PostCommit_Python_Xlang_IO_Direct.yml
+++ b/.github/workflows/beam_PostCommit_Python_Xlang_IO_Direct.yml
@@ -75,7 +75,7 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
- java-version: default
+ java-version: '17'
python-version: |
3.10
3.14
diff --git a/CHANGES.md b/CHANGES.md
index a1fdede9449..c5da08bd1a5 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -79,6 +79,10 @@
* (Python) Removed `google-perftools` from the SDK container images. Users who
wish to use `--profiler_agent=tcmalloc` should install google-perftools APT
package in their custom container images separately
([#39323](https://github.com/apache/beam/issues/39323)).
* `DoFn.process` returning a `str`, `bytes`, or `dict` (instead of an iterable
wrapping one) now raises a `TypeError` rather than silently iterating
per-character/byte/key (Python)
([#18712](https://github.com/apache/beam/issues/18712)).
* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including
runner state mappings and Dataflow update handling
([#39020](https://github.com/apache/beam/issues/39020)).
+* (Python) Typehints of dataclass fields are honored during type inferences.
To restore the behavior of fallback-to-any,
+ use pipeline option `--exclude_infer_dataclass_field_type`
([#38797](https://github.com/apache/beam/issues/38797)).
+ However fixing forward is recommended.
+* (Java) IcebergIO now requires Java 17 at runtime. This raises the floor in
preparation for the Iceberg 1.11.0 upgrade
([#38925](https://github.com/apache/beam/issues/38925)).
## Deprecations
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 0bf3dc11e2d..8e536f0dbe3 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -1257,20 +1257,26 @@ class BeamModulePlugin implements Plugin<Project> {
maxHeapSize = '2g'
}
+ // NOTE: Use the character class "[.]" instead of an escaped "\\." to
match a literal dot in
+ // these Checker Framework -AskipDefs/-AskipUses regexes. When a module
is compiled on an older
+ // host JDK and forked to a newer JDK via javaXXHome (e.g. iceberg's
requireJavaVersion 17 on a
+ // Java 11 CI host), Gradle passes the javac arguments through an
@argfile. Backslash escapes do
+ // not survive that round-trip intact, so "\\." becomes a
literal-backslash regex that matches
+ // nothing and the suppression is silently dropped. "[.]" is
backslash-free and survives.
List<String> skipDefRegexes = []
skipDefRegexes << "AutoValue_.*"
skipDefRegexes << "AutoBuilder_.*"
skipDefRegexes << "AutoOneOf_.*"
- skipDefRegexes << ".*\\.jmh_generated\\..*"
+ skipDefRegexes << ".*[.]jmh_generated[.].*"
skipDefRegexes += configuration.generatedClassPatterns
skipDefRegexes += configuration.classesTriggerCheckerBugs.keySet()
String skipDefCombinedRegex = skipDefRegexes.collect({ regex ->
"(${regex})"}).join("|")
List<String> skipUsesRegexes = []
// zstd-jni is not annotated, handles Zstd(De)CompressCtx.loadDict(null)
just fine
- skipUsesRegexes << "^com\\.github\\.luben\\.zstd\\..*"
+ skipUsesRegexes << "^com[.]github[.]luben[.]zstd[.].*"
// SLF4J logger handles null log message parameters
- skipUsesRegexes << "^org\\.slf4j\\.Logger.*"
+ skipUsesRegexes << "^org[.]slf4j[.]Logger.*"
String skipUsesCombinedRegex = skipUsesRegexes.collect({ regex ->
"(${regex})"}).join("|")
project.apply plugin: 'org.checkerframework'
diff --git a/examples/java/iceberg/build.gradle
b/examples/java/iceberg/build.gradle
index 4d4a1fb4441..2cff15c5fb0 100644
--- a/examples/java/iceberg/build.gradle
+++ b/examples/java/iceberg/build.gradle
@@ -26,8 +26,8 @@ plugins {
applyJavaNature(
exportJavadoc: false,
automaticModuleName: 'org.apache.beam.examples.iceberg',
- // iceberg requires Java11+
- requireJavaVersion: JavaVersion.VERSION_11
+ // iceberg requires Java17+
+ requireJavaVersion: JavaVersion.VERSION_17
)
description = "Apache Beam :: Examples :: Java :: Iceberg"
diff --git a/it/iceberg/build.gradle b/it/iceberg/build.gradle
index f11374a1ca4..3fa42fab8a7 100644
--- a/it/iceberg/build.gradle
+++ b/it/iceberg/build.gradle
@@ -22,8 +22,8 @@ plugins { id 'org.apache.beam.module' }
applyJavaNature(
automaticModuleName: 'org.apache.beam.it.iceberg',
exportJavadoc: false,
- // iceberg ended support for Java 8 in 1.7.0
- requireJavaVersion: JavaVersion.VERSION_11,
+ // iceberg ended support for Java 11 in 1.11.0
+ requireJavaVersion: JavaVersion.VERSION_17,
)
description = "Apache Beam :: IT :: Iceberg"
diff --git a/sdks/java/extensions/sql/iceberg/build.gradle
b/sdks/java/extensions/sql/iceberg/build.gradle
index 893a485e7d8..97a128d69e9 100644
--- a/sdks/java/extensions/sql/iceberg/build.gradle
+++ b/sdks/java/extensions/sql/iceberg/build.gradle
@@ -22,8 +22,8 @@ plugins { id 'org.apache.beam.module' }
applyJavaNature(
automaticModuleName:
'org.apache.beam.sdk.extensions.sql.meta.provider.hcatalog',
- // iceberg requires Java11+
- requireJavaVersion: JavaVersion.VERSION_11,
+ // iceberg requires Java17+
+ requireJavaVersion: JavaVersion.VERSION_17,
)
dependencies {
diff --git a/sdks/java/io/iceberg/build.gradle
b/sdks/java/io/iceberg/build.gradle
index 8142c5f5b90..7cdd32ed90e 100644
--- a/sdks/java/io/iceberg/build.gradle
+++ b/sdks/java/io/iceberg/build.gradle
@@ -23,8 +23,8 @@ import java.util.stream.Collectors
plugins { id 'org.apache.beam.module' }
applyJavaNature(
automaticModuleName: 'org.apache.beam.sdk.io.iceberg',
- // iceberg ended support for Java 8 in 1.7.0
- requireJavaVersion: JavaVersion.VERSION_11,
+ // iceberg ended support for Java 11 in 1.11.0
+ requireJavaVersion: JavaVersion.VERSION_17,
)
description = "Apache Beam :: SDKs :: Java :: IO :: Iceberg"
