loustler opened a new pull request, #11648:
URL: https://github.com/apache/seatunnel/pull/11648
Closes #9599
Closes #9787
This upgrades the shaded Hadoop uber jar from 3.1.4 to 3.4.3 and,
necessarily with it, moves S3A from AWS SDK v1 to v2. I ran into this while
upgrading a production cluster, so most of the claims below come with
measurements attached; I've tried to show the evidence rather than just assert
it. I'd very much welcome correction where I've misread something.
## Why
Two dependency-lifecycle reasons, not features:
1. **Hadoop 3.1.4 is long out of the Hadoop project's active release lines**
— it is the last 3.1.x release, and 3.1 has had no updates since. #9599 asked
for the move on the grounds that 3.4.x carries fewer known CVEs.
2. **AWS SDK for Java v1 reached end-of-support on 2025-12-31.** SeaTunnel
still ships it, because `hadoop-aws` 3.1.4 depends on
`com.amazonaws:aws-java-sdk-bundle`. This is #9787.
These look like two tasks but are one. As the discussion in #9599 already
notes, Hadoop 3.4's S3A *is* the migration from AWS SDK v1 to v2 — the Hadoop
upgrade cannot be taken without the SDK upgrade.
A small note on #9599: it is currently closed as completed, but I may be
missing where that landed — `dev` still contains
`seatunnel-shade/seatunnel-hadoop3-3.1.4-uber` with
`<hadoop3.version>3.1.4</hadoop3.version>`, and
`<hadoop-aws.version>3.1.4</hadoop-aws.version>` in the root pom, and the
attempt in #9669 was closed unmerged. If it was closed by mistake, would you
consider reopening it? Otherwise please feel free to drop the `Closes` line for
it.
## Why the two Hadoop versions must move together
`hadoop-aws` declares `hadoop-common` as `provided` at every version, so at
runtime it links against whatever `hadoop-common` the uber jar ships. They are
**not independently upgradable**. Bumping only `hadoop3.version` breaks S3A
outright — verified by resolving every external class and Hadoop-API member
reference of `seatunnel-hadoop-aws.jar` against the real runtime classpath:
| uber / hadoop-aws | unconditional breaks | conditional |
|---|---|---|
| 3.4.3 / 3.1.4 | **2** | — |
| 3.4.3 / 3.3.6 | 0 | 3 |
| **3.4.3 / 3.4.3** | **0** | **0** |
The two unconditional failures at 3.4.3/3.1.4:
- `NoClassDefFoundError: org/apache/commons/lang/StringUtils` in
`S3AFileSystem.initialize()` — `hadoop-common` dropped commons-lang 2.x after
3.1.4, so nothing on the classpath supplies it. Fails on *any* S3A open.
- `NoSuchMethodError` on `SemaphoredDelegatingExecutor.<init>` in
`S3AFileSystem.create()` — the guava-typed constructor was removed. Fails on
*every* S3A write. Cannot be fixed by adding a dependency; the bytecode calls a
constructor that no longer exists.
For that reason this PR moves `hadoop3.version` and `hadoop-aws.version`
together to 3.4.3, rather than bumping the uber jar alone.
## Changes
### 1. Hadoop 3.1.4 → 3.4.3
- `seatunnel-shade/seatunnel-hadoop3-3.1.4-uber` →
**`seatunnel-hadoop3-3.4.3-uber`** (directory + `artifactId`, following the
existing naming convention). 92 references across 56 files.
- `hadoop3.version` → `3.4.3`, `hadoop-aws.version` → `3.4.3`.
- `finalName` and the build-helper artifact path now derive from
`${hadoop3.version}` instead of repeating the literal, so the next bump is one
line.
- `maven-shade-plugin` pinned to 3.6.2 **for this module only**:
`hadoop-client` 3.4.3 pulls `org.bouncycastle:bcprov-jdk18on`, a multi-release
jar with `META-INF/versions/21` classes at class file major version 65, which
the ASM bundled with older shade releases cannot read (`Unsupported class file
major version 65`).
> **A suggestion, deliberately not done here:** a version-neutral
`seatunnel-hadoop3-uber` artifactId would remove what seems to be the recurring
cause of these upgrade PRs going stale, since every bump currently renames a
module and touches ~56 files. I kept the version in the name to respect the
existing convention, but I'd be glad to change it if maintainers would prefer
that — please just say so.
### 2. AWS SDK v1 → v2
- Root pom `software.amazon.awssdk.version` `2.31.30` → **`2.35.4`**,
matching the `aws-java-sdk-v2.version` that `hadoop-project` 3.4.3 pins.
This alignment matters and is the concern raised in #9787. `hadoop-aws`
3.4.x depends on `software.amazon.awssdk:bundle`, so the shaded
`seatunnel-hadoop-aws.jar` lands in `lib/` next to
`seatunnel-transforms-v2.jar`, which already bundles `bedrockruntime` +
`apache-client`. Measured on the built distribution: **396,759** vs **3,732**
`software/amazon/awssdk/**` entries, sharing `core`, `http`, `regions`, `auth`,
`awscore` and `utils`. If the versions differ, classpath order decides which
copy of that shared infrastructure wins. Keeping them equal makes the
duplicated classes byte-identical, so order stops mattering. This collision did
not exist under SDK v1 (`com/amazonaws/**` vs `software/amazon/awssdk/**`).
- `checkpoint-storage-hdfs` hardcoded `hadoop-aws` 3.1.4 and
`aws-java-sdk-bundle` 1.11.271; both now use the shared properties, so they
cannot silently drift again.
- `seatunnel-dist` and `assembly-seatunnel-bin-ci.xml`:
`com.amazonaws:aws-java-sdk-bundle` → `software.amazon.awssdk:bundle`; the
now-unused `aws-java-sdk.version` property is removed.
- `S3FileBaseOptions.INSTANCE_PROFILE_CREDENTIALS_PROVIDER`:
`com.amazonaws.auth.InstanceProfileCredentialsProvider` →
`org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider`.
This one is not optional, unfortunately: that constant is the **default
value** of `fs.s3a.aws.credentials.provider`, so under SDK v2 every S3 job that
does not name a provider explicitly would fail to instantiate one.
`IAMInstanceCredentialsProvider` is Hadoop's own provider, present since
hadoop-aws 3.3.x and SDK-version-agnostic. S3A's v1-to-v2 adapter cannot help
here — it needs `com.amazonaws:aws-java-sdk-core`, which `hadoop-aws` declares
`provided` and is never packaged.
**Existing job configs keep working.** Users write the *enum constant
name* and `S3HadoopConf` writes `getProvider()` into the Hadoop conf, so
changing only the mapped class name is config-compatible.
### 3. Keep log4j 1.x out of the uber jar
This is the one change that is not a version bump, and it is a prerequisite:
**bumping Hadoop without it ships log4j 1.x into the uber jar.**
Measured on `dev` — `hadoop3.version` set to 3.4.3 with no other change,
then the built jar inspected at its root paths:
| root path | before this PR | after |
|---|---|---|
| `org/apache/log4j/**` | **232** | **0** |
| `org/slf4j/impl/**` | 0 | 0 |
The 232 entries come from `ch.qos.reload4j:reload4j`, the log4j 1.2.17 fork
that `hadoop-common` 3.3.3+ depends on under a new GAV. The existing
`<exclude>log4j:*</exclude>` in the shade `artifactSet` does not match that
GAV, and neither does the `log4j:log4j` `provided` entry in
`dependencyManagement`.
This is worth spelling out, because it is easy to read as cosmetic and
isn't. `lib/*` precedes `seatunnel-starter.jar` on the runtime classpath
(`bin/seatunnel-cluster.sh`), so reload4j's real `org.apache.log4j.Category`
**shadows** the one `log4j-1.2-api` contributes to the starter jar.
Disassembled, the difference is stark:
```
seatunnel-starter.jar (log4j-1.2-api shim) reload4j 1.2.22 (real log4j
1.x)
public void callAppenders(LoggingEvent); public void
callAppenders(LoggingEvent);
Code: 12: monitorenter
0: return // no-op 26: invokevirtual
appendLoopOnAppenders
59: getfield parent
// lock again
```
So code using the `org.apache.log4j` API stops reaching log4j2 and instead
gets an unconfigured log4j 1.x — default root level **DEBUG** — whose
`callAppenders` takes a monitor on every `Category` up to the shared root.
(Note the access flags are `ACC_PUBLIC` on *both*: log4j 1.x locks with a
`synchronized(c)` block inside the method, not a synchronized method. Only the
body distinguishes them.)
**Proposed fix**: add `ch.qos.reload4j:reload4j` at `provided` scope in
`dependencyManagement` and `<exclude>ch.qos.reload4j:*</exclude>` to the shade
`artifactSet`, mirroring how `log4j:log4j` is already handled. The
`org.apache.log4j` API stays available at runtime through `log4j-1.2-api`,
correctly routed to log4j2 and governed by `log4j2.properties`.
Additionally, as **defence in depth** (these are no-ops on current `dev`,
which already manages `org.slf4j:slf4j-reload4j` to `provided`):
- explicit `slf4j-reload4j` / `slf4j-log4j12` exclusions on `hadoop-client`
— Hadoop's own "No slf4j backends for downstream clients" list still names only
`slf4j-log4j12`, never updated after HADOOP-18088 switched the backend to
`slf4j-reload4j` in 3.3.3+;
- `<exclude>org/slf4j/impl/**</exclude>` in both shade modules, so the
guarantee holds for a build that does not inherit this parent's
`dependencyManagement`.
The pattern is deliberately `org/slf4j/impl/**` and **not**
`org/slf4j/**`: only the binding is excluded, never the SLF4J API. It is also
anchored at the jar root, so it intentionally leaves the AWS SDK's own
relocated copy under `software/amazon/awssdk/thirdparty/org/slf4j/**` alone —
those are different fully-qualified names, inert for SLF4J binding resolution,
and required for the SDK's internal logging.
### 4. Declare okhttp 2.x explicitly in `seatunnel-engine-server`
`JobEventHttpReportHandler` is written against the legacy okhttp 2.x API
(`com.squareup.okhttp.*` — a different groupId *and* Java package from the
`com.squareup.okhttp3:okhttp` 4.x managed in the root pom), but the module
never declared it. The classes arrived transitively from `hadoop-hdfs-client`,
which depended on `com.squareup.okhttp:okhttp` 2.7.5 up to Hadoop 3.1.4, moved
to `okhttp3` in 3.3.6, and ships neither in 3.4.x.
As far as I can tell this is a latent undeclared-dependency issue on `dev`
today, which only turns into a build failure once Hadoop moves. Declaring it
explicitly at 2.7.5 — matching the already-present test-scoped `mockwebserver`
— is what the reviewer of #10676 suggested for this same problem, so I've
followed that.
### 5. Docs
- `docs/en` + `docs/zh`: 50 occurrences of `com.amazonaws.auth.*` class
names updated across 12 files (`S3File` source/sink, `Hive` source/sink,
`checkpoint-storage`, `kubernetes/configuration`), plus the `Hadoop 3.1.4`
references in the provider-contract prose.
- `docs/{en,zh}/introduction/concepts/incompatible-changes.md`: new entry
covering the artifactId rename, the SDK v1 → v2 class-name migration table, and
an explicit note that enum-based configs need no change.
## Verification
- `mvn package` on the changed modules (`seatunnel-hadoop3-3.4.3-uber`,
`seatunnel-engine-server`, `connector-file-s3`, `checkpoint-storage-hdfs`) with
`-am`: **BUILD SUCCESS**.
- Uber jar at the root paths that matter: `org/slf4j/impl/**` 0,
`org/slf4j/**` 0, `org/apache/log4j/**` 0 (232 without the guard).
- Binary compatibility of `seatunnel-hadoop-aws.jar` against the real 4-jar
runtime classpath at 3.4.3/3.4.3: 485 class references and 471 member
references resolve; 0 unconditional and 0 conditional breaks.
- On a full distribution built from an equivalent change set: exactly one
SLF4J binding on the runtime classpath (`seatunnel-starter.jar`), no `lib/` jar
carrying a root `org/slf4j/impl/**` or `org/apache/log4j/**`, and
`IAMInstanceCredentialsProvider` present.
## Known trade-off: distribution size
`hadoop-aws` 3.4.x depends on `software.amazon.awssdk:bundle`, which
contains every AWS service client, so the shaded `seatunnel-hadoop-aws.jar`
grows from ~83 MB to ~658 MB and the binary tarball roughly doubles.
Narrowing the dependency to what S3A actually uses (`s3`, `sts`, `dynamodb`,
`apache-client` and their shared core) should bring it to roughly 80–120 MB. I
left that out of this PR on purpose: S3A resolves provider classes by name via
reflection, so a missing module surfaces only at runtime and would need real S3
integration testing to validate, which felt like a separate change. If you
would rather not merge the size increase, I'm happy to take that on in this PR
instead — I'd appreciate your guidance on which you'd prefer.
## Pre-existing issue found but not fixed here
`JsonSerialization.getMapper()` in `hadoop-aws` is broken **even with the
current 3.1.4 uber jar**: the uber jar relocates Jackson to
`shade.hadoop.com.fasterxml.jackson` while `hadoop-aws` references the original
package. It affects only `RoleModel` (assumed-role IAM policy JSON). I've left
it alone as out of scope here, but wanted to flag it in case it's useful.
## Remaining AWS SDK v1 usage
Two E2E test modules still import `com.amazonaws.*`:
`connector-databend-e2e` (3 files) and `connector-file-s3-e2e` (1 file). All
production connector main sources are already on SDK v2 (`amazondynamodb`,
`amazonsqs`, `iceberg`, `jdbc`, `transforms-v2`), so after this PR the shipped
artifact should no longer carry SDK v1. I'm happy to convert the test modules
here or in a follow-up, whichever fits your review preferences better.
## Does this PR introduce a *user-facing* change?
Yes — documented in `incompatible-changes.md` (en + zh). Two breaking
changes: the uber jar `artifactId` rename, and `com.amazonaws.*`
credentials-provider class names no longer resolving. Jobs using the typed enum
values are unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
