loustler opened a new pull request, #20:
URL: https://github.com/apache/seatunnel-shade/pull/20
@hawk9821 @davidzollo — this carries two changes that were made against
`apache/seatunnel`'s in-repo `seatunnel-shade/` tree and now have to live here
instead.
### Why it's here
- apache/seatunnel#11673 relocated Jackson in `seatunnel-hadoop-aws` to
match the uber jar. Merged there as `2586f37e96df`.
- apache/seatunnel#11851 reverted it, because shade had moved to this repo.
Its follow-up note asks for the fix to be resubmitted here, which is what
commit 1 does.
- apache/seatunnel#9993 then landed the actual cutover — `seatunnel-shade/`
is gone from `apache/seatunnel` and consumers pull `seatunnel-shade-*` from
Central.
- apache/seatunnel#11648 is my open PR upgrading the shaded Hadoop to 3.4.3
and moving S3A to AWS SDK v2. Three of its 88 files were shade poms; those are
commit 2 here. The other 85 (connectors, dist, e2e, docs) stay in
`apache/seatunnel` and are blocked until this releases.
### Commit 1 — Jackson relocation in `seatunnel-shade-hadoop-aws`
`hadoop-aws` carries no Jackson classes of its own. It references them
through `hadoop-common`, which this jar does not bundle either —
`seatunnel-shade-hadoop3-uber` supplies it, and that jar already relocates
`com.fasterxml.jackson` to
`${seatunnel.shade.package}.hadoop.com.fasterxml.jackson`. So
`JsonSerialization#getMapper` is declared to return the relocated
`ObjectMapper`.
Left unrelocated here, `RoleModel`'s call site keeps the original descriptor
`()Lcom/fasterxml/jackson/databind/ObjectMapper;` and fails at runtime with
`NoSuchMethodError` on any `fs.s3a.assumed.role.*` path.
The relocation uses the **uber jar's** shaded pattern, not this module's own
prefix — the point is to agree with the class that supplies the method. Nothing
is added to the jar; only references are rewritten.
### Commit 2 — Hadoop 3.4.3 and AWS SDK v2
**Both modules move together.** `hadoop-aws` declares `hadoop-common` as
`provided`, so at runtime it links against whatever `hadoop-common` the uber
jar ships. `hadoop-aws` 3.1.4 against a 3.4.x uber jar fails with
`NoClassDefFoundError org/apache/commons/lang/StringUtils` in
`S3AFileSystem.initialize()` (hadoop-common dropped commons-lang 2.x after
3.1.4) and `NoSuchMethodError` on `SemaphoredDelegatingExecutor.<init>` in
`S3AFileSystem.create()` (the guava-typed constructor was removed).
**Version placement.** Both bumps are declared in the child modules' own
`<properties>`, not in the parent, following the incremental-release rule in
`README.md` — the parent is published at 3.0.0 and its properties are frozen.
Happy to move them to the root pom if you'd rather do a full re-release.
**AWS SDK bundle filtering.** `hadoop-aws` 3.4.x depends on
`software.amazon.awssdk:bundle`, which is 686,119,507 bytes and carries all 410
AWS services. Resolving every `software/amazon/awssdk` reference in the 466
classes of `hadoop-aws` 3.4.3 — as type references and again as dotted strings,
to catch reflective lookups — yields exactly three: `s3`, `sts`, `kms` (the
last only `KmsClient`/`KmsClientBuilder`, from `EncryptionS3ClientFactory`). No
`dynamodb`; S3Guard was removed in Hadoop 3.4.0.
An allowlist filter keeps those three plus the 25 non-service packages,
dropping 380,121 entries and 564,452,812 compressed bytes. The shaded jar comes
out at **23,231,873 bytes instead of roughly 0.6 GB** — worth caring about for
something ASF mirrors and archives indefinitely.
Two things in that filter are deliberate and easy to "simplify" wrongly, so
they're commented in the pom:
1. It's an allowlist, not a denylist. maven-shade's `SimpleFilter` is
`!(isIncluded && !isExcluded)`, so an exclude can never be re-included and
"everything except these services" isn't expressible.
2. The bundle is filtered rather than replaced by the individual
`s3`/`sts`/`kms`/`apache-client` modules, because `hadoop-aws` is compiled
against the bundle's *relocated* Apache HTTP classes —
`ConfigureShadedAWSSocketFactory` references
`software/amazon/awssdk/thirdparty/org/apache/http/...`, which standalone
`apache-client` doesn't carry. `NetworkBinding` loads that class reflectively
and swallows the failure, so the substitution wouldn't fail the build; it would
quietly turn `fs.s3a.ssl.channel.mode` into a no-op.
**SLF4J binding.** `hadoop-client` 3.3.3+ pulls `org.slf4j:slf4j-reload4j`
through `hadoop-common` (HADOOP-18088), and Hadoop's own "no slf4j backends for
downstream clients" exclusion list still names only `slf4j-log4j12`. This
project manages no SLF4J binding, so the uber jar would package
`org/slf4j/impl/**` — `apache/seatunnel`'s root pom used to keep that out, and
that safety net doesn't exist here. SLF4J would bind to an unconfigured log4j
1.x whose root logger defaults to DEBUG, Parquet would install
`RecordConsumerLoggingWrapper`, and every Parquet writer thread in the JVM
would serialise on `Category.callAppenders`.
Guarded twice: exclusions on the `hadoop-client` dependency, and a jar-level
`org/slf4j/impl/**` filter in both modules. Only the binding, never the API —
and the pattern is anchored at the jar root, so the SDK's own relocated copy
under `software/amazon/awssdk/thirdparty/org/slf4j/**` is kept (inert for
binding resolution, and the SDK needs it).
### Release note
This produces two new coordinates, so there's no Nexus conflict with what's
already published:
- `seatunnel-shade-hadoop3-uber:3.4.3-3.0.0`
- `seatunnel-shade-hadoop-aws:3.4.3-3.0.0`
```bash
mvn -B -DskipTests -Drat.skip=true \
-pl seatunnel-shade-hadoop3-uber,seatunnel-shade-hadoop-aws \
clean deploy
```
apache/seatunnel#11648 can't build until those are on Central — after that
it's a two-property bump (`seatunnel.shade.hadoop.version`,
`seatunnel.shade.hadoop-aws.version`) plus the non-shade files. If you can give
me a rough sense of the release turnaround I'll plan that PR around it.
### How it was verified
This repo has no CI, so everything below was run by hand on JDK 11:
```bash
mvn -B -DskipTests -Drat.skip=true -Dgpg.skip=true \
-pl seatunnel-shade-hadoop3-uber,seatunnel-shade-hadoop-aws clean install
```
| Check | Result |
|---|---|
| Build | `BUILD SUCCESS`; `seatunnel-shade-hadoop3-uber-3.4.3-3.0.0.jar`
(64,690,345 B), `seatunnel-shade-hadoop-aws-3.4.3-3.0.0.jar` (23,231,873 B) |
| `maven-shade-plugin` 3.4.1 vs bcprov's Java 21 classes | fine — the
existing `META-INF/versions/21/**` exclusion from #16 covers `bcprov-jdk18on`;
no plugin bump needed |
| SLF4J binding at jar root | 0 entries under `org/slf4j/impl/` in both
jars; SLF4J API still present in the uber jar (38 entries) |
| SDK's relocated slf4j | 16 entries kept, all under
`software/amazon/awssdk/thirdparty/org/slf4j/impl/` |
| AWS services in hadoop-aws jar | exactly `s3`, `sts`, `kms` |
| Jackson descriptors agree | uber's `JsonSerialization#getMapper()` returns
`org.apache.seatunnel.shade.hadoop.com.fasterxml.jackson.databind.ObjectMapper`;
hadoop-aws's `RoleModel` call site is
`…getMapper:()Lorg/apache/seatunnel/shade/hadoop/com/fasterxml/jackson/databind/ObjectMapper;`
|
| Unrelocated Jackson refs | 0 — all 13 occurrences across hadoop-aws's 466
classes are relocated; 0 Jackson classes bundled |
### Not included
#11673 also added `tools/dependencies/check_shaded_jackson_refs.py` and a CI
step that asserted the descriptor agreement above. There's no `.github/` in
this repo, so I left it out rather than adding a script nothing runs. Glad to
bring it over if you'd like to introduce CI here — it's the check I ran by hand
for the last two table rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
