loustler commented on PR #11648: URL: https://github.com/apache/seatunnel/pull/11648#issuecomment-5204807431
@DanielLeens — the `seatunnel-hadoop-aws` Jackson relocation problem we kept out of this branch as #11655 now has a fix up: **#11673**. I confirmed the mechanism at the bytecode level rather than by reading poms. The uber jar ships: ``` public org.apache.seatunnel.shade.hadoop.com.fasterxml.jackson.databind.ObjectMapper getMapper(); ``` while `hadoop-aws`'s call site kept the original descriptor: ``` invokevirtual JsonSerialization.getMapper:()Lcom/fasterxml/jackson/databind/ObjectMapper; ``` The JVM matches on name + descriptor, so that resolves against nothing — `NoSuchMethodError` on any `fs.s3a.assumed.role.*` path. The fix is a single relocation that uses the *uber jar's* shaded pattern rather than this module's own prefix, since the point is to agree with the class that supplies the method. Nothing is added to the jar; only references are rewritten (194 classes under `org/apache/hadoop/` before and after). Scope kept tight as you asked on the issue: one relocation, no change to the uber jar, no Hadoop or SDK version change, no unrelated AWS cleanup. **One finding that bears directly on this PR.** On `dev`'s hadoop-aws 3.1.4 the defect touches 3 classes (`RoleModel` and two nested types). On **3.4.3 — the version this branch bumps to — it touches 6**, and the three extra ones fail *silently* rather than loudly: `SuccessData`, `PendingSet` and `SinglePendingCommit` carry unrelocated `@JsonProperty` annotations. The relocated Jackson looks for relocated annotations, does not find them, and does not throw — it falls back to default property naming and inclusion, so the S3A committer writes `_SUCCESS` and `.pending` files with the wrong shape and nothing surfaces an error. So this is not only a pre-existing `dev` bug that happens to be adjacent to this PR: on the version this branch ships it gets strictly worse, and the new mode is one no stack trace would ever point at. #11673 is based on `dev` (3.1.4), so once this branch lands the same relocation covers the additional three. #11673 also adds a jar-level check (`tools/dependencies/check_shaded_jackson_refs.py`) wired into the existing `dependency-license` job — that is what surfaced the extra three classes, which no assumed-role test would have reached. Separately, on Issue 1 from this round (the `PaimonWithS3IT.privilegeEnabledPaimonSourceUnAuthorized` hang): rather than assume either way on attribution, I'm running `paimon-connector-it` against a pristine `dev` baseline to see whether the hang reproduces without any of this PR's changes. I'll follow up here with that result and a concrete proposal once it finishes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
