DanielLeens commented on PR #11307:
URL: https://github.com/apache/seatunnel/pull/11307#issuecomment-5370748173
@SEZ9 Quick clarification — the review body isn't actually cut off. I just
re-pulled the raw body directly via the GitHub API (`GET
/repos/apache/seatunnel/pulls/11307/reviews/4978389826`) and it's the full
11,995 characters, well under GitHub's ~65536-char single-comment limit, so
nothing was dropped on my end. This may be a rendering artifact on the PR page
rather than a real truncation.
For convenience, here's the part right after the "Before" snippet — the
"after" (post-merge, regressed) code path plus the fix guidance:
```java
// After (current head, post-merge):
boolean spark35 =
container.getDockerImageName().startsWith("apache/spark:3.5");
String dependencyDirectory = spark35 ? "/opt/spark/jars" :
"/tmp/seatunnel/plugins/Iceberg/lib";
DependencyJar.of(S3AFileSystem.class).copyTo(container, dependencyDirectory);
DependencyJar.of(AmazonS3.class).copyTo(container, dependencyDirectory);
```
Root cause: `DependencyJar.of(...)` resolves the **modular** `hadoop-aws` /
`aws-java-sdk-s3` jars off the local Maven test classpath, not the
previously-used **shaded/uber** `aws-java-sdk-bundle` jar. The modular jars
need an external Guava; Spark 3.5.8 only ships its own bundled
`guava-14.0.1.jar`, so `S3AFileSystem`'s `SimpleAWSCredentialsProvider` hits
`NoSuchMethodError: Preconditions.checkArgument(boolean, String, Object,
Object)` at runtime — deterministic on both the JDK 8 and JDK 11 lanes,
confirmed in the fork CI run.
Two fix options (from the review, location `IcebergSourceIT.java:93-101`):
- **Option A** (restores previously-verified behavior): for the `spark35`
branch, stage a shaded/uber jar (e.g. `aws-java-sdk-bundle`) instead of the
modular jars, so no external Guava is required.
- **Option B**: keep `DependencyJar` for consistency with `dev`, but stage
the modular jars into an isolated SeaTunnel plugin/classpath location added via
`--jars` / `spark.driver.extraClassPath` instead of dumping them into Spark's
own `jars/` directory, preserving classpath isolation from Spark's bundled
Guava.
Once one of those lands and the Iceberg S3 E2E lane goes green on the
updated head, I'll do a fresh full re-review per the protocol — no need to wait
on me for anything else in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
