loustler commented on PR #11648:
URL: https://github.com/apache/seatunnel/pull/11648#issuecomment-5201160883
Thanks — all three are addressed. Issue 1 turned out not to be a
classloading problem at all, and Issue 3 turned out to be a real regression
rather than a one-off, so both write-ups below lead with the evidence.
## 1. `paimon-connector-it` — the jar was never built
The comparison you asked for points at the build, not at classloading.
`connector-file-s3-e2e/pom.xml` declares a dependency on
`seatunnel-hadoop-aws`. `connector-paimon-e2e` and `connector-hudi-e2e` did
not. `paimon-connector-it` runs `-pl :connector-paimon-e2e -am`, and in run
`31003361628` that reactor has 85 modules with **no** `SeaTunnel : Shade :
Hadoop : AWS` among them — so
`seatunnel-shade/seatunnel-hadoop-aws/target/seatunnel-hadoop-aws.jar` never
existed.
A missing host path does not fail the copy. Testcontainers 1.17.6
`MountableFile.transferTo` writes a zero-byte tar entry for it:
```
MountableFile.forHostPath("/definitely/not/here/seatunnel-hadoop-aws.jar")
→ no exception
→ entry: opt/seatunnel/lib/seatunnel-hadoop-aws.jar size=0 isFile=true
```
That is why the container started normally — every other test in the job
passed — with an empty `lib/seatunnel-hadoop-aws.jar` and no S3A anywhere on
the classpath. Hence `ClassNotFoundException` rather than the old
`NoSuchMethodError`.
Hudi in the same run is the control:
| | `paimon-connector-it` | `all-connectors-it-5` (Hudi) |
|---|---|---|
| `Shade : Hadoop : AWS` built | 0 | 1 |
| `NoClassDefFoundError: .../fs/s3a` | 16 | 0 |
| fails at | S3A absent | thread check |
So the mechanism itself works. Hudi only got a real jar because that job's
module list is wide enough to build it for an unrelated reason — run
`connector-hudi-e2e` on its own and it breaks identically. Both poms now
declare it, and both single-module reactors build the jar (paimon 85 → 86
modules).
The dependency is declared with its transitives excluded: the jar is only
read off disk and copied, never linked against. In `connector-paimon-e2e` it is
also declared last, so the S3A that module's own JVM resolves stays the one
`paimon-s3-impl` bundles, per the ordering comment already in that pom.
The three copies now go through one helper,
`AbstractTestContainer#copyHadoopAwsToContainerLib`, which throws with the fix
instruction if the jar is absent. That covers your 3.2 suggestion, and it means
the next person to hit this gets an error at the copy instead of a
`ClassNotFoundException` two minutes later.
## 2. `all-connectors-it-5` — Option A, plus a third name
Applied. The dump has three families, not two:
| thread | count | daemon |
|---|---|---|
| `sdk-ScheduledExecutor-<pool>-<n>` | 187 | yes |
| `s3-analytics-accelerator-<n>` | 33 | yes |
| `idle-connection-reaper` | 1 | yes |
`s3-analytics-accelerator-*` would have failed the check on its own, so all
three are allowlisted. Only the reaper identifies itself in its stack
(`software.amazon.awssdk.http.apache.internal.conn.IdleConnectionReaper$ReaperTask`);
the other two are parked pool workers with no library frame, so the comment
describes what the dump shows rather than naming a creator I cannot see.
One thing worth saying plainly: 187 is one scheduler per S3 client, and
hadoop-aws builds a client per `S3AFileSystem` instance, so a multi-table job
accumulates them. They are all daemon threads parked on an empty queue and do
not hold the JVM open, but the count is a property of that
client-per-filesystem model, not something the test is doing wrong.
## 3. `engine-k8s-it` — not a one-off, and it is mine
It is deterministic: red on all three commits where it ran (`cdd32c898`,
`fdff4b65c`, `128b94d85`). Your dev check holds up — the `befd31c453` job
genuinely ran for 7 minutes and passed on both JDKs.
It has nothing to do with S3 (zero `s3a` or `hadoop-aws` strings in the
log). It fails on the first statement of `runDiscoveryTest`,
`Config.defaultClient()`:
```
NoSuchFieldError: Companion
NoClassDefFoundError: Could not initialize class
okhttp3.internal.concurrent.TaskRunner
```
The module's test classpath is byte-identical to dev — 160 artifacts, zero
diff, with and without `-Pci`. That is what sent me inside the jars, since
`dependency:build-classpath` cannot see shaded content.
This PR added `com.squareup.okhttp:okhttp:2.7.5` to
`seatunnel-engine-server` at compile scope, because hadoop 3.4.x no longer
supplies it for `JobEventHttpReportHandler`. On dev that library reached the
module through the `provided` uber jar and was therefore never shaded. Now it
is a compile dependency, and `seatunnel-starter.jar` bundles it:
```
com/squareup/okhttp/ 184 entries
okio/ 35 entries <- okio 1.6.0
```
okio 1.x is plain Java, okio 3.x is Kotlin, and they share the `okio`
package. The Kubernetes client's okhttp3 4.12.0 is compiled against okio 3.x,
so when the starter's okio 1.6.0 wins the classpath there is no
`okio.ByteString.Companion` field — exactly the error above.
`seatunnel-starter` now relocates `okio` to
`${seatunnel.shade.package}.okio`. Verified on the rebuilt jar: `okio/` 0
entries, `org/apache/seatunnel/shade/okio/` 35, and
`com.squareup.okhttp.RequestBody` bytecode now references
`org/apache/seatunnel/shade/okio/ByteString` with no bare `okio/` reference
left. okhttp 2.x itself needs no relocation — `com.squareup.okhttp` is its own
package and collides with nothing.
Thanks for holding the line on this one. A rerun would have looked like a
flake and hidden a real regression.
## Not touched
`all-connectors-it-3` (PostgresCDC, flaky) and `kafka-connector-it` (red on
dev as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
