davidzollo opened a new pull request, #12028:
URL: https://github.com/apache/seatunnel/pull/12028
### Purpose
Adds Zeta E2E regression coverage for the classloader leak fixed by #11812
("[Fix][Zeta] Release classloaders after failed task deployment"). This is one
PR in a series adding E2E regression tests for previously-unguarded Zeta engine
bug fixes (see #12027 for the first in the series, which this PR follows in
house style).
### What #11812 fixed
`TaskExecutionService#deployTask` deserializes every task in a `TaskGroup`
in a loop, acquiring a classloader reference for each task's jars right before
deserializing that task's data. Before #11812, if a later task in the group
failed to deserialize (or its context publication failed), the classloader
references already acquired for the earlier tasks in that same attempt were
silently dropped instead of released — the outer `catch (Throwable t)` in
`deployTask` never called `classLoaderService.releaseClassLoader(...)` for
them. Every failed deployment attempt against a multi-task group leaked one
reference per already-processed task, and because nothing else in the system
ever releases a reference for a `TaskGroupContext` that was never published,
the leak was permanent and cumulative across repeated attempts.
### What coverage was missing before this PR
`ClassLoaderEnableCacheModeIT` / `ClassLoaderDisableCacheModeIT`
(`seatunnel-e2e/.../e2e/classloader/`) already prove classloader/thread counts
stay bounded across 10 repeated **successful** job restarts (submit, run to
completion, resubmit). That is a different code path entirely: their trigger is
a job that runs and finishes; this bug only manifests when deployment of a
multi-task group **fails partway through**, after some tasks' classloaders were
already acquired. Those tests cannot see this regression because they never
construct a failing deployment. The only existing coverage of this exact bug is
the unit test that shipped with #11812 itself
(`TaskExecutionServiceTest#testDeployTaskReleasesClassLoadersWhenDeserializationFails`),
which is single-node, single-attempt, and bypasses the E2E/cluster layer
entirely.
### What this test does
`TaskDeploymentClassLoaderLeakIT#testFailedMultiTaskDeploymentDoesNotLeakClassLoaders`:
1. Boots a real 2-node cluster (one master, one worker) via
`SeaTunnelServerStarter`, matching the house style already established by
`SplitClusterPendingJobLifecycleFailoverIT` (#12027).
2. Obtains the worker's real `TaskExecutionService` and
`DefaultClassLoaderService` directly (`deployTask` runs on the worker in
production — the master issues it as an RPC), since engineering a specific
task's deserialization to fail needs white-box control over the serialized
payload.
3. Repeats 10 times (matching `ClassLoaderITBase`'s iteration count): builds
a 3-task `TaskGroup` where the first two tasks are minimal valid `Task`
implementations, each with its own jar (standing in for "earlier tasks" that
successfully acquire a classloader), and the third task's serialized payload is
a plain `String` instead of a `Task`. Deploying that group makes `deployTask`'s
deserialization loop throw `ClassCastException` on the third task — after the
first two tasks' classloaders have already been acquired. Each attempt uses a
fresh jobId/jars/`TaskGroupLocation` so no state is shared between attempts.
4. After every attempt, asserts:
- `deployTask` reports failure and never publishes a `TaskGroupContext`
(`TaskGroupContextNotFoundException` on lookup).
- The classloader reference count for each of the 3 tasks' jars is
exactly 0 (`DefaultClassLoaderService#queryClassLoaderReferenceCount`) — the
same precise signal #11812's own unit test uses.
- The service's total live classloader count
(`DefaultClassLoaderService#queryClassLoaderCount()`) has returned to the
pre-loop baseline — a coarser, `ClassLoaderITBase`-style bounded-growth check.
Classloader cache mode is disabled for this test specifically so a released
classloader is actually evicted from the cache (the engine's default cache mode
never evicts a cached entry, which would mask this signal), making this a
meaningful additional check on top of the exact reference-count assertion.
All of these assertions are synchronous — `deployTask`'s failure path
releases classloaders in the same `catch` block before returning — so no
`Awaitility` polling is needed for them; `Awaitility` is only used to wait for
the 2-node cluster to form at startup.
### Why the trigger is reliable
The "serialize a `String` where a `Task` is expected" mechanism is not a
synthetic exception — it is a genuine deserialization/type mismatch, and it is
the exact mechanism #11812's own accompanying unit test uses to reproduce this
bug. I additionally verified the underlying call path
(`com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject#deserializeWithCustomClassLoader`,
which every task in this test goes through since all 3 tasks use non-empty
jars) by decompiling it: it deserializes the `String` successfully and returns
it as `Object`; the `ClassCastException` actually fires at
`TaskExecutionService.deployTask`'s own implicit checkcast on the assignment to
the `Task task` local variable, immediately after the classloader for that task
has already been recorded in `deployTask`'s `acquiredClassLoaderJars` list.
Reusing a proven mechanism (rather than inventing a new poison-task type)
minimizes the risk that this test's trigger is itself unreliable or diverges
from what
the original fix actually guards against.
### Test plan
- `./mvnw spotless:apply -pl
seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base -am -nsu` —
BUILD SUCCESS.
- `./mvnw install -pl
seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base -am -nsu
-Dmaven.gitcommitid.skip=true -DskipTests -Dmaven.test.skip=true
-Dspotless.check.skip=true -T 3C` — BUILD SUCCESS.
- The test itself is not run locally per this series' convention (Apache
SeaTunnel local-verification policy restricts local runs to
formatting/compilation only); it will run under this PR's GitHub Actions CI
(`seatunnel-e2e/seatunnel-engine-e2e` job).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
