davidzollo opened a new pull request, #12106:
URL: https://github.com/apache/seatunnel/pull/12106
## Summary
Zeta-engine-core E2E test initiative, Tier 2 (verified-fragile current
architecture, no existing
coverage): documents today's actual dynamic-slot resource accounting on a
single worker.
In the default `dynamic-slot: true` worker mode:
- `ResourceUtils#applyResourceForTask` requests every task's slot with `new
ResourceProfile()`
(see the `// TODO custom resource size` comment there), which is always 0
CPU / 0 heap memory.
- `DefaultSlotService#getNodeResource` always advertises the worker's own
CPU capacity as
`CPU.of(0)` (only heap memory is a real value), and
`DefaultSlotService#requestSlot` grants a
slot whenever `unassignedResource.enoughThan(profile)`, trivially true for
a 0/0 profile.
- Granting a slot then runs `unassignedResource.subtract(profile)` with that
same 0/0 profile,
a mathematical no-op, so the tracked unassigned resource never actually
decreases no matter how
many slots have already been handed out.
Net effect: the built-in slot capacity check never meaningfully rejects a
request on CPU grounds
in dynamic-slot mode. A single worker can be handed an arbitrarily large
number of concurrent
dynamic slots with no capacity-based pushback, bounded only by real JVM/OS
limits the resource
manager has zero visibility into. This is a missing capacity *safety net*,
not a correctness bug:
the added test confirms the job still completes normally at this concurrency.
Existing coverage (`SeaTunnelSlotIT`) only exercises the fixed-slot path (3
or 10 slots); no
existing test exercised dynamic-slot mode under concurrency anywhere close
to real capacity
limits.
## What the test does
`DynamicSlotNoCapacityCheckIT` starts a single local Zeta node
(`dynamic-slot: true`, the shared
test default) and submits one BATCH job: a `FakeSource` with `parallelism =
80` (row.num=5,
split.num=6, tiny data volume) into a `console` sink.
Scale: 80 concurrent source+sink task groups (one per partition, chained via
an in-memory queue
regardless of committer presence) + 1 split-enumerator coordinator task
group = 81 total task
groups, each requiring its own slot. This is well beyond what a
capacity-aware system would ever
grant on a small test worker (the sibling fixed-slot tests use only 3 or 10
slots as their whole
worker capacity), while staying small enough to run reliably on a shared CI
runner.
Observed / asserted:
- The worker's initial advertised capacity is `CPU.of(0)` with a real,
positive heap memory value.
- All 81 requested task groups are granted a slot with zero capacity-based
rejections
(`WorkerProfile#getAssignedSlots().length == 81` while the job is running).
- The worker's tracked unassigned resource (both CPU and heap memory) is
byte-for-byte unchanged
before vs. during the job, proving the capacity accounting never actually
decremented across
all 81 grants.
- The job still completes with `JobStatus.FINISHED` at this concurrency: no
data-processing
regression, only an absent resource guard rail.
## Test plan
- `./mvnw spotless:apply -pl
seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base -am -nsu`
- `./mvnw install -pl
'seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base,!seatunnel-engine/seatunnel-engine-ui'
-am -nsu -Dmaven.gitcommitid.skip=true -DskipTests -Dspotless.check.skip=true
-T 3C`
followed by confirming `DynamicSlotNoCapacityCheckIT.class` exists under
`seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/target/test-classes/...`
(compilation only -- per this repository's local-validation policy, actual
IT execution and
the numbers above are validated by this PR's GitHub CI, not locally).
- Test-only change: no `src/main/**` or `pom.xml` files touched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
