davidzollo opened a new pull request, #12107:
URL: https://github.com/apache/seatunnel/pull/12107
## Purpose
No existing E2E test in this suite exercises sustained backpressure
between a fast source and a slow sink — every existing scenario either
runs at matched throughput or checks only a momentary condition. This
adds one, as a Tier-3 scale/stress scenario in a larger Zeta engine-core
E2E initiative (Tier 1 already shipped 9 PRs for proven-incident
regressions).
## What this test does
`BackpressureSlowSinkIT` pairs a fast `FakeSource` with a deliberately
slow `InMemory` sink (a new opt-in `write_delay_ms` config option) and
runs it under sustained backpressure for ~90 seconds with checkpointing
and observability enabled. It asserts:
- Strict checkpoint continuity: the completed-checkpoint count is
monotonic non-decreasing and gains at least 3 new completions during
the window — not just "the job didn't crash."
- Genuine bounded-queue backpressure, via the existing realtime-metrics
REST endpoint: positive configured queue capacity and a measured
positive `emitBlockedNs` on the slow edge, using the same field names
and endpoint already exercised by the existing `RealtimeMetricsRestIT`.
## Shared test-infrastructure change:
`InMemorySinkFactory`/`InMemorySinkWriter`
The new `write_delay_ms` option defaults to `0L` and is only checked
after the pre-existing `writer_sleep` block — purely additive. I
verified every existing caller of the `InMemory` sink (13 `.conf` files
and 5 `.java` references, including `pending_jobs_streaming.conf`,
`pending_jobs_streaming_lifecycle.conf`, and `ClassLoaderITBase`) — none
set this new option, so all of them get the default and run the exact
same code path as before.
## Test plan
- 2 files modified (shared test-infra sink, additive only), 2 new files
(test + resource config); no production code changed.
- `./mvnw spotless:apply` on the affected modules — succeeded.
- `./mvnw install -DskipTests` (module + dependency chain,
`seatunnel-engine-ui` excluded since unmodified) — confirmed genuine
compilation via the actual `.class` files present under
`target/test-classes` for both the new test and the modified shared
sink classes (not run with `-Dmaven.test.skip=true`, which skips test
compilation entirely rather than just execution).
- Full test execution is left to CI per this repository's E2E
conventions (Hazelcast-cluster-backed, not run in this sandbox).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
