DanielLeens opened a new pull request, #13103:
URL: https://github.com/apache/seatunnel/pull/13103
Yes, contributing to SeaTunnel.
### Purpose
Adds an E2E regression guard for #11472 ("finished batch job reports
inflated metrics after a worker failover"), which was fixed by #11495
("[Fix][Zeta] Reset metrics after pipeline recovery"). Test-only change; no
production code is touched.
### Why existing tests miss it
- `StateTransitionCleanupTest` (added in #11495) covers the fix with a
mocked `JobMaster` and a reflective call to `SubPlan#prepareRestorePipeline`.
It never runs a real restored job.
- The worker-down tests in `ClusterFaultToleranceIT` check the sink output
and only log `getJobMetrics`. A restored job that over-counts still passes them.
- Nothing asserts the metrics a real job reports after a real pipeline
restore, so a regression of #11495 would go unnoticed.
### What the test does
`ClusterFaultToleranceIT#testBatchJobMetricsResetAfterWorkerDownRestore`:
1. Starts two nodes with static slots and the `SLOT_RATIO` strategy, then
waits until both workers are registered, following the
`SlotRatioAllocateStrategyIT` pattern. Task groups are spread evenly, so the
master node hosts at least one reader and the worker node hosts at least one
task group.
2. Runs a batch FakeSource -> LocalFile job, 4 readers x 1000 rows, from a
new template that differs from `cluster_batch_fake_to_localfile_template.conf`
only in `checkpoint.interval = 600000`. No periodic checkpoint can complete, so
the restored attempt reads every row again from the start.
3. Waits until all 4 readers report `SourceReceivedCount > 0` through the
real metrics API, then shuts down the worker node.
4. After `FINISHED`, checks that the pipeline was actually restored
(`SubPlan#getPipelineRestoreNum() >= 1`). It then waits for the coordinator to
release the job, so the metrics come only from the finished-job history.
5. Asserts `SourceReceivedCount == 4000` and `SinkWriteCount == 4000`. If
`IntermediateQueueSize` is reported, every entry must be 0. The sink output is
also checked with the existing `assertOutputRecoveredAndStable`.
### How it fails on pre-fix code
Task locations stay the same across restores, and a new task seeds its
metrics context from the master's metrics store. Before #11495, nothing cleared
that store before the restore. The canceled reader on the surviving node
reports its non-zero counts to the store when its task group finishes
(`TaskExecutionService` `taskDone` -> `updateMetricsContextInImap`), and the
restored reader continues counting from that value. So `SourceReceivedCount`
comes out at least 4000 plus that reader's pre-failure rows, and the
exact-equality assertion fails.
Exact equality is safe after the fix. No checkpoint can complete, so the
restored attempt counts exactly 4000. The one stale case that #11495 documents
is a periodic worker report, already in flight, that lands within a
millisecond-scale window against a 10 s report interval. A stale report carries
the same counts as the pre-fix bug, so no weaker bound could tell the two
apart. That leaves equality as the strongest assertion available.
### Verification
Spotless only locally; GitHub CI is the verification.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
