DanielLeens opened a new pull request, #13102:
URL: https://github.com/apache/seatunnel/pull/13102
Yes, contributing to SeaTunnel.
### Purpose of this pull request
Adds an E2E regression guard for #12385, which was fixed by #12387
("[Fix][Zeta] Restore jobs from their own latest checkpoint on master
failover").
After a master switch, `CheckpointManager` only loaded a checkpoint when
`restoreSourceJobId != null` (a regression from #11421). A normally submitted
job has no restore source, so the new master's `CheckpointCoordinator` started
with `pipelineState == null`, and the next pipeline restore re-read every
source from the beginning. For CDC sources this means a full re-snapshot.
#12387 only added mocked unit tests in `CheckpointManagerTest`, so no E2E test
protects this behavior.
### Why existing tests miss it
-
`CheckpointCoordinatorFailoverIT#testStreamJobContinuesAfterMasterFailover`
only checks that the checkpoint id keeps growing after failover. The id counter
lives in an IMap, so it keeps growing even with the bug.
- `ClusterFaultToleranceIT#testStreamJobRestoreIn2NodeMasterDown` only
asserts a lower bound
(`FaultToleranceFakeSourceAssertions.assertOutputRecoveredAndStable`). A full
replay produces more rows than expected, so it still passes that check.
- Because tasks are spread over both nodes, a master-down test can also pass
without redeploying the whole pipeline.
### What the test does
Adds
`ClusterFaultToleranceIT#testStreamJobRestoreFromOwnCheckpointAfterMasterDown`.
It reuses the existing `cluster_batch_fake_to_localfile_template.conf`:
STREAMING FakeSource (1000 rows per reader, parallelism 6, `split.num = 5`,
`split.read-interval = 3000`) to a transactional LocalFile sink, with a
checkpoint every 5 s.
1. Start node1 alone and submit the job, so every task runs on the master.
Then start node2.
2. Wait until at least 50% of the 6000 rows are committed and the checkpoint
overview shows a completed checkpoint, then shut down node1.
3. Wait for the first checkpoint that completes after node1 is gone. Such a
checkpoint can only come from the pipeline restored on node2. Then wait 40 more
seconds of checkpoints, which is more than enough time for a full replay to be
committed.
4. Assert the output has reached at least the total and stopped growing (the
existing helper), then assert `lines <= 6000 + 6 * 200` (one replayed split per
reader at most).
5. Cancel the job and assert it ends as CANCELED.
The test adds no new conf or files, and changes no existing assertions.
### How it fails on the pre-fix code
If #12387 is reverted (the `masterFailoverRestore` branch in
`CheckpointManager` and the `restart` argument in
`JobMaster#initCheckPointManager`), the restored coordinator has no state. The
pipeline then re-reads all 6000 rows, and the sink writes them into new
transactions (with no restored state, the file name gets a fresh `uuidPrefix`).
The final count becomes at least 3000 + 6000 = 9000, so the new upper-bound
assertion fails ("... which means the sources were replayed from the
beginning"). With the fix, the job resumes from the checkpoint and the count
stays at most 7200.
### Verification
Spotless only locally; GitHub CI is the verification.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
