SEPURI-SAI-KRISHNA opened a new pull request, #13209:
URL: https://github.com/apache/seatunnel/pull/13209

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Closes #13208.
   
   `CheckpointBarrierTriggerOperation` retried for up to 60 s when the target 
task group was missing and its state was not an end state. A pipeline restore 
resets the state to `CREATED` and deploys a new task group at the same 
`TaskGroupLocation`, so a barrier that missed the failed task group was 
triggered on the restored one. With the final barrier (`Long.MAX_VALUE`), the 
restored reader stopped before reading, the restored coordinator skipped every 
acknowledgement, and a batch job stayed `RUNNING` forever. The issue has the 
logs and the measurements.
   
   The operation now looks the task group up once and fails with 
`TaskGroupContextNotFoundException` when it is not there. That is safe because 
no task starts before every task of its pipeline has reported `READY_START` and 
the coordinator has called `notifyTaskStart`, and a deployment publishes its 
context before it starts its tasks. So when a barrier exists, its target task 
group is registered, and a missing one has ended. The coordinator does not act 
on the RPC result (and #12314, if merged, ignores a failure whose pending 
checkpoint is no longer current).
   
   Only this operation changes. `BarrierFlowOperation` and the other task 
operations keep their retry: their senders wait with `join()`, so the sending 
task group cannot end, and the pipeline cannot restore, while they are pending. 
The issue lists the operations that could cross a restore in theory.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. A batch job that failed and restored at the moment its source finished 
could stay `RUNNING` forever; it now restores normally and ends `FINISHED` or 
`FAILED`. No option, default, API or state format changes. On the worker, a 
dropped stale barrier is logged at ERROR by Hazelcast, as on `dev` when the 
task group has ended.
   
   ### How was this patch tested?
   
   - 
`TaskExecutionServiceTest.testBarrierForAMissingTaskGroupDoesNotReachTheNextDeployment`:
 the state is `CREATED`, the barrier is sent for a task group that is not 
deployed, and a task group is deployed at that location 1 s later. The barrier 
must not reach it, and the operation must fail with 
`TaskGroupContextNotFoundException`. On `dev` the barrier reaches the new task 
group (`expected: <0> but was: <1>`).
   - `testBarrierReachesARunningTaskGroup`: a barrier for a running task group 
is still delivered.
   - `TaskExecutionServiceTest` 22 of 22 with the fix, 21 of 22 without it 
(only the new stale test fails). 13 engine-server test classes, 83 tests, pass, 
including `CheckpointCoordinatorTest`, `SavePointTest`, 
`SavePointBusySourceTest`, `CheckpointErrorRestoreEndTest`, 
`CheckpointTimeOutTest`, `CheckpointBarrierTriggerErrorTest` and 
`JobMasterMasterFailoverResourceTest`. `spotless:check` passes.
   - Whole jobs in process: with the final barrier held 1.5 s on the worker and 
a sink that fails 500 ms after its first write, `dev` delivered the barrier to 
a newer deployment in 18 of 18 attempts and 1 of 9 jobs hung; with this change 
6 of 6 jobs ended `FAILED` after 3 restores with no stale acknowledgements. 
Normal batch jobs (in-memory, 5 splits with parallelism 2, two tables) still 
finish, and failing batch and streaming jobs still fail after 3 restores.
   - No E2E is added: the race needs the barrier to miss its task group within 
a few milliseconds of a failure, which an E2E cannot force without a timing 
hook. The unit test reproduces the mechanism deterministically.
   
   Fork run: 
https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42588706253 
passes, apart from `connector-file-sftp-it`. That job fails because the 
`atmoz/sftp:alpine-3.7` image is gone from Docker Hub (#13202, fixed by 
#13203); it is unrelated to this change and fails on `dev` too.
   
   ### Check list
   
   * [X] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
 (no new jar)
   * [X] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs (not needed: no 
option or behavior a user configures)
   * [X] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR (not needed: no incompatible change)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to