SEPURI-SAI-KRISHNA opened a new issue, #13208:
URL: https://github.com/apache/seatunnel/issues/13208

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22)
 and found no similar issues. #10997 has the same symptom, but no task failed 
or restored there. No open PR changes `CheckpointBarrierTriggerOperation`. 
#12314 changes how the coordinator handles a failed barrier RPC; it does not 
cover this case, because here the RPC succeeds.
   
   ### What happened
   
   On Zeta, a batch job whose pipeline restores after a task failure can stay 
`RUNNING` forever. Nothing fails, and nothing is written after the restore.
   
   The cause is a checkpoint barrier from the failed attempt that is delivered 
to the restored attempt.
   
   `CheckpointBarrierTriggerOperation` sends a barrier from the checkpoint 
coordinator to the source enumerator. On the worker it looks up the target task 
group. When it is not there, it retries every 2 s, up to 30 times, while 
`taskIsEnded` is false:
   
   ```java
   exception -> exception instanceof TaskGroupContextNotFoundException
           && !server.taskIsEnded(taskLocation.getTaskGroupLocation())
   ```
   
   A restore resets the task group state to `CREATED`, which is not an end 
state, and deploys a new task group at the same `TaskGroupLocation`. So a 
barrier that misses the failed task group keeps retrying, and reaches the new 
one:
   
   1. The source finishes, and the coordinator triggers the final barrier 
(`Long.MAX_VALUE`, `COMPLETED_POINT_TYPE`).
   2. The sink fails at the same moment. The barrier finds no task group, and 
the pipeline state is reset to `CREATED`, so the operation retries.
   3. 3 s later the restored task group is deployed. The next retry finds it 
and triggers the old final barrier on it.
   4. The restored tasks take it as the final barrier, and the reader stops 
before it reads anything. The restored coordinator has no such pending 
checkpoint, so it drops every acknowledgement (`skip already ack checkpoint id: 
9223372036854775807`). The final checkpoint never completes, and the job never 
ends.
   
   Whether the job hangs depends on timing. If the stale barrier arrives after 
the restored reader has already read its rows, the job goes on, and only the 
barrier is wrong.
   
   ### How I checked it
   
   The first run was the Docker E2E on dev plus a GraphQL connector change; all 
others were on dev `b7ef4936fe`, in process with `AbstractSeaTunnelServerTest`, 
JDK 17.
   
   - **Found in a real job.** While changing `GraphQLIT`, a FakeSource (two 
tables) to GraphQL sink job on Zeta failed every row, because the server 
rejected the mutation. After the first restore the job stayed `RUNNING` for 
over an hour. The coordinator triggered the final barrier at 15:17:58.871, 
while the sink task group was failing. Exactly 4 s later (two retries), at 
15:18:02.887, three acknowledgements for checkpoint `9223372036854775807` 
arrived from the restored tasks, and were skipped. The restored reader never 
produced a row.
   - **Again in CI, with no test hook.** In the fork run 
https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42597013855 (job 
`updated-modules-integration-test-part-1 (11, ubuntu-latest)`, the same GraphQL 
change), the GraphQL source job `graphql_rejected_query_to_assert.conf` on Zeta 
failed and was restored 3 times within 10 s. 3 s after the third restore, at 
10:26:14.260, three acknowledgements for checkpoint `9223372036854775807` were 
skipped. The job then stayed `RUNNING` for 5 hours, until the CI job was 
cancelled at its 300 minute limit.
   - **Stock connectors.** FakeSource (two tables, one row each) to the 
`InMemory` test sink with `throw_runtime_exception_list`: 1 of 4 jobs hung with 
the same log lines, and the other 3 failed after 3 restores. The race is 
narrow; in later runs 0 of 56 hung.
   - **The operation on its own, deterministic.** I set the task group state to 
`CREATED`, sent the operation for a task group that is not deployed, and 
deployed a task group at that location 1 s later. The new task group received 
the barrier, 3 of 3. A barrier for a running task group is delivered as normal.
   - **A whole job with the window widened.** A test sink fails 500 ms after 
its first write, and the final barrier is held for 1.5 s on the worker before 
it is handled, as if the RPC were slow. In 18 of 18 attempts the first try 
found no task group in state `CREATED`, and every one of them was then 
delivered to a newer deployment. 1 of 9 jobs hung.
   
   ### Proposed fix
   
   Do not retry a missing task group in `CheckpointBarrierTriggerOperation`. A 
deployment publishes its task group context before it starts its tasks, and 
every task (enumerator, reader and sink, committer) then waits in `READY_START` 
until the coordinator has seen `READY_START` from every task of the pipeline 
and calls `notifyTaskStart`. So by the time any barrier exists, every task 
group of that deployment is registered on its worker. A missing task group 
means that deployment has ended, and its barrier can be dropped. The 
coordinator does not act on these RPC results today, and with #12314 it ignores 
a failure whose pending checkpoint is no longer current. On the worker, 
Hazelcast logs the dropped operation at ERROR with its stack trace, as it 
already does on `dev` when the task group has ended; I can turn that into one 
INFO line if you prefer.
   
   With that change in the same setups:
   - the deterministic case drops the stale barrier, with 
`TaskGroupContextNotFoundException`, and still delivers to a running task group;
   - 6 of 6 widened-window jobs end `FAILED` after 3 restores, with no stale 
acknowledgements;
   - normal batch jobs still finish, and the failing jobs still fail after 3 
restores;
   - 83 tests in 13 engine-server test classes pass, and one more is disabled 
in the source: `CheckpointCoordinatorTest`, `SavePointTest`, 
`SavePointBusySourceTest`, `CheckpointErrorRestoreEndTest`, 
`CheckpointTimeOutTest`, `CheckpointBarrierTriggerErrorTest`, 
`TaskExecutionServiceTest`, `JobMasterMasterFailoverResourceTest` and others.
   
   A stricter alternative is to carry the target execution id in the operation 
and drop the barrier on a mismatch. That also covers an operation delayed past 
the restore, but the master would have to keep the execution id of each 
deployment, which it does not today, including after a master failover. I would 
start with the smaller change unless you prefer this one.
   
   Other task operations use the same retry (added in #3009). Most of them 
cannot cross a restore, because the sending task waits for them with `join()`: 
its task group cannot end while the operation is pending, and the restore waits 
for every task group to end. I checked this with `AssignSplitOperation`: 
holding it on the worker held up the restore until it returned. 
`BarrierFlowOperation` is sent the same way. The barrier trigger is different, 
because the coordinator does not wait for it. Two groups could cross in theory, 
and I have not seen either: the reader-to-enumerator operations 
(`SourceNoMoreElementOperation`, `SourceRegisterOperation`, 
`RequestSplitOperation`, `RestoredSplitOperation`), which are sent with an 
interruptible `get()`, and the other coordinator-to-task operations 
(`NotifyTaskStartOperation`, `NotifyTaskRestoreOperation`, 
`CheckpointFinishedOperation`, `CheckpointEndOperation`), which no task waits 
for. A stale `NotifyTaskStartOperation` would let a restored tas
 k start before its pipeline is ready, which the reasoning above relies on not 
happening. The execution id alternative would cover all of them. I would look 
at them separately.
   
   I would like to submit a PR with the fix and a unit test.
   
   ### SeaTunnel Version
   
   dev (b7ef4936fe)
   
   ### SeaTunnel Config
   
   ```conf
   env {
     parallelism = 1
     job.mode = "BATCH"
   }
   
   source {
     FakeSource {
       tables_configs = [
         {
           schema = { table = "t1", fields { id = int, name = string } }
           rows = [ { kind = INSERT, fields = [1, "a"] } ]
         },
         {
           schema = { table = "t2", fields { id = int, name = string } }
           rows = [ { kind = INSERT, fields = [2, "b"] } ]
         }
       ]
     }
   }
   
   sink {
     # Any sink whose write fails while the source finishes. In the test I used 
the
     # InMemory test sink with throw_runtime_exception_list.
     InMemory {
       throw_runtime_exception_list = ["e0", "e1", "e2", "e3", "e4"]
     }
   }
   ```
   
   ### Running Command
   
   ```shell
   # InMemory is a test connector, so I ran the job in process through
   # AbstractSeaTunnelServerTest#startJob and waited for the job result.
   ```
   
   ### Error Exception
   
   ```log
   CheckpointCoordinator - received restore CheckpointCoordinator with 
alreadyStarted: false
   SubPlan - Wait 3s and then restore the pipeline Job ...
   FakeSourceSplitEnumerator - Assigning splits to readers 0 [...]
   CheckpointCoordinator - skip already ack checkpoint id: 9223372036854775807
   CheckpointCoordinator - skip already ack checkpoint id: 9223372036854775807
   CheckpointCoordinator - skip already ack checkpoint id: 9223372036854775807
   (nothing more; the job stays RUNNING)
   ```
   
   ### Zeta or Flink or Spark Version
   
   Zeta
   
   ### Java or Scala Version
   
   JDK 17 (in process). The GraphQL run used the JDK 11 engine image.
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to