luoluoyuyu opened a new pull request, #19119:
URL: https://github.com/apache/iotdb/pull/19119

   ## Motivation
   
   A pipe sink subtask can fail to start and still be reported as started, and 
it can never recover by itself:
   
   1. `PipeSubtaskExecutor.start()` calls `allowSubmittingSelf()` before 
`submitSelf()`. When `submitSelf()` throws - for example because the worker 
pool has been shut down - the "submitting self" flag has already been set. 
Every later `start()` then takes the "already running" branch, logs only at 
debug level and returns normally, so `PipeSinkSubtaskLifeCycle.start()` 
increments `runningTaskCount` and the config node sees a successful start while 
the sink never transfers a single event.
   
   2. `PipeAbstractSinkSubtask.submitSelf()` sets `isSubmitted` before 
scheduling the subtask on the worker. When the schedule is rejected, nothing is 
dispatched but the sign stays set, so the subtask can not be submitted again.
   
   3. `IoTConsensusV2SubtaskExecutor` is a node-wide singleton shared by all 
the consensus pipes. `PipeSubtaskExecutorManager.getConsensusExecutor()` handed 
it out without checking whether it had been shut down, so after a shutdown 
every consensus pipe registered later was bound to a dead pool and could never 
start its sink.
   
   4. `PipeTaskAgent.startPipeTask()` let the exception of `PipeTask.start()` 
escape without rolling back the task it had just created. The task meta was 
already registered, so the following meta sync found an existing task meta with 
an unchanged leader and never called `start()` again: the pipe stayed `RUNNING` 
in the metadata while running nothing until the node restarted.
   
   These were observed together on a 2.0.11 cluster: the shared consensus 
executor was shut down at 20:14, 384 consensus pipes were re-created at 20:16, 
every sink submission was rejected, and the pipes stayed "running" without 
transferring any event until the node was restarted.
   
   ## Modification
   
   - `PipeSubtaskExecutor.start()`: roll back the "submitting self" state when 
`submitSelf()` fails, log the failure at error level and rethrow it, so the 
failure is visible to the caller and the subtask can be started again instead 
of being silently marked as running.
   - `PipeAbstractSinkSubtask`: clear `isSubmitted` in 
`onDisallowSubmittingSelf()`, so a submission that was never dispatched to the 
worker pool can be submitted again.
   - `PipeSubtaskExecutorManager.getConsensusExecutor()`: recreate the executor 
and install it back into `IoTV2GlobalComponentContainer` when the current one 
has been shut down.
   - `PipeTaskAgent.startPipeTask()`: drop the failed task together with its 
task meta, so that the next meta sync creates and starts it again, and log the 
failure instead of carrying it away.
   
   ## Tests
   
   - `PipeSubtaskExecutorTest#testStartWithRejectedSubmission`: a rejected 
submission rolls back the submitting state and still fails.
   - `PipeSinkSubtaskExecutorTest#testStartAfterShutdown`: starting a sink 
subtask on a shut-down executor throws and does not leave the subtask in the 
submitting state.
   - `PipeSubtaskExecutorManagerTest`: the consensus executor is a singleton 
while it is alive and is recreated after it has been shut down.
   - `PipeDataNodeTaskAgentTest#testFailedTaskStartIsRolledBackForRetry`: a 
failed task start drops the task and its task meta.
   
   The pipe agent test set (49 tests) passes on the 2.0.11 branch this was 
developed on, together with the default and `-P with-zh-locale` builds. The 
datanode module could not be compiled locally against current `master` because 
the Thrift compiler is not available in this environment, so its compilation is 
left to CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to