luoluoyuyu opened a new pull request, #19119: URL: https://github.com/apache/iotdb/pull/19119
## Motivation A pipe sink subtask can fail to start and still be reported as started, and it can never recover by itself: 1. `PipeSubtaskExecutor.start()` calls `allowSubmittingSelf()` before `submitSelf()`. When `submitSelf()` throws - for example because the worker pool has been shut down - the "submitting self" flag has already been set. Every later `start()` then takes the "already running" branch, logs only at debug level and returns normally, so `PipeSinkSubtaskLifeCycle.start()` increments `runningTaskCount` and the config node sees a successful start while the sink never transfers a single event. 2. `PipeAbstractSinkSubtask.submitSelf()` sets `isSubmitted` before scheduling the subtask on the worker. When the schedule is rejected, nothing is dispatched but the sign stays set, so the subtask can not be submitted again. 3. `IoTConsensusV2SubtaskExecutor` is a node-wide singleton shared by all the consensus pipes. `PipeSubtaskExecutorManager.getConsensusExecutor()` handed it out without checking whether it had been shut down, so after a shutdown every consensus pipe registered later was bound to a dead pool and could never start its sink. 4. `PipeTaskAgent.startPipeTask()` let the exception of `PipeTask.start()` escape without rolling back the task it had just created. The task meta was already registered, so the following meta sync found an existing task meta with an unchanged leader and never called `start()` again: the pipe stayed `RUNNING` in the metadata while running nothing until the node restarted. These were observed together on a 2.0.11 cluster: the shared consensus executor was shut down at 20:14, 384 consensus pipes were re-created at 20:16, every sink submission was rejected, and the pipes stayed "running" without transferring any event until the node was restarted. ## Modification - `PipeSubtaskExecutor.start()`: roll back the "submitting self" state when `submitSelf()` fails, log the failure at error level and rethrow it, so the failure is visible to the caller and the subtask can be started again instead of being silently marked as running. - `PipeAbstractSinkSubtask`: clear `isSubmitted` in `onDisallowSubmittingSelf()`, so a submission that was never dispatched to the worker pool can be submitted again. - `PipeSubtaskExecutorManager.getConsensusExecutor()`: recreate the executor and install it back into `IoTV2GlobalComponentContainer` when the current one has been shut down. - `PipeTaskAgent.startPipeTask()`: drop the failed task together with its task meta, so that the next meta sync creates and starts it again, and log the failure instead of carrying it away. ## Tests - `PipeSubtaskExecutorTest#testStartWithRejectedSubmission`: a rejected submission rolls back the submitting state and still fails. - `PipeSinkSubtaskExecutorTest#testStartAfterShutdown`: starting a sink subtask on a shut-down executor throws and does not leave the subtask in the submitting state. - `PipeSubtaskExecutorManagerTest`: the consensus executor is a singleton while it is alive and is recreated after it has been shut down. - `PipeDataNodeTaskAgentTest#testFailedTaskStartIsRolledBackForRetry`: a failed task start drops the task and its task meta. The pipe agent test set (49 tests) passes on the 2.0.11 branch this was developed on, together with the default and `-P with-zh-locale` builds. The datanode module could not be compiled locally against current `master` because the Thrift compiler is not available in this environment, so its compilation is left to CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
