sql-wj opened a new issue, #18697: URL: https://github.com/apache/dolphinscheduler/issues/18697
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened Describe the bug `DinkyTask.submitApplicationV0()` and `submitApplicationV1()` call `Thread.currentThread().interrupt()` inside a `catch (Exception)` block. `DinkyTask` executes on the worker's `TaskExecutorWorker` thread, so when a Dinky submit fails, the interrupt flag of that worker thread gets set. On the worker's next `Condition.await()`, the JDK contract makes `await()` throw `InterruptedException` immediately (the flag is already set), the worker hits `break`, exits its `while (true)` loop and terminates. The fixed-size thread pool submits each worker only once and never re-submits, so that slot is lost permanently. Once all 100 slots are exhausted, the worker rejects every new task with `All ExclusiveThreadTaskExecutorWorker are busy`. ## Which version of DolphinScheduler 3.4.0, 3.4.1, 3.4.2, 3.4.3 — all affected. Confirmed present in the `3.4.3` tag. ## To Reproduce 1. Deploy any of the affected versions and configure a Dinky task. 2. Make the Dinky submit call fail (e.g. Dinky service unavailable, or the "No such file or directory" error while writing the temp SQL file). 3. Observe the worker log: `TaskExecutorWorker(id=X) is interrupted`. 4. Repeat until the worker log shows `All ExclusiveThreadTaskExecutorWorker are busy`; from then on no new task is accepted. ## Root cause `Thread.currentThread().interrupt()` is meant to restore the interrupt flag inside `catch (InterruptedException)`. Here it sits in `catch (Exception)`, and the try block only performs HTTP calls and JSON parsing — it throws `IOException` / runtime exceptions, never `InterruptedException`. So the flag is set on a thread that was never actually interrupted. Affected code in `dolphinscheduler-task-plugin/dolphinscheduler-task-dinky/src/main/java/org/apache/dolphinscheduler/plugin/task/dinky/DinkyTask.java`: ```java // submitApplicationV0() — line 138 } catch (Exception ex) { Thread.currentThread().interrupt(); // <-- should be removed log.error(DinkyTaskConstants.SUBMIT_FAILED_MSG, ex); setExitStatusCode(EXIT_CODE_FAILURE); throw new TaskException(DinkyTaskConstants.SUBMIT_FAILED_MSG, ex); } // submitApplicationV1() — line 169 (same mistake, copied from V0) } catch (Exception ex) { Thread.currentThread().interrupt(); // <-- should be removed ... } ``` For contrast, the correct pattern is already used in `trackApplicationStatusV0/V1`, where `interrupt()` is inside `catch (InterruptedException)`. ## Failure chain (verified against source) 1. Dinky submit fails → `catch (Exception)` sets the interrupt flag. 2. `TaskException` is thrown → worker's inner `catch (Throwable)` → `onTaskExecutorFailed`. 3. Worker loop reaches `activeTaskExecutorEmptyCondition.await()`. Because the interrupt flag is already set, `await()` throws `InterruptedException` immediately (JDK contract). 4. `TaskExecutorWorker` catches it and `break`s, exiting the `while (true)` loop and terminating the thread. 5. `AbstractTaskExecutorContainer.startAllThreadTaskExecutorWorker()` submits each worker only once; there is no re-submit logic → the slot is permanently lost. 6. After 100 such losses, `selectIdleWorker()` always returns empty → `All ExclusiveThreadTaskExecutorWorker are busy`. Secondary effect: `selectIdleWorker()` determines idleness only by `registeredTaskExecutorSize == 0` and does not check thread liveness, so a dead worker (thread terminated, `registered == 0`) can be selected again. The task dispatched to it is registered but never executed and never finalized — silently stuck in `DISPATCH` (no error, no retry, no alert). ## Runtime evidence - `TaskExecutorWorker(id=X) is interrupted` — 42 occurrences within ~48 minutes. - 543 dispatched vs 501 finalized — 42 orphan tasks, matching the 42 killed workers. - `All ExclusiveThreadTaskExecutorWorker are busy` — 3300+ occurrences over ~9 hours. - The interrupted worker ids included 97/98/99; `selectIdleWorker()` picks the lowest-numbered idle worker first, so reaching ids 97–99 proves the entire 100-slot pool had actually been exhausted. ## Proposed fix Delete the two `Thread.currentThread().interrupt();` lines: ```java } catch (Exception ex) { // Thread.currentThread().interrupt(); // removed log.error(DinkyTaskConstants.SUBMIT_FAILED_MSG, ex); setExitStatusCode(EXIT_CODE_FAILURE); throw new TaskException(DinkyTaskConstants.SUBMIT_FAILED_MSG, ex); } ``` - Failure handling is unaffected: the failure is still propagated by `TaskException` + `setExitStatusCode(EXIT_CODE_FAILURE)`. - Legitimate interrupts (`shutdownNow()` / task kill) are set by the external caller and never go through these lines, so normal stop/kill is unaffected. - `trackApplicationStatusV0/V1` must stay unchanged. ## Additional context This is the same interrupt-flag misuse pattern as previously reported in issue #6112. ### What you expected to happen When a Dinky task submit fails, only that task should be marked FAILED. The worker thread hosting it should keep running and continue accepting subsequent tasks. The worker slot should not be leaked, and the worker should never end up rejecting all new tasks with "All ExclusiveThreadTaskExecutorWorker are busy". DinkyTask.submitApplicationV0() and submitApplicationV1() call Thread.currentThread().interrupt() inside a catch (Exception) block. That try block only does HTTP calls / JSON parsing and throws IOException, never InterruptedException, so this spuriously sets the interrupt flag on the worker thread (which was never actually interrupted). On the worker's next Condition.await(), the already-set flag makes await() throw InterruptedException immediately (JDK contract), so the worker hits break, exits its while(true) loop and dies. The fixed-size thread pool only submits each worker once and never re-submits, so the slot is permanently lost. After 100 such losses the worker rejects all new tasks. ### How to reproduce 1. Deploy any affected version (3.4.0 – 3.4.3) and configure a Dinky task. 2. Make the Dinky submit call fail. Two ways to do this: - stop the Dinky service so the HTTP submit call fails; or - trigger the "No such file or directory" error that occurs while the task writes its temporary SQL file. 3. Check the worker log: it shows "TaskExecutorWorker(id=X) is interrupted" for the worker thread that hosted the failed task. 4. Repeat step 2. Each failure kills one more worker thread. Once all 100 slots are gone, the worker log floods with "All ExclusiveThreadTaskExecutorWorker are busy" and the worker stops accepting any new task. ### Anything else _No response_ ### Version 3.4.3 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
