luoluoyuyu opened a new pull request, #19140:
URL: https://github.com/apache/iotdb/pull/19140

   ## Description
   
   Fixes the DataNode thread leak where, after databases are created and 
dropped, DataNode threads — mainly the `Pipe-Processor-Executor-Pool-*` threads 
— are never reclaimed and the thread count stays at its peak until the DataNode 
exits.
   
   ### Root cause
   
   `PipeProcessorSubtaskWorker.runMayThrow()` used to run an unbounded loop:
   
   ```java
   while (true) {
     if (!subtasks.isEmpty()) {
       runSubtasks();
     }
     sleepIfNecessary(...);
   }
   ```
   
   A worker thread therefore never returned to the pool, so the processor 
executor's `allowCoreThreadTimeOut(true)` with a 60-second keep-alive could 
never take effect. Every worker that had ever been started kept its thread 
forever, even after all of its subtasks were closed/deregistered when the 
corresponding database was dropped.
   
   ### Design
   
   Workers are now started on demand and exit when they have nothing left to do:
   
   1. **Per-worker lifecycle state.** Each `PipeProcessorSubtaskWorker` records 
its own `workerThread` and an `isRunning` flag. 
`PipeProcessorSubtaskWorkerManager` still pre-creates the worker objects in its 
`workers[]` array, but no thread is submitted until a subtask is actually 
assigned.
   2. **Start on demand.** `schedule()` sets the wake-up callback, adds the 
subtask and calls `start()`. `start()` submits the worker to the thread pool 
only if the worker is not already running and has at least one subtask.
   3. **Exit when empty.** Each loop iteration first drops closed subtasks. If 
the set is empty, `stopIfNoSubtask()` re-checks emptiness under the lifecycle 
monitor, marks the worker as not running and lets `runMayThrow()` return. The 
pool thread then becomes idle and is reclaimed by the existing core-thread 
timeout.
   4. **Restart on demand.** A later `schedule()` — or a wake-up from 
`PipeProcessorSubtaskExecutionGuard` — starts a fresh worker thread for the 
same worker object.
   5. **Executor shutdown.** The loop and `start()` also observe 
`workerThreadPoolExecutor.isShutdown()` and stop without resubmitting, which 
also fixes the thread leak that unit tests and agent restart paths could 
observe.
   6. **Manager recreation.** `PipeProcessorSubtaskWorkerManager` is a 
process-wide singleton bound to the global processor executor. 
`PipeProcessorSubtask.bindExecutors()` now rebuilds it when the cached manager 
is bound to a shutdown pool, so subtasks are never scheduled onto a dead 
executor.
   
   ### Concurrency
   
   The start/exit race is handled by a dedicated `lifecycleMonitor` in the 
worker:
   
   - The emptiness check, the `isRunning` transition and the decision to submit 
all happen atomically. A subtask scheduled concurrently with the exit cannot be 
lost: either the exiting worker sees the new subtask and keeps running, or the 
scheduler sees `isRunning == false` and starts a new thread.
   - Only one thread can execute `runMayThrow()` for a given worker at a time, 
so a worker never runs twice concurrently.
   - An `exitedNormally` flag distinguishes a normal exit from an abnormal one: 
on an unexpected exception the `finally` block resets the state and restarts 
the worker if it still has subtasks, while a normal exit does not clobber a 
newer run.
   - `workerThread` is cleared in `finally` only when it still points at the 
exiting thread.
   
   A paused/stopped pipe keeps its subtask registered, so its worker keeps 
running (waking up once per second); the thread is released when the pipe task 
is deregistered or closed, which is the lifecycle described by the issue.
   
   ### Tests
   
   New/updated unit tests:
   
   - `PipeProcessorSubtaskWorkerTest`
     - `testWorkerThreadIsReleasedWhenIdleAndStartedAgainOnDemand`: uses a real 
pool with a 100 ms core-thread keep-alive and asserts `poolSize` 0 → 1 after a 
subtask is scheduled, 1 → 0 after the subtask is closed, and that scheduling 
another subtask creates a new thread.
     - `testConcurrentSchedulesSubmitAtMostOneWorker`: 16 threads concurrently 
schedule subtasks to a worker backed by a recording executor and exactly one 
submission is made; after the worker exits, 16 threads schedule again and 
exactly one new submission is made.
     - `testWorkerManagerSchedulesWatcher`: asserts that constructing the 
manager no longer submits worker threads; the first submission happens only 
when a subtask is scheduled.
   - 
`PipeProcessorSubtaskExecutorTest#testWorkerThreadIsReleasedAfterAllSubtasksAreDeregistered`:
 registers a subtask through the public executor API, waits for `poolSize == 
1`, deregisters it and waits for `poolSize == 0` (1 s keep-alive).
   - Existing worker/guard tests were adapted to the new constructor that takes 
the owning executor.
   
   Verification performed locally:
   
   - `PipeProcessorSubtaskWorkerTest` (10), `PipeProcessorSubtaskExecutorTest` 
(7), `PipeProcessorSubtaskExecutionGuardTest` (5), `PipeProcessorSubtaskTest` 
(2), `PipeSubtaskExecutorManagerTest` (2) — all pass.
   - Full-reactor `mvn test-compile -DskipTests` and `mvn test-compile 
-DskipTests -P with-zh-locale` — both `BUILD SUCCESS`.
   - `mvn spotless:apply` applied.
   
   ### Scope
   
   - No public API or configuration change. The only API addition is a 
test-only constructor parameter for the worker keep-alive time and a test-only 
pool-size accessor.
   - The issue also mentions SchemaRegion Ratis gRPC threads. Those are created 
by the node-level `SchemaRegionConsensusImpl` singleton (one 
`RatisConsensus`/`RaftServer` per DataNode), so they do not grow per database 
and are intentionally not changed here. If they still grow in a reproduction, 
they need a separate investigation with a fresh thread dump.
   - Manual verification: run `jstack <pid> | grep -c 
'Pipe-Processor-Executor-Pool'` before and after `DROP DATABASE`; the count 
should drop back to 0 (plus the one long-lived scheduler thread) once the pipe 
tasks are deregistered.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods.
   - [ ] added or updated version, __license__, or notice information
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold for code coverage.
   - [ ] added integration tests.
   - [ ] been tested in a test IoTDB cluster.
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   - `PipeProcessorSubtaskWorker`: on-demand worker lifecycle, lifecycle 
monitor, restart on demand.
   - `PipeProcessorSubtaskWorkerManager`: no eager submission, shutdown 
observation, test accessors.
   - `PipeProcessorSubtask`: recreate the cached manager when its executor has 
been shut down.
   - `PipeSubtaskExecutor` / `PipeProcessorSubtaskExecutor`: testable 
keep-alive constructor, `getWorkerThreadPoolSize()`.
   - `DataNodePipeMessages` (en/zh): log constant for rejected worker 
submission.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to