luoluoyuyu opened a new pull request, #19140:
URL: https://github.com/apache/iotdb/pull/19140
## Description
Fixes the DataNode thread leak where, after databases are created and
dropped, DataNode threads — mainly the `Pipe-Processor-Executor-Pool-*` threads
— are never reclaimed and the thread count stays at its peak until the DataNode
exits.
### Root cause
`PipeProcessorSubtaskWorker.runMayThrow()` used to run an unbounded loop:
```java
while (true) {
if (!subtasks.isEmpty()) {
runSubtasks();
}
sleepIfNecessary(...);
}
```
A worker thread therefore never returned to the pool, so the processor
executor's `allowCoreThreadTimeOut(true)` with a 60-second keep-alive could
never take effect. Every worker that had ever been started kept its thread
forever, even after all of its subtasks were closed/deregistered when the
corresponding database was dropped.
### Design
Workers are now started on demand and exit when they have nothing left to do:
1. **Per-worker lifecycle state.** Each `PipeProcessorSubtaskWorker` records
its own `workerThread` and an `isRunning` flag.
`PipeProcessorSubtaskWorkerManager` still pre-creates the worker objects in its
`workers[]` array, but no thread is submitted until a subtask is actually
assigned.
2. **Start on demand.** `schedule()` sets the wake-up callback, adds the
subtask and calls `start()`. `start()` submits the worker to the thread pool
only if the worker is not already running and has at least one subtask.
3. **Exit when empty.** Each loop iteration first drops closed subtasks. If
the set is empty, `stopIfNoSubtask()` re-checks emptiness under the lifecycle
monitor, marks the worker as not running and lets `runMayThrow()` return. The
pool thread then becomes idle and is reclaimed by the existing core-thread
timeout.
4. **Restart on demand.** A later `schedule()` — or a wake-up from
`PipeProcessorSubtaskExecutionGuard` — starts a fresh worker thread for the
same worker object.
5. **Executor shutdown.** The loop and `start()` also observe
`workerThreadPoolExecutor.isShutdown()` and stop without resubmitting, which
also fixes the thread leak that unit tests and agent restart paths could
observe.
6. **Manager recreation.** `PipeProcessorSubtaskWorkerManager` is a
process-wide singleton bound to the global processor executor.
`PipeProcessorSubtask.bindExecutors()` now rebuilds it when the cached manager
is bound to a shutdown pool, so subtasks are never scheduled onto a dead
executor.
### Concurrency
The start/exit race is handled by a dedicated `lifecycleMonitor` in the
worker:
- The emptiness check, the `isRunning` transition and the decision to submit
all happen atomically. A subtask scheduled concurrently with the exit cannot be
lost: either the exiting worker sees the new subtask and keeps running, or the
scheduler sees `isRunning == false` and starts a new thread.
- Only one thread can execute `runMayThrow()` for a given worker at a time,
so a worker never runs twice concurrently.
- An `exitedNormally` flag distinguishes a normal exit from an abnormal one:
on an unexpected exception the `finally` block resets the state and restarts
the worker if it still has subtasks, while a normal exit does not clobber a
newer run.
- `workerThread` is cleared in `finally` only when it still points at the
exiting thread.
A paused/stopped pipe keeps its subtask registered, so its worker keeps
running (waking up once per second); the thread is released when the pipe task
is deregistered or closed, which is the lifecycle described by the issue.
### Tests
New/updated unit tests:
- `PipeProcessorSubtaskWorkerTest`
- `testWorkerThreadIsReleasedWhenIdleAndStartedAgainOnDemand`: uses a real
pool with a 100 ms core-thread keep-alive and asserts `poolSize` 0 → 1 after a
subtask is scheduled, 1 → 0 after the subtask is closed, and that scheduling
another subtask creates a new thread.
- `testConcurrentSchedulesSubmitAtMostOneWorker`: 16 threads concurrently
schedule subtasks to a worker backed by a recording executor and exactly one
submission is made; after the worker exits, 16 threads schedule again and
exactly one new submission is made.
- `testWorkerManagerSchedulesWatcher`: asserts that constructing the
manager no longer submits worker threads; the first submission happens only
when a subtask is scheduled.
-
`PipeProcessorSubtaskExecutorTest#testWorkerThreadIsReleasedAfterAllSubtasksAreDeregistered`:
registers a subtask through the public executor API, waits for `poolSize ==
1`, deregisters it and waits for `poolSize == 0` (1 s keep-alive).
- Existing worker/guard tests were adapted to the new constructor that takes
the owning executor.
Verification performed locally:
- `PipeProcessorSubtaskWorkerTest` (10), `PipeProcessorSubtaskExecutorTest`
(7), `PipeProcessorSubtaskExecutionGuardTest` (5), `PipeProcessorSubtaskTest`
(2), `PipeSubtaskExecutorManagerTest` (2) — all pass.
- Full-reactor `mvn test-compile -DskipTests` and `mvn test-compile
-DskipTests -P with-zh-locale` — both `BUILD SUCCESS`.
- `mvn spotless:apply` applied.
### Scope
- No public API or configuration change. The only API addition is a
test-only constructor parameter for the worker keep-alive time and a test-only
pool-size accessor.
- The issue also mentions SchemaRegion Ratis gRPC threads. Those are created
by the node-level `SchemaRegionConsensusImpl` singleton (one
`RatisConsensus`/`RaftServer` per DataNode), so they do not grow per database
and are intentionally not changed here. If they still grow in a reproduction,
they need a separate investigation with a fresh thread dump.
- Manual verification: run `jstack <pid> | grep -c
'Pipe-Processor-Executor-Pool'` before and after `DROP DATABASE`; the count
should drop back to 0 (plus the one long-lived scheduler thread) once the pipe
tasks are deregistered.
<hr>
This PR has:
- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods.
- [ ] added or updated version, __license__, or notice information
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for code coverage.
- [ ] added integration tests.
- [ ] been tested in a test IoTDB cluster.
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `PipeProcessorSubtaskWorker`: on-demand worker lifecycle, lifecycle
monitor, restart on demand.
- `PipeProcessorSubtaskWorkerManager`: no eager submission, shutdown
observation, test accessors.
- `PipeProcessorSubtask`: recreate the cached manager when its executor has
been shut down.
- `PipeSubtaskExecutor` / `PipeProcessorSubtaskExecutor`: testable
keep-alive constructor, `getWorkerThreadPoolSize()`.
- `DataNodePipeMessages` (en/zh): log constant for rejected worker
submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]