Caideyipi opened a new pull request, #18549:
URL: https://github.com/apache/iotdb/pull/18549
## Description
### Problem
`PipeSinkSubtaskManager` reused sink resources according to sink attributes.
As a result, different pipes with identical sink parameters could share a sink
lifecycle, pending queue, executor, and connector. Stopping, deregistering, or
reporting a critical sink failure for one pipe could therefore affect another
pipe.
### Changes
- Key sink subtasks and their queues, executors, connectors, task IDs, and
metrics by pipe name, creation time, and sink attributes.
- Scope DataNode and ConfigNode sink critical-exception propagation to the
identified pipe instead of every pipe with equal sink parameters.
- Keep explicit pipe name and creation time authoritative, including when
task metadata is detached or concurrently removed.
- Retain the legacy single-argument manager methods as deprecated
compatibility APIs. They resolve only a unique matching pipe and fail with a
localized `PipeException` when the match is ambiguous.
- Avoid shutting down the shared IoTConsensus V2 executor when an individual
pipe is deregistered.
- Add pipe identity dimensions to sink metrics, including batch histograms.
### Compatibility and concurrency
There is no SQL or configuration change. Existing callers with a unique
legacy sink match continue to work. Concurrent task removal cannot redirect a
critical exception to another pipe, and an explicitly identified pipe is still
transitioned to `STOPPED` when its task map has already been removed.
### Verification
Passed before PR creation:
- Spotless for `iotdb-core/node-commons`, `iotdb-core/datanode`, and
`iotdb-core/confignode`.
- DataNode Checkstyle with zero violations.
- `git diff --check`.
- All 16 `PipeHeartbeatParserTest` tests.
The focused DataNode tests were blocked before test execution by a
pre-existing, unrelated compilation mismatch at `ConfigNodeClient.java:217`:
`IConfigNodeRPCService.Iface.getProcedureResult(long)` is missing. This PR does
not modify that code.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] concurrent read and write
- [x] added comments explaining the "why" and the intent of the code
wherever it would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `PipeSinkSubtaskManager`, `PipeSinkSubtask`, and `PipeTaskSinkStage`
- `PipeTaskAgent` and `PipeDataNodeTaskAgent`
- `PipeHeartbeatParser`
- `PipeDataRegionSinkMetrics` and `PipeSchemaRegionSinkMetrics`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
