Caideyipi opened a new pull request, #18076:
URL: https://github.com/apache/iotdb/pull/18076
## Motivation
Pipe control operations currently sit behind coarse-grained
ConfigNode/DataNode locks. When a transient per-pipe failure happens, the old
retry path slept for 3 seconds while still holding the
pipe/task-coordinator/node procedure locks, so one problematic pipe could make
other pipe operations feel stuck. On the DataNode side, `pushMultiPipeMeta`
also stopped at the first per-pipe failure, which made one bad pipe prevent the
rest of the batch from being applied.
This PR focuses on low-risk acceleration instead of timeout tuning, so users
can observe real progress via `show pipes` rather than getting a quick timeout
with unclear server-side state.
## Changes
- ConfigNode pipe procedure retry path
- Remove the 3s lock-held sleep before retrying the same pipe procedure
state.
- Mark only retry executions as `isYieldAfterExecution=true`, so the
procedure executor releases the global locks and requeues the procedure before
retrying.
- Keep normal successful pipe states serialized; this avoids interleaving
create/alter/drop calculate/write/push phases and avoids stale meta push risks.
- ConfigNode release side effects
- Update `PipeMetaSyncProcedure` last-synced version only after successful
completion.
- Update `PipeProcedureMetrics` timer only when the procedure is actually
finished, not on retry-yield lock releases.
- DataNode multi-pipe meta push
- Continue processing the remaining pipe metas/drop requests after a
per-pipe failure message.
- Collect only non-null per-pipe exception messages.
- Return success iff no per-pipe exception messages were collected.
- Preserve already collected per-pipe messages if a later unexpected
exception aborts the batch.
- Extract a package-visible helper/handler to make the batch behavior
unit-testable without mocking the global `PipeDataNodeAgent`.
## Tests
Added focused UT coverage:
- `AbstractOperatePipeProcedureV2Test`
- Successful state execution does not yield.
- Retry state returns quickly without the previous 3s lock-held sleep.
- Retry yield flag is reset after the next successful execution.
- State-machine execution yields only before the retry threshold and does
not keep yielding after failure.
- `DataNodeInternalRPCServiceImplPushMultiPipeMetaTest`
- Multi-pipe push continues after one per-pipe failure.
- Multi-pipe drop continues after one per-pipe failure.
- Successful multi-pipe push/drop responses do not carry exception
messages.
- Already collected per-pipe messages are retained when a later pipe
throws unexpectedly.
- Invalid request and unexpected exception paths return pipe-push-meta
error without invoking unrelated handlers.
## Verification
- `git diff --check` passed.
- `git diff --cached --check` passed before commit.
Local Maven verification was attempted but blocked by local environment
issues unrelated to this patch:
- `mvn -o test -pl iotdb-core/confignode
-Dtest=AbstractOperatePipeProcedureV2Test -DfailIfNoTests=false` fails during
main compile because the local cached dependencies do not match current source
APIs (`PipeMetaKeeper`, `SystemConstant`, `TopicConstant`, etc.).
- `mvn -o test -pl iotdb-core/datanode
-Dtest=DataNodeInternalRPCServiceImplPushMultiPipeMetaTest
-DfailIfNoTests=false` and later Spotless/test-compile attempts hit JVM
native-memory/pagefile crashes on this Windows environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
