Caideyipi opened a new pull request, #18049: URL: https://github.com/apache/iotdb/pull/18049
## Description This PR improves the pipe creation memory admission check from a coarse per-pipe estimate to an incremental estimate based on the tasks and sink subtasks that this DataNode will actually create. Changes include: - Estimate memory by local region tasks, and also check newly added single-region tasks. - Account for sink subtask reuse so shared sink pending queues/subtasks are not charged repeatedly. - Reuse the actual sink subtask key and parallelism logic from PipeSinkSubtaskManager. - Match task builder parameter blending/preprocessing during admission estimation. - Estimate batch memory from connector.batch.size-bytes/sink.batch.size-bytes, including leader-cache endpoint sharding. - Estimate TsFile send read buffer with pipeSinkReadFileBufferSize. - Skip admission checks for idempotent create no-ops, while checking before dropping old tasks during recreate. ## Verification - mvn spotless:apply -pl iotdb-core/datanode,iotdb-core/node-commons - git -c core.preloadIndex=false diff --check -- iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/agent/task/PipeTaskAgent.java iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/PipeDataNodeTaskAgent.java iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/builder/PipeDataNodeTaskBuilder.java iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/subtask/sink/PipeSinkSubtaskManager.java - mvn "-Ddevelocity.off=true" "-Dcheckstyle.skip=true" "-Dspotless.check.skip=true" "-DskipTests" compile -pl iotdb-core/node-commons Also ran datanode compile: - mvn "-Ddevelocity.off=true" "-Dcheckstyle.skip=true" "-Dspotless.check.skip=true" "-DskipTests" compile -pl iotdb-core/datanode - It reached javac but failed on existing generated-source/dependency issues such as missing IFill, Accumulator, and org.apache.commons.collections4.comparators, not in the modified pipe files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
