Caideyipi opened a new pull request, #18483:
URL: https://github.com/apache/iotdb/pull/18483
## Description
### Pooled TsFile parsing
- Adds opt-in `processor.tsfile-parser.parallelism` for the built-in
do-nothing processor.
- Runs admitted TsFile parser tasks in a shared executor while respecting
the existing global and per-Pipe-region parser admission limits.
- Keeps custom processors and the default configuration serial.
### Ordering, retry, and lifecycle
- Lets the source TsFile own the ordered commit id while parsed tablets
retain its progress reference, so progress is committed only after all
generated output is released.
- Preserves parser progress across retryable failures, retries permanent
failures through the existing subtask retry path, and releases in-flight or
pending events on close.
- Allows progress-report and heartbeat control events to pass the parser
barrier without splitting a run of parseable TsFiles.
### Parser output memory backpressure
- Byte-accounts parsed tablets waiting in the sink queue and reserves
downstream memory headroom before publishing them, preventing parallel parsers
from exhausting the memory needed to drain the queue.
- Accounts tablet sink batches as tablet memory and releases queue
reservations on poll, clear, and discard paths.
- Adds focused tests for parser concurrency, retry, progress ownership,
memory admission, and pending-queue reservation release.
### Scope
This PR intentionally does not include the TsFile local sink, its
writer/tests, or the source-TsFile grouping metadata used only by that sink.
Configurable TsFile parser selection is already present on `dev/1.3` via #18449
and is not duplicated here.
### Verification
- `mvn spotless:apply -DskipTests`
- `mvn -pl iotdb-core/datanode -am
'-Dtest=PipeProcessorSubtaskExecutorTest,PipeRawTabletInsertionEventTest,PipeMemoryManagerTest,PropertiesTest'
'-Dsurefire.failIfNoSpecifiedTests=false' '-DskipITs' '-Dcheckstyle.skip=true'
'-Dspotless.check.skip=true' test` (26 tests, 0 failures)
- `mvn -pl iotdb-core/node-commons -am '-Dtest=BlockingPendingQueueTest'
'-Dsurefire.failIfNoSpecifiedTests=false' '-DskipITs' '-Dcheckstyle.skip=true'
'-Dspotless.check.skip=true' test` (2 tests, 0 failures)
- `mvn -pl 'iotdb-core/node-commons,iotdb-core/datanode' -am '-DskipTests'
'-Dspotless.check.skip=true' checkstyle:check` (0 violations)
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] concurrent read and write
- [x] added documentation for new or modified features or behaviors.
- [x] added Javadocs for non-trivial methods.
- [x] added comments explaining intent where the concurrency and memory
behavior is not obvious.
- [x] added or updated unit tests for new code paths.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `PipeProcessorSubtask`
- `PipeEventCollector`
- `PipeRawTabletInsertionEvent`
- `PipeTsFileInsertionEvent`
- `PipeMemoryManager`
- `BlockingPendingQueue` / `UnboundedBlockingPendingQueue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
