Caideyipi opened a new pull request, #18261:
URL: https://github.com/apache/iotdb/pull/18261
## Description
This is the parser progress recovery part of the Pipe resilience work. It is
independent of #18260 and uses the existing parser memory reservation API on
master.
### Preserve parsing progress
`PipeTsFileInsertionEvent` now retains:
- the tablet iterator created by the TsFile parser;
- the currently parsed but not successfully consumed tablet;
- the parsed tablet count and count-operation progress.
If tablet consumption fails, the event yields its active parser memory
reservation but keeps the parser, iterator, and pending tablet. The next
subtask retry reacquires parser memory and resumes from the exact same tablet
object instead of reopening and reparsing the TsFile from the beginning.
### Parser lifecycle
Call sites that split Pipe TsFile events now close the parser only after
successful completion. Processing exceptions stop at the failed tablet and
propagate immediately. Non-Pipe TsFile implementations retain their original
finally-close behavior.
Explicit event close and resource finalization still release the pending
tablet and parser resources.
### OOM retry backoff
Local parser OOM retries use bounded exponential backoff derived from the
existing Pipe memory retry interval and maximum retry count. When the retry
policy decides to yield, later subtask retries resume from saved progress.
### Tests
`TsFileInsertionEventParserTest` verifies that an OOM does not close the
parser and that the next retry receives the identical failed tablet object. The
full parser test class passes.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read and write
- [x] added comments explaining the progress and resource-lifecycle
decisions.
- [x] added or updated unit tests for the retry path.
<hr>
##### Key changed/added classes
- `PipeTsFileInsertionEvent`
- `PipeProcessorSubtask`
- `PipeEventCollector`
- `AggregateProcessor`
- `DownSamplingProcessor`
- `WebSocketSink`
- `TsFileInsertionEventParserTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
