Caideyipi opened a new pull request, #18260:
URL: https://github.com/apache/iotdb/pull/18260
## Description
This is the parser admission and fairness part of the Pipe resilience work.
### Global and per-Pipe limits
- Add `pipe_tsfile_parser_in_flight_max_num`, defaulting to half of the
available processors with a minimum of 1.
- Add `pipe_tsfile_parser_in_flight_max_num_per_pipe`, defaulting to 1.
- Enforce both limits even when Pipe memory accounting is disabled. Since
every admitted parser reserves `pipe_tsfile_parser_memory`, the per-Pipe limit
also provides an approximate parser memory quota.
### Fair admission
`PipeMemoryManager` keeps a FIFO request set per Pipe and a round-robin
queue across Pipes. After a Pipe obtains one parser, its remaining requests
move behind other waiting Pipes. A busy Pipe therefore cannot continuously
admit TsFiles while another Pipe waits.
Each TsFile event owns a stable reservation key. Timeout, explicit close,
and resource finalization cancel pending reservations so an abandoned request
cannot block the queue head.
When only hard-threshold memory headroom remains, the selector skips Pipes
that already own a parser and admits a waiting Pipe with no parser first.
### Tests
- Verify global and per-Pipe limits together.
- Verify round-robin admission when one Pipe has multiple waiting TsFiles.
- Verify hard-threshold headroom goes to a Pipe that has no active parser.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read and write
- [x] added comments explaining non-obvious scheduling decisions.
- [x] added unit tests for the new admission paths.
<hr>
##### Key changed/added classes
- `PipeMemoryManager`
- `PipeTsFileInsertionEvent`
- `PipeMemoryManagerTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
