diegomrsantos opened a new issue, #4175: URL: https://github.com/apache/iggy/issues/4175
Part of #4174. Affected areas: Iggy server, Performance. `DiskReadPlan::read_disk` in [poll_plan.rs](https://github.com/apache/iggy/blob/8f1c0e260076f9fbe49539e904e43895bc24c3fe/core/partitions/src/poll_plan.rs) starts with a 1 MiB chunk, capped by the remaining persisted bytes. The requested message count does not affect that initial size. `read_chunk_with_retry` allocates the buffer and reads the chunk before `walk_disk_chunk` selects the requested records. A poll requesting one message with a 256 byte payload can therefore read substantially more than the complete batch needed to serve it. Adjacent polls may also read overlapping file ranges. This could add allocation and kernel copying costs even when file data is already in the OS cache. Earlier diagnostic profiles found substantial CPU time in `__arch_copy_to_user`, the kernel function that copies data into application buffers: | Workload | Baseline sampled CPU, µs per poll | Candidate sampled CPU, µs per poll | | --- | ---: | ---: | | Explicit offset, without concurrent writes | 46.973 | 45.758 | | Explicit offset, with concurrent writes | 57.175 | 62.033 | Profiled baseline: `dc2b382097ed7fb6cfeb8a31fed0fbc74af4433e`. Profiled candidate: `a699ff91c6b9126502ea112ac676d96d8d107975`. Each value is the arithmetic mean of two diagnostic captures, with 33,000 completed polls per capture and opposite version orders. The workload used one shard, one sequential TCP consumer, explicit offsets, automatic commits disabled, and one message with a 256 byte payload per poll. The concurrent producer targeted 50,000,000 payload bytes per second in batches of 100 messages. Linux perf sampled software `cpu-clock` at 499 Hz in Docker Desktop's ARM64 Linux VM on an M1 Pro. These are estimates of CPU time sampled in the function itself, normalized by completed polls. They are not elapsed poll latency or predicted savings. The copying cost in both revisions motivates investigating the shared read path, but does not establish how much copying comes from unnecessary reads. CPU with concurrent writes also includes serving the producer. These historical profiles do not establish the cause of later performance differences or physical storage traffic. The parent reproduces the shared evidence and links the separate latest timing campaign; the original diagnostic reports and raw profiles remain archived locally. The proposed investigation is: 1. Measure bytes requested from files, bytes returned to consumers, read calls, repeated file ranges and rereads caused by incomplete batches. Record complete batch sizes and sparse index starting positions so necessary batch reads are distinguishable from avoidable extra bytes. 2. Use those measurements to evaluate one focused change to initial read sizing. The decoder needs complete encoded batches. The existing implementation grows the chunk and rereads when a batch does not fit, so reducing the initial size can increase syscalls and repeated work. 3. Compare the same source with and without the change through actual polling using fresh paired runs. Cover requests for one message and larger responses, large batches, active and sealed segments, and polling with and without concurrent writes. Measure elapsed time, CPU, memory, read calls and rereads while validating returned records. The implementation already caches file descriptors and sealed indexes and avoids zeroing read buffers before filling them. Share counters with the repeated batch processing investigation in #4174, while keeping their implementation comparisons separate: fewer read bytes do not necessarily mean fewer verified batches. A successful change should reduce unnecessary reads and demonstrate a useful performance improvement without materially regressing larger polls. Preserve checksum and layout validation, offsets, timestamps, contiguous results, partial read handling and existing corruption behavior. If buffer reuse is introduced, bound retained memory and define its handling of segment growth, purge, truncation and history changes. This investigation can also finish with a documented result that the avoidable work is too small, or that the extra calls and rereads outweigh the saving. No code change is required without supporting measurements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
