diegomrsantos opened a new issue, #4174: URL: https://github.com/apache/iggy/issues/4174
This tracks investigation of disk polling costs and possible optimizations following #4119. The aim is to identify costs that remain in current code, measure their practical value, and preserve the correctness of poll completion and consumer progress. Excess read bytes and repeated batch processing are the strongest code leads. Owner service, tail latency and task construction need more current diagnostics. The observations below do not establish that every area needs a code change, or that any one area explains the measured difference under writes. **Work items** - [ ] Investigate excess read bytes for small disk polls. - [ ] Investigate repeated batch verification and record scanning. - [ ] Measure owner completion service under concurrent polling. - [ ] Investigate poll tail latency under concurrent writes. - [ ] Measure task construction and completion payload costs. The first two items can share read and batch counters. Owner service and tail investigations can share stage timestamps. Each item should have its own decision and, when warranted, a separate implementation comparison. **Latest comparison** The [published report and raw evidence](https://github.com/diegomrsantos/iggy/releases/tag/pr-4119-final-review-evidence-2026-09-13) compare candidate `c7e6fb1655bd48c18c2fb567917f88ec1bdf7fcc` with baseline `585ab692210b4e900cdc1153e75842378a33370e`. Positive changes mean more elapsed time per completed poll. | Case | Baseline µs per poll | Candidate µs per poll | Mean time change | Pointwise 95% interval | | --- | ---: | ---: | ---: | --- | | Explicit offset | 258.42 | 258.67 | +0.10% | [−1.35%, +1.69%] | | Explicit offset with writes | 291.93 | 298.71 | +2.32% | [+0.41%, +4.36%] | | Next | 303.86 | 302.40 | −0.48% | [−1.41%, +0.44%] | | Consumer group | 330.16 | 331.63 | +0.45% | [−0.59%, +1.45%] | | Resident | 133.44 | 128.41 | −3.77% | [−5.27%, −2.38%] | The campaign used 12 independently launched baseline and candidate pairs per case, with balanced randomized order. All 120 measured runs completed 2,304,000 nonempty polls without recorded errors, timeouts or cancellations. The separate 20 run pilot is excluded. Estimates are ratios of arithmetic means, with 20,000 bootstrap resamples of whole pairs. Intervals are pointwise, without adjustment across cases or secondary metrics. Intervals spanning zero do not establish equivalence. With writes, the p99 change was +3.40% [−0.62%, +7.63%]. The earlier Next p99 increase was not reproduced: the rerun estimate was −0.84% [−2.37%, +0.69%]. These are geometric means of paired ratios of run p99 values, not pooled request percentiles. No automatic 3% margin conclusion follows. The rerun followed reported high Docker memory consumption and a recent guest restart. Existing checks recorded no CPU throttling or container memory limit events, but there was no extra memory monitor. This does not establish memory causation or rule out host pressure. Earlier and later campaigns remain separate. **Older diagnostic profiles** The following observations predate the latest implementation. Profiled baseline: `dc2b382097ed7fb6cfeb8a31fed0fbc74af4433e`. Profiled candidate: `a699ff91c6b9126502ea112ac676d96d8d107975`. | Observed work | Workload | Baseline sampled CPU, µs per poll | Candidate sampled CPU, µs per poll | | --- | --- | ---: | ---: | | `__arch_copy_to_user`, function itself | Explicit offset | 46.973 | 45.758 | | `__arch_copy_to_user`, function itself | Explicit offset with writes | 57.175 | 62.033 | | Checksum and hash category, including called functions | Explicit offset | 100.474 | 101.810 | | Checksum and hash category, including called functions | Explicit offset with writes | 109.583 | 110.524 | These are descriptive arithmetic means of two diagnostic captures per revision and workload, each with 33,000 completed polls and opposite version orders. Linux perf sampled software `cpu-clock` at 499 Hz. Sample periods were normalized by completed polls. Inclusive categories overlap with callers and must not be added to their CPU totals. Profiling can perturb execution; no confidence interval or speedup claim is assigned to these observations. Copying and verification were substantial in both old variants. This motivates checking shared work, but does not identify avoidable bytes, repeated batches or the cause of the latest mixed workload difference. The detailed historical profile reports and raw captures are retained locally; their relevant numerical observations are reproduced here. They are distinct from the published final comparison archive linked above. **Shared measurement contract** The experiments used one shard and one sequential TCP consumer with payloads of 256 bytes. Explicit offset cases returned one message without automatic commits. The latest Next and resident cases returned one message with automatic commits; group polls returned 100 messages with automatic commits. Explicit offset runs measured 33,000 polls, and the other latest cases measured 10,000. Separate fixtures provided 100 warmup polls. Preparation, preload and shutdown were outside the consumer window. The producer targeted 50,000,000 payload bytes per second in batches of 100. In the latest comparison it achieved 50,154,102.84 to 50,505,824.75 bytes per second, within its 95–105% guard. Written messages per completed poll averaged 57.45 for baseline and 58.81 for candidate. Mixed server CPU includes serving writes and cannot be interpreted as consumer CPU alone. Measurements ran in Docker Desktop's ARM64 Linux VM on an M1 Pro MacBookPro18,1, with server work pinned to CPU 0 and fixtures on a guest volume. Reads could hit the OS cache. The latest disk cases used replicated durability, a flush threshold of one, 16 MiB segments and disabled preallocation. These results do not establish cold storage latency, physical disk traffic, multishard scaling or behavior under many concurrent consumers. Exact offsets were validated for explicit offset cases. Raw Next and group records do not establish exact returned offsets or payload identity; correctness tests provide separate evidence. Qualifying a new experiment should validate its returned work and preserve these workload distinctions. **Initial diagnostic milestone and completion** Inspect the existing run, resource and producer records, then collect a small fixed current diagnostic set with and without writes. Share counts of bytes read, bytes hashed, batches visited and pump turns, plus read start → completion → enqueue → owner acceptance → reply resumption. A reply becoming ready and its caller resuming are separate events. For a proposed optimization, compare one source revision with the same source plus only that change. Record exact binary identities and preserve the relevant output, integrity, durability and ownership contracts. Keep clean timing separate from diagnostic instrumentation, assess variation with identical binaries where needed, and declare the worthwhile effect and fixed measurement budget before confirmation. Coordinate representative Linux workloads with existing benchmark work when possible. Retained shutdown receives, reserved completion capacity, numeric history IDs and ownership simplifications are already in the implementation reviewed here. Old profiles are not measurements of their current costs. Source references in the children are pinned to `8f1c0e260076f9fbe49539e904e43895bc24c3fe`, the merged PR head, rather than an evolving master. The tracker is complete when each child has a documented measurement and decision, or is explicitly removed from scope. Rejecting an optimization hypothesis is a valid outcome. No child requires weakening correctness or producing a patch merely to close the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
