diegomrsantos opened a new issue, #4178: URL: https://github.com/apache/iggy/issues/4178
Part of #4174. Affected areas: Iggy server, Performance. Determine whether the unresolved change in poll tail latency under concurrent writes has a reproducible cause, and whether that cause warrants a code change or a better measurement control. This is a lower priority investigation until a recurring delay or an explicit tail requirement makes the decision concrete. The [published final comparison and raw evidence](https://github.com/diegomrsantos/iggy/releases/tag/pr-4119-final-review-evidence-2026-09-13) compare candidate `c7e6fb1655bd48c18c2fb567917f88ec1bdf7fcc` with baseline `585ab692210b4e900cdc1153e75842378a33370e`. | Case | Change in run p99 | Pointwise 95% interval | | --- | ---: | --- | | Explicit offset | −0.02% | [−2.79%, +2.88%] | | Explicit offset with writes | +3.40% | [−0.62%, +7.63%] | | Next | −0.84% | [−2.37%, +0.69%] | | Consumer group | +0.86% | [−0.89%, +2.65%] | | Resident | −2.33% | [−6.29%, +1.23%] | Positive values mean higher p99. These estimates are geometric means of paired ratios of each run's p99, not percentiles pooled across requests. Each case used 12 independently launched baseline and candidate pairs with balanced randomized order. Pointwise percentile intervals use 20,000 bootstrap resamples of whole pairs, without adjustment across cases or secondary metrics. The fixed sample was not extended according to its results, and pilots were excluded. The interval with writes includes a small improvement and a meaningful increase. It therefore does not establish a tail regression or demonstrate a 3% margin. Mean time under the same mixed workload increased from 291.93 to 298.71 µs per poll, or +2.32% [+0.41%, +4.36%], which is a separate metric. The preceding campaign using the same binaries reported Next p99 +6.79% and consumer group p99 +6.06%. Those increases were not reproduced in the rerun. Both campaigns are retained separately. The rerun followed reported high Docker memory consumption and a recent guest restart, but did not independently manipulate and continuously measure memory pressure. No container CPU throttling or memory limit events were recorded; those checks cannot rule out host pressure or establish memory causation. The workload used one shard, one sequential TCP consumer and payloads of 256 bytes. Explicit offset cases returned one message without automatic commits, for 33,000 measured polls per run. Next and resident cases returned one message with automatic commits, and group polls returned 100, for 10,000 polls per run. Each run used separate warmup and measured fixtures. All 120 measured runs completed successfully, totaling 2,304,000 nonempty polls. The mixed producer targeted 50,000,000 payload bytes per second in batches of 100 and achieved about 50.15–50.51 MB per second. Mixed server CPU includes serving writes. Measurements ran in Docker Desktop's ARM64 Linux VM on an M1 Pro, with guest volume fixtures and possible OS cache hits. This does not establish independent arrival behavior, cold storage latency or performance across deployment environments. The investigation should: 1. Inspect the published raw poll timelines, run order and resource records, preserving valid slow runs. Identify whether slow operations cluster around a particular stage or resource condition. 2. Reuse the parent's current diagnostics and the owner service investigation. If more detail is necessary, capture sparse timestamps for read completion, enqueue, owner acceptance and caller resumption, alongside scheduler waits and queue observations. CPU profiles alone do not locate time spent waiting. 3. State the latency boundary, intended load model and worthwhile effect before confirmation. If independent arrivals matter, use an appropriate load generator and include scheduling delays, errors and timeouts. 4. Confirm a specific recurring mechanism or environment control with a fresh fixed budget and declared estimator. Keep diagnostic instrumentation separate from clean timings and preserve the existing polling and integrity contracts. More tracing can create the tail being investigated. Additional repetitions on one laptop VM cannot establish physical device latency or remove systematic environment bias. Do not pool the campaigns, trim valid slow runs or continue sampling until a preferred margin passes. Completion is a documented explanation and decision, or a bounded finding that the available evidence does not justify further tuning. No memory or queue change should be selected solely from the difference between these campaigns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
