bryancall opened a new pull request, #13726: URL: https://github.com/apache/trafficserver/pull/13726
## What The txn_box multi-ramp autests (`txn_box_multi-ramp-1`, `-2` and `-3`) check each bucket's ramped count against a flat plus or minus 5 percentage point window. Each transaction draws its own random number, so a bucket count is binomial, and a flat window sits a different number of standard deviations from the mean at each target: | Target | Standard deviation at 1000 requests | Flat window | Sigmas | | --- | --- | --- | --- | | 10% | 9.5 | 50..150 | 5.3 | | 30% | 14.5 | 250..350 | 3.45 | At 3.45 sigma a correct build misses the window about once in 2,000 runs per 30% bucket. Each test has two such buckets, so across the three tests that is about one flake in 340 runs. This change sizes the window as 5 standard deviations either side of the expected count, from `sqrt(RepeatCount * p * (1 - p))`. That gives every bucket the same margin, and a correct build misses it less than once in a million runs per bucket. Targets 0 and 100 have a standard deviation of zero, so their window stays exact. A failure now also reports the expected count and how many sigma away the observed count was. ## Verification A bake-off in the CI image (`ci.trafficserver.apache.org/ats/fedora:42`, GCC), looping all three multi-ramp tests on two 32-core hosts. The first round ran the old checker on one host and the new one on the other. The second round runs both side by side on each host, against the same `traffic_server` binary. Each run's bucket counts were recorded, so both checkers can also be scored against identical traffic. | | Test runs | Failures | | --- | --- | --- | | Old checker, run for real | 3,273 | 1 (`'v1/video/alias' failed with 249 not in 250..350`) | | New checker, run for real | 3,279 | 0 | | Old checker, scored against every recorded run | 6,648 | 5 (0.075%, predicted 0.098%) | | New checker, scored against every recorded run | 6,648 | 0 (predicted 0.00016%) | Every failure was a 30% bucket landing just outside 250..350, at counts of 248, 249, 352, 353 and 355. Every run logged all 6,000 transactions, so none of the failures came from a timeout or a truncated log. The observed standard deviations were 14.3 and 14.6 for the two 30% buckets and 9.45 for the 10% bucket, against the binomial prediction of 14.5 and 9.5. The new window is still sensitive. I changed `multi-ramp-2.cfg.yaml` to ramp the 30% buckets at 20%, and the new checker failed them at -6.3 and -7.0 sigma: ``` 'v1/video/search' failed with 208 not in 227..373 (expected 300, -6.3 sigma) 'v1/video/alias' failed with 198 not in 227..373 (expected 300, -7.0 sigma) ``` ## Not in this PR On clang builds these tests fail on every run, for an unrelated reason: the `lt` comparison rejects integers. That is #13724. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
