rustyrazorblade opened a new pull request, #103: URL: https://github.com/apache/cassandra-easy-stress/pull/103
## Problem The adaptive rate limiter optimizer runs every 5 seconds, but reads a p99 from Dropwizard's default `ExponentiallyDecayingReservoir`, which still weights samples from about five minutes ago. The loop therefore acts on evidence that predates its own last adjustment. Once latency crosses the target it cuts 10% every interval while the signal barely moves, so the cuts compound: twelve intervals is `0.9^12`, or 28% of capacity, reached in a minute. The limiter can end up far below the real knee of the throughput curve while the cluster is already healthy. Several other defects sat on top of that. All of them are observed in practice, not theoretical. ## Changes **The signal.** The four operation timers now use `Timer(SlidingTimeWindowArrayReservoir(10, TimeUnit.SECONDS))`. The window is a named constant, because the optimizer's settle time is derived from it. **A settle time.** After any change the loop waits one full latency window, so each decision sees the rate it last set. This is what stops the compounding. **Reduce against achieved throughput.** On a breach the new limit is `min(rate, throughput) * 0.9`. Under overload the client falls behind its own limit, so a limit sitting above the achieved rate constrains nothing and cutting it changed nothing. **Remove the guard that blocked reductions.** `utilizationRatio < 0.9` refused to cut whenever throughput sat below the limit. That is exactly the overload signature, so the guard blocked the only corrective action in the case that needed it. The guard against raising an unused limit stays; that one is correct. **A dimensionless increase gain.** `cbrt(headroom) / maxLatency` has units of ms^(-2/3), so its response depended on the size of the target, and not monotonically: at 10% of budget used it gave +3.16%, +4.48% and +0.97% for targets of 10 ms, 100 ms and 1000 ms. It is now `1 + 0.05 * (1 - latencyRatio)`, which gives +4.5% in all three cases. `latencyRatio` now drives all three branches, and the climb tapers smoothly into the dead band. **A rate floor** at a hundredth of the starting rate, so a run of reductions cannot strand a test near zero. **The operation gate counts all four types.** It counted only mutations and selects, which made `getPopulatePhaseLatency()` unreachable and left delete-only workloads unable to optimize. Note this is new behaviour for the populate path, not restored behaviour: those lines have never executed. **No optimizer without a latency target.** It has nothing to steer towards, so it only cost the user the ramp up. A plain `--rate 5000` run no longer starts at 500 and take a minute to reach the rate it was given. It prints why. **Lifecycle and concurrency.** The optimizer's `java.util.Timer` was never cancelled and was not a daemon, so it outlived the run and kept adjusting a finished test's limiter. It is now named, a daemon, and cancelled in the `finally` block. `execute()` and `reset()` are `@Synchronized`, fixing both the visibility of `isStepPhase` and the interleaving between the timer thread and the main thread. ## Visible behaviour change The reported p99 now covers ten seconds rather than a five-minute decay. The console column, the Prometheus export and the server API all change with it. The console reporter runs every 3 seconds, so its p99 column previously lagged the run badly; it now tracks it. The HDR and Parquet outputs are unaffected. `HdrCollector` records raw per-operation timings into its own histograms. ## Tests Six new tests, each of which fails without its fix: - `MetricsTest` asserts the timers keep every sample recorded inside the window. The old default reservoir caps itself at 1028 samples, so the sample count separates the two with no sleep. - Reduction against achieved throughput, the rate floor, the settle window, target-independent gain, and a cold throughput tracker reporting `0.0`. Full suite green: 129 tests, 0 failures, 1 skipped, including the integration suites against a real Cassandra container. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
