> On Sat, Oct 31, 2026 at 11:54:32AM +0500, Roman Khapov wrote:
> I did a lot of benchmarks in last time, and more are incoming.
> While benchmarking with pgbench I keep hitting the same annoyance: the final
> report says only "latency average" and "latency stddev",
> which tells nothing about the distribution. p95/p99 is usually what you
> actually want when evaluating tail latency.
Yep, having pgbench reporting percentiles is definitely a good idea.
> What if we add new option, smth like
> --latency-percentiles[=p1,p2,...], which extends the final reports with:
> latency min = 4.114 ms
> latency max = 15.213 ms
> latency p50 = 6.097 ms
> latency p95 = 12.211 ms
> latency p99 = 13.883 ms
>
> Main design points, kept deliberately simple for a first version:
> - percentiles are estimated from a fixed-size log-scale histogram
> (32 sub-buckets per octave, 4 KB, zero allocations, a few ops per
> transaction; no overhead at all when the option is not given). A more
> clever estimator (t-digest and friends) can replace it later behind
> the same interface, but seemed like overkill to start with;
> - the histogram is a separate member of StatsData next to the latency
> SimpleStats, so existing stats structs/log files/per-command table and
> the default output stay untouched;
> - semantics match existing latency stats: successful, non-skipped
> transactions only;
I'm less sure about this particular implementation though. From what I
understand buckets and sub-buckets width is log-scale, meaning that the
higher it gets (overall and withing a bucket) the more uncertainly we
get, right? It could be quite annoying at times, like with bpftrace hist
where too large buckets sometimes obscure interesting data variation on
the far side of the range. Even more, I've managed to get a test run
where the p95 was larger than the reported max latency:
latency average = 67397.720 ms
latency stddev = 2167.818 ms
latency min = 63949.915 ms
latency max = 70166.801 ms
latency p95 = 70464.307 ms
I think this can be an argument for more complicated, but more precise
algorithm for percentiles estimation.
On top of that it would be great if the actual algorithm implementation
would not be tightly coupled with the pgbench itself, but rather will be
agnostic to the call site, allowing to use it everywhere.
Aside of already mentioned issues, the interface could be probably
improved:
* It looks weird to me that the same option enables min/max and the
percentiles. While I see where it's coming from, if the option is
called "latency-percentiles", it's reasonable to expect it to show
percentiles, and have a separate option for min/max.
* There are examples of options that could be repeated multiple times,
e.g. the pgbench script to invoke. Maybe it would be easier to do the
same with the percentiles, instead of coming up with a format to
scrabmle everything into a string.
* Pgbench progress also reports latency average, maybe worth extending
it as well.
