Hello Peter,
On the "adequate return" point, my opinion is that currently pgbench is just
below the feature set needed to be generally usable, so not improving it is
a self-fullfilling ensurance that it will not be used further. Once the
"right" feature set is reached (for me, at least extracting query output
into variables, having conditionals, possibly a few more functions if some
benches use them), whether it would be actually more widely used by both
developers and users is an open question.
FWIW, I think that pgbench would become a lot more usable if someone
maintained a toolset for managing pgbench. Something similar to Greg
Smith's pgbench-tools project, but with additional features for
instrumenting the server. There would be a lot of value in integrating
it with third party tooling, such as perf and BCC, and in making it
easy for non-experts to run relevant, representative tests.
Things like the rate limiting and alternative distributions were
sorely needed, but there are diminishing returns. It's pretty clear to
me that much of the remaining low hanging fruit is outside of pgbench
itself.
It happens that I might start something on the line of what you are
suggesting above.
However there is a minimal set of features needed in pgbench itself,
especially on the scripting side (functions, variables, conditions, error
handling... which are currently work in progress). I do not think that it
would be make any sense to re-implement all detailed data collection, load
throttling, client & thread handling outside pgbench, just because there
is a missing basic feature such as a particular hash function or a stupid
\if on the result of a query to implement a simple benchmark.
None of the more recent pgbench enhancements seem to make it easier to
use.
I agree that "easier to use" is a worthy objective, and that its answer is
probably partly outside pgbench (although there could be parts inside, eg
json/CSV/... outputs to help collect performance data for instance).
--
Fabien.
