gabotechs commented on PR #23975:
URL: https://github.com/apache/datafusion/pull/23975#issuecomment-5188626225
> Another thing we could do potentially is to add some sort of mode to the
benchmark runner ("stats verification mode" perhaps?)
What I found challenging is how to integrate this with existing benchmark
runner without it getting in the way of the classical "performance benchmarks"
path.
For example, typical performance benchmarks have the option to specify the
number of iterations, which does not make much sense with planing stats vs
execution metrics benchmarks, and planing stats vs execution metrics benchmarks
need to render the output in a very certain way for them to be useful (plans
displayed with q-error per operator).
What I found is that integrating it in existing benchmarks would require
some "if normal_path; do this; else if stats_vs_metrics_path; do this other
different thing" conditional logic that could end up getting in the way of
other people contributing to the classical benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
