Hello Benchmarking Dev!

First, thanks for your hard work on bringing and keeping reliable
benchmarks with automatic single CL blaming tools.

I recently have been digging more into low-level scheduling primitives on
the v8 side and it tends to move benchmarks in interesting ways.

*The problem:* pinpoint only identifies and then narrows down to a single
CL for *regressions*.
*The request:* pinpoint should identify and narrow down to a single CL
for *improvements
as well*.

CLs in the scheduling space tend to move *many* benchmarks and it's hard to
tell what it actually improved/regressed.

For example did r534414 overall improve or regress things
<https://chromeperf.appspot.com/group_report?rev=534414>? It looks like it
went both ways, but only regressions were pinpointed down to my CL, what
about the other dozens of green graphs, are they coincidental or also
caused by my CL? I could launch bisects on all the green dots but not only
is that tedious it will also result in pinging a bunch of people (

After fixing <http://crbug.com/809961> the pinpointed regressions I'm left
to wonder whether this was a no-op or an overall improvement.

Not knowing what is an overall improvement is an engineering problem as it
denies data that could otherwise serve as a hint to paradigms that should
be encouraged and reproduced for further gains.

If a CL regresses one thing and improves ten. The only way to specifically
know that it improved those 10 things is to have it reverted for that 1
regression and then have the revert cause its own set of "regressions"
(i.e. unimprovements)...

Feels that all the data and tooling is there, can we just enable automatic
pinpoint for improvements? I understanding that filing bugs for
improvements is weird but as a first pass I'm sure the vast majority of
engineers would be glad to be told that they're making things better,
regardless of the medium through which the news is delivered!


v8-dev mailing list
You received this message because you are subscribed to the Google Groups 
"v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to v8-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to