Hello Benchmarking Dev! First, thanks for your hard work on bringing and keeping reliable benchmarks with automatic single CL blaming tools.
I recently have been digging more into low-level scheduling primitives on the v8 side and it tends to move benchmarks in interesting ways. *The problem:* pinpoint only identifies and then narrows down to a single CL for *regressions*. *The request:* pinpoint should identify and narrow down to a single CL for *improvements as well*. CLs in the scheduling space tend to move *many* benchmarks and it's hard to tell what it actually improved/regressed. For example did r534414 overall improve or regress things <https://chromeperf.appspot.com/group_report?rev=534414>? It looks like it went both ways, but only regressions were pinpointed down to my CL, what about the other dozens of green graphs, are they coincidental or also caused by my CL? I could launch bisects on all the green dots but not only is that tedious it will also result in pinging a bunch of people ( go/catabug/4225). After fixing <http://crbug.com/809961> the pinpointed regressions I'm left to wonder whether this was a no-op or an overall improvement. Not knowing what is an overall improvement is an engineering problem as it denies data that could otherwise serve as a hint to paradigms that should be encouraged and reproduced for further gains. If a CL regresses one thing and improves ten. The only way to specifically know that it improved those 10 things is to have it reverted for that 1 regression and then have the revert cause its own set of "regressions" (i.e. unimprovements)... Feels that all the data and tooling is there, can we just enable automatic pinpoint for improvements? I understanding that filing bugs for improvements is weird but as a first pass I'm sure the vast majority of engineers would be glad to be told that they're making things better, regardless of the medium through which the news is delivered! Thanks! Gab -- -- v8-dev mailing list v8-dev@googlegroups.com http://groups.google.com/group/v8-dev --- You received this message because you are subscribed to the Google Groups "v8-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
