On Mon, Jan 11, 2016 at 3:08 PM, <jma...@mozilla.com> wrote: > Currently we run a very outdated version of V8 (version 7) in Talos. This > has since been replaced with Octane in the world of benchmarks. > > AWFY (arewefastyet.com), has been running Octane and catching regressions > faster than Talos. There is missing coverage in AWFY, specifically e10s, > pgo, aurora/beta. There are plans to add coverage for this in Q1. > > A main reason for pushing to turn off V8, is that the benchmark is > outdated and regressions could not be the most useful use of developers > time if there is a regression only seen on V8 instead of Octane. While > this does point out that we are leaning towards building performance for a > specific benchmark and ignoring other tests, we could argue that is what we > should be doing. >
As one of the people pushing for this change, let me clarify that this is not about focusing on Octane and ignoring less important benchmarks so much as it is about ignoring a specific, buggy, benchmark. v8_7 is gameable because it calls builtin functions and then doesn't use the results in any way. In some cases, it's valid to optimize for content code not using results, but in others it's just benchmark gaming. Which is fine[1] as long as it doesn't cost too much time or prevents other optimizations or even correctness fixes. In bug 1174671 various developers spent non-trivial efforts on analyzing just such a case[2]. That is actually the better of two possible bad outcomes. The worse would've been to back out a patch that fixes correctness issues, improves performance in other tests, and paves the way for further improvements. > The reason I am posting here is to find out if there are reasons we should > keep v8 running in Talos. We still plan to turn it off once AWFY coverage > matches the coverage of Talos V8. > > You can reference bug 1174671 for some history. > [1] Even required sometimes if the benchmark is high-profile and not optimizing it causes us to lose in benchmark comparisons. This used to be such a case, but nowadays nobody cares about v8_7. [2] To be fair, there was another regression that would've required analysis anyway, but the point stands: we, IMO, wasted time on analyzing a regression nobody should ever have looked at. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
