>From a CI perspective, I chatted with :gps in person in SF about this. While I personally am of the opinion of CI pulling in the external repo, :gps and release folks want to vendor the tests in. This is because they do not want our tree's open-ness to be dependent on an external service (GitHub). But, like, I'm pretty sure GH's SLA is better our own...
Given that, here are the concrete next steps: - Import test262 into m-c, keeping it up-to-date by periodic imports. - Get a tier-2 test262 task (i.e. test262 failing does not close the tree) and gather feedback about the update process/repo perf with the 20k extra tests. - If everything looks well, remove the tier-2 task and run test262 as part of regular jstests. On Wed, Jan 25, 2017 at 11:26 AM, Tom Schuster <t...@schuster.me> wrote: > Thank you (and anbda of course)! I am so glad we are finally going to > track test262 progress \o/. > This should also make it really easy for everyone to figure out which > tests are still failing. > > I second Shu's opinion on defaults and excluding directories. > > What is the process of keeping test262 up-to-date going to be? > > -Tom > > > On Wed, Jan 25, 2017 at 7:34 PM, Shu-yu Guo <s...@mozilla.com> wrote: > > Woohoo, thanks for the CI work! > > > > Outside of CI, I am strongly in favor jstests.py running test262 by > > default. Concretely, I am in favor of: > > - Default set to be everything, including test262, and > > - Ability to exclude entire subdirectories > > > > They supersede many of our tests and is the general source of compliance > > truth for many corner cases. The increased running time is worth it for > > correctness when implementing new features. JIT hackers can most likely > get > > by most of the time without running the full suite. > > > > On Wed, Jan 25, 2017 at 9:55 AM, Steve Fink <sf...@mozilla.com> wrote: > > > >> anba has done some major work to get test262 runnable as a CI test, and > >> I've been looking into creating taskcluster jobs for it. One thing that > >> could use input from a wider audience is how we want it to work with > >> running jstests manually. The tests replace the js/src/tests/test262 > >> directory, which means that if we do nothing, the runtime of jstests.py > on > >> an opt build goes from about 50 seconds to about 260 seconds on my > machine. > >> I haven't tried a debug build. > >> > >> One option is to suck it up. If everyone (but me) tends to use jit-tests > >> for their smoke tests already, then it doesn't matter. And you can > always > >> explicitly choose subdirectories to run. > >> > >> Another option, which I've implemented but wanted some feedback before > >> landing, is to say that running jstests.py with no arguments gives you > "the > >> default set", rather than its current "everything". And that default set > >> would be "all but test262". If you want everything, you can say > >> > >> ./jstests.py $JS * > >> > >> But in order to implement that, I had to implement directory exclusions. > >> Which means it wouldn't be much extra effort to implement > >> --exclude=test262, in which case the first option is a little more > >> tolerable: |./jstests.py $JS| would give you everything, and if you > didn't > >> want the huge pile of test262 tests, you could do |./jstests.py $JS > >> --exclude=test262|. > >> > >> A third possibility that comes to mind is to have presets. |./jstests.py > >> $JS --smoke| or something. > >> > >> Opinions? > >> > >> > >> > >> _______________________________________________ > >> dev-tech-js-engine-internals mailing list > >> dev-tech-js-engine-internals@lists.mozilla.org > >> https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals > >> > > > > > > > > -- > > shu > > _______________________________________________ > > dev-tech-js-engine-internals mailing list > > dev-tech-js-engine-internals@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals > -- shu _______________________________________________ dev-tech-js-engine-internals mailing list dev-tech-js-engine-internals@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
