If the objection against run-bindings-tests is that they're not part of some larger test script which developers can run locally, it's very easy to add a wrapper script which runs all known testing harnesses.
The test tests which currently run on the bots include: run-webkit-tests (minutes) run-javascriptcore-tests (45s) test-webkitpy (32s) test-webkitperl (2.0s) run-binding-tests (2.4s) run-api-tests (failed on my machine, so I can't tell you how long) There are a couple other scripts which run on specific platforms, but that's the core set. run-webkit-tests is the bulk of the time, taking multiple minutes on a modern machine (even with NRWT). I'm happy to write a run-all-tests script which runs all known tests that platform can handle. :) -eric On Thu, Sep 8, 2011 at 12:19 PM, Oliver Hunt <oli...@apple.com> wrote: > > On Sep 8, 2011, at 11:55 AM, James Robinson wrote: > > We used to not run these tests on the bots. This meant that people would > change the bindings code and not update the expected results, so the > expected results were always massively out of date. This meant when > patching the bindings scripts you could not rely on run-bindings-tests at > all, because the expectations were already broken before you made any > changes! This it not theoretical, it happened to be multiple times and I > know I'm not the only one. > The real problem here is that people check in without looking at the bots > and then do not respond when the bots go red. That's a people problem. > > The real problem is that we have a test suite: run-webkit-tests, that > everyone runs (it's even mentioned at step 4 on > http://www.webkit.org/coding/contributing.html , which is apparently not > running all the tests. > I run-webkit-tests before i commit (new-run-webkit-tests has ensured that > any prior complains about time taken no longer exist, so kudos to those folk > \o/ ), and yet I end up breaking the build in a way that would show up > locally if the tests were simply run. run-webkit-tests should run all of > the webkit tests -- not some subset, all of them. If failing a cross > platform test can turn the bots red, then that test should be covered by > run-webkit-tests. > The other problem is "people check in without looking at the bots". I do > try to watch the bots, but the time between me landing and the bots actually > going red can literally be hours. Of course i'm away from irc/email > whatever when I land a patch at 4pm and it turns the bots red at 11pm. > I appreciate this isn't as much of a problem for people who don't work on > code for which all changes heralds a world rebuild and effect (apparently) > every single test that exists in the repository, but it's certainly > frustrating for those of us who do. > (This is ignoring the overly aggressive rollouts of large patches the break > only single platforms due to platform specific code that is difficult for > anyone outside of that platform to fix) > --Oliver > > - James > >> >> - WBR, Alexey Proskuryakov >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
