On Mon, Dec 5, 2011 at 1:53 PM, Eric U <[email protected]> wrote: > On Mon, Dec 5, 2011 at 1:01 PM, Dirk Pranke <[email protected]> wrote: >> We never implemented the "general way of marking subdirectories as >> needing to run serially", but it would be easy to do if we needed to >> [the 'http' dirs are still special-cased in the code]. > > Does it also special-case the storage tests that may collide? I'm > thinking particularly of the filesystem/filewriter tests, but anything > that deals with quota may also have issues. >
It does not, although it would be easy to add such support. I will also note that by default all of the tests in the same directory are run sequentially in a single thread, so that may be why it hasn't been much of an issue. -- Dirk >> There is code now (landed a few months ago) to control how many http >> tests run in parallel separately from the main parallelism flag (so >> you can run 16 workers but only 2 http tests at a time). I implemented >> it but never flipped the switch because it was in the middle of Eric >> flipping the bots over to NRWT in the first place. We should try >> experimenting with this to see at some point once everything >> stabilizes otherwise (this is the max_locked_shards() call in >> manager.py; there's no command line flag). >> >> -- Dirk >> >> On Mon, Dec 5, 2011 at 11:17 AM, Ojan Vafai <[email protected]> wrote: >>> Why is that? I don't know about other ports, but AFAIK, chromium writes to a >>> mock clipboard and the Apple mac port writes to a local OS clipboard >>> instance instead of the global one, specifically to avoid copy/paste tests >>> interacting. Even without running tests in parallel, it's probably a good >>> idea in order to avoid copy-pastes the developer is doing from affecting the >>> test run. >>> >>> That said, I believe we have a general way of marking subdirectories as >>> needing to run serially (which is what we do for http) if there are other >>> reasons we need to. >>> >>> On Mon, Dec 5, 2011 at 11:12 AM, David Levin <[email protected]> wrote: >>>> >>>> I believe there are some tests (copy/paste) that it would be very hard to >>>> fully shard due to how they work. >>>> >>>> dave >>>> >>>> >>>> >>>> On Mon, Dec 5, 2011 at 11:08 AM, Ojan Vafai <[email protected]> wrote: >>>>> >>>>> I looked at one example that didn't exit >>>>> early: http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29/builds/35153/steps/layout-test/logs/stdio >>>>> >>>>> In that case, the http tests were the long tail and took 6 minutes longer >>>>> than all the other tests. We don't split the http tests up because every >>>>> time we've tried it's caused too much flakiness. It's unclear if the >>>>> flakiness points to a bug in the test harness (e.g. in how we setup >>>>> apache) >>>>> or to bugs in the tests themselves or both. If someone has time to look >>>>> into >>>>> this, this is probably the biggest benefit to be found in NRWT runtime >>>>> when >>>>> running tests in parallel. >>>>> >>>>> FYI, NRWT outputs a log of the runtime after each run: >>>>> >>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO worker/9: 4696 >>>>> tests, 1746.63 secs >>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO worker/8: 1177 >>>>> tests, 1693.47 secs >>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO worker/3: 1408 >>>>> tests, 2033.51 secs >>>>> 2011-12-03 03:09:30,018 58036 printing.py:462 INFO worker/2: 941 >>>>> tests, 2119.65 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/1: 1121 >>>>> tests, 2041.97 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/0: 1453 >>>>> tests, 2515.75 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/7: 1189 >>>>> tests, 1731.12 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/6: 3556 >>>>> tests, 2114.37 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/5: 948 >>>>> tests, 2097.13 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/4: 1411 >>>>> tests, 1716.66 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/15: 795 >>>>> tests, 2027.16 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/14: 1123 >>>>> tests, 1732.72 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/13: 425 >>>>> tests, 2021.25 secs >>>>> 2011-12-03 03:09:30,019 58036 printing.py:462 INFO worker/12: 1175 >>>>> tests, 1710.09 secs >>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO worker/11: 3462 >>>>> tests, 2096.30 secs >>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO worker/10: 1449 >>>>> tests, 1722.68 secs >>>>> 2011-12-03 03:09:30,020 58036 printing.py:462 INFO 31120.45 >>>>> cumulative, 1945.03 optimal >>>>> >>>>> That shows you that, if we fully sharded all the tests, they would in >>>>> theory take 1945 seconds to run, but worker/0 (the worker that runs the >>>>> http >>>>> tests) took 2515 seconds to run. >>>>> >>>>> Ojan >>>>> >>>>> On Mon, Dec 5, 2011 at 6:55 AM, Adam Roben <[email protected]> wrote: >>>>>> >>>>>> On Dec 2, 2011, at 6:55 PM, Eric Seidel wrote: >>>>>> >>>>>> > The SnowLeopard bot went from a 1 hr 4 min (!?!) cycle time, to 38 min >>>>>> > (still !?!). >>>>>> >>>>>> I suspect our Mac test bots could use a dose of RAM. Many of them only >>>>>> have 3GB, since when you're running tests one by one you don't really >>>>>> need >>>>>> much more. >>>>>> >>>>>> -Adam >>>>>> >>>>>> _______________________________________________ >>>>>> webkit-dev mailing list >>>>>> [email protected] >>>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> webkit-dev mailing list >>>>> [email protected] >>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>>>> >>>> >>> >>> >>> _______________________________________________ >>> webkit-dev mailing list >>> [email protected] >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>> >> _______________________________________________ >> webkit-dev mailing list >> [email protected] >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
