So I was a little surprised by this last night. 1. This was relatively slow on my macbook, running locally takes ~40s. From just now:
ikilpatrick-macbookpro:src ikilpatrick$ touch third_party/blink/web_tests/external/wpt/css/css-tables/test.html ikilpatrick-macbookpro:src ikilpatrick$ time third_party/blink/tools/collect_web_tests.py external/wpt real 0m40.492s user 0m23.007s sys 0m41.715s 2. I had a bunch of uncommitted tests in various wpt directories that I needed to move out to somewhere else (rather than the script just adding committed tests). (I'd also note that uploading patches is already relatively slow, do we have metrics for how long developers are waiting for upload scripts to run? - I suspect this is due to WPT presubmit scripts but unsure) Ian On Tue, Aug 17, 2021 at 10:20 AM 'Dirk Pranke' via blink-dev < [email protected]> wrote: > > > On Tue, Aug 17, 2021 at 10:08 AM 'Weizhong Xia' via blink-dev < > [email protected]> wrote: > >> tl;dr: We introduced a mechanism that pre-collects test cases to >> //third_party/blink/web_tests/AllTestsByDirectories.json, and uses a >> presubmit check to ensure this file is up to date. You can stop here if you >> don't write web test cases. >> >> Dear blink-devs, >> >> You may have noticed that when you try to upload a CL, you are prompted >> to update AllTestsByDirectories.json if your CL changes web test cases. >> Yes, today we landed a CL >> <https://chromium-review.googlesource.com/c/chromium/src/+/3067372> that >> pre-collects test cases and saves that to the json file. The reason to do >> this is that rwt tries to collect tests on every swarming bots, and that >> takes about 30 seconds (the best case when running on SSD). This is not a >> small amount of time as we are targeting a 10-min CQ. After this CL lands, >> collecting tests now takes less than 2 seconds for all platforms. >> >> Impact to you: if your CL adds/deletes/renames web test cases, you will >> be prompted. Follow the instructions there should be enough. But if you >> still have an issue, (even though we have tested many different scenarios >> we can think about), feel free to ping/email me. >> >> There is no change to how you run rwt locally. We added a new switch to >> tell the swarming bots to load tests from the json file. This switch is >> turned off by default. Developers usually don't need to turn this on, >> unless all of your tests are saved to the json file and you are running a >> full test so want to save about 30 seconds. >> > > FWIW, I think this is an important and probably worthwhile change > (disclaimer: I did review it and was involved in the design). The number of > tests we have now is substantial and the startup delay is a significant > cost. We need to be looking into ways to reduce this. > > I think it's possible we should turn this on by default and/or do some > hooking to make things aware of when you've added new tests (e.g., via git > status) to try and make this a little more seamless, but I also think > landing it off by default was a good first step. > > Some may ask if we can just generate this at compile time. Unfortunately, > no, we can't, because there's no way to keep it correctly up-to-date when > you do add tests after having generated the file at least once [ we asked > this ourselves and actually tried this approach and broke deterministic > builds :( ]. > > I'm happy to hear other thoughts. Perhaps there are other ways we can > optimize things to be fast enough ... > > -- Dirk > > >> >> Thanks for your time! >> >> Cheers, Weizhong >> >> -- >> You received this message because you are subscribed to the Google Groups >> "blink-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiprRWy_UCC0w2oohbSGGTbeP%3DdkyYnBOx7j8Y-abDSe2A%40mail.gmail.com >> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiprRWy_UCC0w2oohbSGGTbeP%3DdkyYnBOx7j8Y-abDSe2A%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEoffTAc389E6wteVXUAuKX8ZBFWBa2UEMwAbd9oCxJ9pxNCKQ%40mail.gmail.com > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEoffTAc389E6wteVXUAuKX8ZBFWBa2UEMwAbd9oCxJ9pxNCKQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJL3UpS64fRvbJLf%2B3uFA_6UDgXoLWgUFGZSN7mAsVFOrP8syA%40mail.gmail.com.
