On 1/27/2014, 7:44 AM, Andrew Halberstadt wrote:
A downside to chunking such that skipped tests are taken into account is that it will make some chunks take longer to run than others. This negatively impacts the scheduling algorithms and ultimately increases infrastructure load.
I assume you are talking about the tests that are skipped in the manifest, not on the command line, right? In that case, yes. But I should also note that using the test file count as a proxy for the runtime requirements is... poor. :-)
The effects are especially noticeable on new platforms/applications that disable large swathes of tests to begin with (e.g b2g, android). For example, while trying to get reftests running on B2G emulators I saw cases where one chunk took 20 minutes while the next took ~120. I actually wrote a patch to do the opposite of what you are suggesting here (to not count skipped tests while chunking). Correct me if I'm wrong, but I think the issue you are trying to solve is how to figure out what try syntax is needed to run a certain test.
Yes, that's correct. > I
think there are better ways to solve this problem. I.e a more expressive try syntax such that you can just specify the test or directory of tests to run and not care about what chunk it is in. Bonus points if we can configure mozharness to run only those tests and ignore all other tests that happen to be in the same chunk.
But doesn't that mean that the code which decides what jobs to schedule should know about what will happen as part of those jobs? I may be wrong, but to the best of my knowledge, that information doesn't exist where we would want trychooser to use it.
Cheers, Ehsan _______________________________________________ dev-platform mailing list [email protected] https://lists.mozilla.org/listinfo/dev-platform
