Kenneth, We have similar issues as well -- wanting to integrate Piglit tests into other testing frameworks -- so this hack would be nice to have for us too.
Jose ----- Original Message ----- > Recently, Gordon informed us that it took QA over two hours to do a > full > Piglit run (with all.tests) on Sandybridge, and over 8.5 hours on > Pineview. > This was a huge shock to us, as I am able to do the same run in 8 > minutes > on my Sandybridge laptop. With a touch of insight and profiling, we > were > able to find the discrepancy---and it turned out to be something both > hilarious and strange. > > Piglit has nearly 10,000 tests. Developers run them like so: > > $ ./piglit-run.py tests/all.tests results/snb-master-1 > > Piglit's runner starts up, executes all.tests (which generates the > list > of available tests), and then runs all 10,000 tests, writing the JSON > results file. > > Starting piglit-run.py has a small cost: it takes about 1 second (on > my > laptop) to process all.tests, as it walks over the filesystem to find > shaders and does a bunch of stuff. Most Piglit users don't mind, > because > the run takes 10 minutes anyway. In other words, the amortized > overhead > is 1 second / 10000 tests = 0.0001 seconds/test. Practically > nothing, > and certainly not worth bothering to optimize. Also, as we add more > and > more tests, the amortized cost goes down. > > However, QA invokes Piglit differently: they created their own test > infrastructure which runs each test individually, limiting the amount > of time tests can run (in case of infinite loops) and checking for > GPU > hangs after each one. If the GPU hangs, their runner reboots the > system > and continues the run where it left off. This is critically > important > to maintain a robust, fully automated lab. QA also needs to support > multiple test suites, such as Intel's OGLconform and the Khronos ES2 > conformance test suite. > > Instead, QA uses their runner to launch piglit-run.py tests/all.tests > -t <test name> for each of the 10,000 tests. This causes it to read > all.tests---which again, takes 1 second---for each of the 10,000 > tests. > This results in *2.7 hours* of overhead on a Sandybridge CPU; it's > much > worse on Pineview. And, as we add more tests, this cost goes up. > > So while most Piglit users get a nice 1/10000, QA gets a nasty > 1*10000! > > Ultimately, I'd like to improve Piglit's runner to be more robust, > and > hopefully allow QA to use it directly rather than requiring wrapper > infrastructure. > > However, that may take some time and effort, so instead this series > implements a quick hack: a new piglit-print-commands.py script which > prints out a list of test names and the command line piglit-run.py > would have used to invoke it. This allows QA to invoke tests > directly > and avoid piglit-run.py's startup cost. > > I'm not a huge fan of this, but it's useful for them, and simple > enough > that I don't feel too bad about including it upstream. > > _______________________________________________ > Piglit mailing list > [email protected] > http://lists.freedesktop.org/mailman/listinfo/piglit > _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
