This patch is fine, but just to keep things in perspective, we really need a new regression framework, so I don't want to see too much effort invested in the current system before we know how much of it will carry over into the future.
On Fri, Apr 6, 2012 at 12:40 PM, Gabe Black <[email protected]> wrote: > > > 1. Supress non-pass/fail information so the relevant lines don't scroll > right off the screen, forcing a rerun to get a summary (may already be > possible?) > The current designed behavior does this, though in sort of an opposite way. When the test runs, all the output is displayed, but if a test does not need to be rerun (as determined by scons), then only the pass/fail line is printed. The intent is that, once you've run the tests once, invoking the same set of tests again via scons will generate a summary of the pass/fail status. The nice thing about this approach is that it fits naturally into the dependency framework: test output always gets printed when the test is run, and we rely on dependency checking to detect a "rerun" and only print a summary. Of course, this depends on scons correctly determining that tests do not need to be rerun. However, doing things in the "opposite" order (which is how I interpret what Gabe is suggesting) would be much more involved, I think. We could suppress more test output on the initial run, to make the pass/fails stand out more clearly. I think some of the output diff suppression regexes aren't working properly which generates more output than there should be. 2. Sort passes and fails so fails are last and grouped together. > Again, this already happens, in an incomplete, indirect and opposite way. The regression cron script on zizzer that mails out the nightly output prints the fails first, then the passes, so that the former are easier to spot at the top of the output email. Something like this could be added to the util/regress script. > 3. It's nice to see things pass/fail as they finish, so maybe let them > finish normally while collecting results. Then sort them into passes and > fails, and print a summary with passes in one group, fails in another, and > maybe even a count of each. Then you can see if your super fast test failed > right away without waiting for the slow one, and after all the regressions > run all night, you don't have to run them again to actually see what > happened. And failures won't be lost in the crowd. > This sounds like an extension to #2. I think util/regress is the right place for something like this. One simple thing to try would just be to have util/regress run scons twice, leveraging the behavior described above in #1, possibly filtering/sorting the output of the second run as described here. Of course, this approach would work best if (1) scons never spuriously rebuilds anything and (2) scons didn't take a long time to figure out that nothing needs to be rebuilt. I don't think the former problem isn't too bad, in practice, since even though scons does like to re-run some of the intermediate steps (due to that source file sorting issue that's been discussed in the past), it seems to recognize that the outputs haven't changed and thus doesn't rebuild the binary or rerun the test. At least that's been my experience. > 4. Send the summary/sorted list in the emails. We've had failures we > didn't see for a while because they were surrounded by a bunch of passes. > Done, see #2. I think the code I added to print the failures first were in response to the same episode that you're recalling. Steve _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
