And to somewhat ironically add to this, we should actually run the tests we've got. I find myself with one knee elevated and some spare time, so I've been working on that trie idea I sent to the list a little while ago. I discovered that building the unit tests has been broken for a week or two shy of a year, so obviously nobody's been running those for at least that long, myself included of course. It would be great since we already have those tests and they should run quickly if we could make them part of regressions somehow. I expect they'd almost never fail, but it would be good to periodically check that those basic things they test still work.
I have a fix for the building issue, btw, and I'll post a review of it when it's convenient, or sooner if somebody's anxious. Gabe On 04/07/12 11:03, Ali Saidi wrote: > > > I'd like to further highlight Steve's point. We're really in need of > a better regression system. There are probably three aspects that need > to be improved. > > 1) We need tests that run good and bad scripts. This > should test for fails when they should (if you haven't specified some > key component, you should get a error message not a crash). We need > short tests that verify output like trace flags, checkpoints, > restoration, sampling, etc. Pretty much every time someone identifies a > bug on the mailing list with the running of gem5 we should be adding a > test that will catch that in the future. > > 2) We need different > regression tests. Booting OSes and running SPEC apps is OK, but it > probably takes longer to run that it should. We really need smaller > tests that actively test parts of the system with known results. For > example running a benchmark that hasa branch on every cache line should > have known icache hit/miss rates. This test could verify that the icache > hit/miss rates don't change. These tests and the immutable stats could > be specially flagged. When we make changes we know are going to change > the stats output slightly we could still verify correct execution with > these tests. > > 3) To have a hope of doing these we really need a better > regression system. I started creating one using pytest, and it can run a > hello world regression at this point, but unfortunately I haven't had > any time to work on it and it really stalled with the problem of how to > easily build configurations programmatically that test certain behavior > without lots of code duplication. We need a system where we can more > easily identify which tests we want to run, be able to add tests via > EXTRAS, skip tests that don't have the required inputs, verify different > kinds of output depending on if the test is supposed to fail or not. > > > If we had such a system we should be able to use a continuous > integration framework (e.g. Hudson, Jenkins, etc) which could test every > test and they're committed and automatically promote changesets that > pass all tests. > > Thanks, > > Ali > > On 07.04.2012 12:32, Steve Reinhardt > wrote: > >> This patch is fine, but just to keep things in perspective, > we really need >> a new regression framework, so I don't want to see too > much effort invested >> in the current system before we know how much of > it will carry over into >> the future. > > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
