I'd like to further highlight Steve's point. We're really in need of a better regression system. There are probably three aspects that need to be improved.
1) We need tests that run good and bad scripts. This should test for fails when they should (if you haven't specified some key component, you should get a error message not a crash). We need short tests that verify output like trace flags, checkpoints, restoration, sampling, etc. Pretty much every time someone identifies a bug on the mailing list with the running of gem5 we should be adding a test that will catch that in the future. 2) We need different regression tests. Booting OSes and running SPEC apps is OK, but it probably takes longer to run that it should. We really need smaller tests that actively test parts of the system with known results. For example running a benchmark that hasa branch on every cache line should have known icache hit/miss rates. This test could verify that the icache hit/miss rates don't change. These tests and the immutable stats could be specially flagged. When we make changes we know are going to change the stats output slightly we could still verify correct execution with these tests. 3) To have a hope of doing these we really need a better regression system. I started creating one using pytest, and it can run a hello world regression at this point, but unfortunately I haven't had any time to work on it and it really stalled with the problem of how to easily build configurations programmatically that test certain behavior without lots of code duplication. We need a system where we can more easily identify which tests we want to run, be able to add tests via EXTRAS, skip tests that don't have the required inputs, verify different kinds of output depending on if the test is supposed to fail or not. If we had such a system we should be able to use a continuous integration framework (e.g. Hudson, Jenkins, etc) which could test every test and they're committed and automatically promote changesets that pass all tests. Thanks, Ali On 07.04.2012 12:32, Steve Reinhardt wrote: > This patch is fine, but just to keep things in perspective, we really need > a new regression framework, so I don't want to see too much effort invested > in the current system before we know how much of it will carry over into > the future. _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
