Hi everybody. I'd like to start a conversation about testing strategies and gem5. Please let me know if my understanding is out of date, but I think the primary mechanism we use for testing is running benchmarks, booting, etc., and making sure the stats haven't changed. There are a few things that make that not so great.
1. Benchmarks can take a LONG time to run. I'd like to know whether my change is probably good in a couple seconds, not a couple hours. 2. There isn't much of an indication of *what* went wrong, just that something somewhere changed. 3. There isn't much of an indication *if* something went wrong. For a certain class of changes, it's reasonable to expect the stats to stay the same. For instance, a simulator performance optimization shouldn't change the stats. If you add a new device, change how execution works, fix some microcode, etc., you just have to guestimate if the amount of change looks reasonable and update the stats. Which, per 1, can take hours. 4. Merge conflicts. If two people make changes that affect the stats, one will go in first, and the other person will have to rebase on top of those changes and rerun the stats. Which, per 1, can take hours. I know writing new tests isn't what most people want to be doing with their time (including me), but as far as I can see this is a big shortcoming of the simulator as it stands. I think we would get a lot of benefit from more unit tests of both base functionality (we have a little of that), and of device models, execution bits, etc. (we have none?). We could either expand on the unit test code we have, or bring in an existing framework like this one: https://code.google.com/p/googletest/ I've never used that or know much of anything about it. It *should* be easy for us to use our modularity and object oriented design to pull pieces of the simulator into test harnesses and make sure they do reasonable things in isolation. If it isn't maybe that's something we should fix too. We should also think about how to make it easy/automatic to run unit tests, and how to get them to run automatically alongside the nightly regression runs. Gabe _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
