Hi all, Based off of the conversation on [1] I think we should have a broader discussion of how to include test code.
We have a number of different options with tradeoffs. I'd like to get this right the first time and not have to go back and change things, so let's try to hash this out now. :) There are multiple goals in mind when choosing testing workloads. In my mind, there's the following, but I may be missing some other goals as well. 1) Short. We want tests to run in seconds or minutes, not hours 2) High coverage. Best case scenario is that we have a bunch of really short tests that execute in seconds that each coveres a different (small) part of gem5 and a couple of slightly longer tests that cover a larger part of gem5. 3) Unencumbered by licensing (for example, SPEC wouldn't work) 4) Tests performance in an understandable way (relates to 2). Simple workloads like matrix multiply are nice here. We know the memory footprint so we should be able to predict the cache misses, for instance. Or predict the number of FLOPs. # Option 1: Include simple code from other sources This is what I've currently done. I went to the LLVM test suite ( https://github.com/llvm-mirror/test-suite/) and grabbed a couple of workloads. My thoughts with using the llvm test suite is that they would have a number of workloads with different characteristics, be unencumbered, and short. It turns out most of the workloads aren't that short, so I chose a couple of the smallest ones I could find first. # Option 2: Roll our own I don't think we have the developer time to do this generally. The RISC-V tests are a good example of how we can do a great job when we do have time, though. # Option 3: Choose other well-known benchmarks test suites I'm open to other ideas. I just went with what I thought was quick and easy. # How to include the workloads We also have to decide how to include the workloads. I feel strongly that we should not include binaries in the repo, so I developed the "DownloadedProgram" fixture for the new test infrastructure that will automatically download the application when needed from gem5.org or any other location. On the other hand, I'm less sure about where to include the code for the tests. I don't *love* including it in the gem5 repo. However, it's really nice to be able to see all of the code for a test in one place and not have to search through multiple repos and our less than stellar documentation. It's important to keep things simple in the tests, otherwise people won't use it. I'm open to having a separate gem5-test-workloads repository if this is something that others want (maybe with submodules pointing to other projects like llvm-suite and linux). In the long term, I think this is the best solution, but I had decided it wasn't worth the headache in the short term. Let me know what others think. I'd like to either get this right the first time or have a solid plan for how we're going to get it right in the future. Cheers, Jason [1] https://gem5-review.googlesource.com/c/public/gem5/+/15857/1/tests/gem5/cpu_tests/benchmarks/FloatMM.c _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
