On 2/7/19 3:28 PM, Jason Lowe-Power wrote: > Hi all, > > Based off of the conversation on [1] I think we should have a broader > discussion of how to include test code. > > We have a number of different options with tradeoffs. I'd like to get this > right the first time and not have to go back and change things, so let's > try to hash this out now. :) > > There are multiple goals in mind when choosing testing workloads. In my > mind, there's the following, but I may be missing some other goals as well. > 1) Short. We want tests to run in seconds or minutes, not hours > 2) High coverage. Best case scenario is that we have a bunch of really > short tests that execute in seconds that each coveres a different (small) > part of gem5 and a couple of slightly longer tests that cover a larger part > of gem5. > 3) Unencumbered by licensing (for example, SPEC wouldn't work) > 4) Tests performance in an understandable way (relates to 2). Simple > workloads like matrix multiply are nice here. We know the memory footprint > so we should be able to predict the cache misses, for instance. Or predict > the number of FLOPs. > > # Option 1: Include simple code from other sources > This is what I've currently done. I went to the LLVM test suite ( > https://github.com/llvm-mirror/test-suite/) and grabbed a couple of > workloads. My thoughts with using the llvm test suite is that they would > have a number of workloads with different characteristics, be unencumbered, > and short. It turns out most of the workloads aren't that short, so I chose > a couple of the smallest ones I could find first. > > # Option 2: Roll our own > I don't think we have the developer time to do this generally. The RISC-V > tests are a good example of how we can do a great job when we do have time, > though. > > # Option 3: Choose other well-known benchmarks test suites > I'm open to other ideas. I just went with what I thought was quick and easy. > > # How to include the workloads > We also have to decide how to include the workloads. I feel strongly that > we should not include binaries in the repo, so I developed the > "DownloadedProgram" fixture for the new test infrastructure that will > automatically download the application when needed from gem5.org or any > other location. > > On the other hand, I'm less sure about where to include the code for the > tests. I don't *love* including it in the gem5 repo. However, it's really > nice to be able to see all of the code for a test in one place and not have > to search through multiple repos and our less than stellar documentation. > It's important to keep things simple in the tests, otherwise people won't > use it. > > I'm open to having a separate gem5-test-workloads repository if this is > something that others want (maybe with submodules pointing to other > projects like llvm-suite and linux). In the long term, I think this is the > best solution, but I had decided it wasn't worth the headache in the short > term. >
I'm strongly in favor of this separate repo + submodules approach. > Let me know what others think. I'd like to either get this right the first > time or have a solid plan for how we're going to get it right in the future. > > Cheers, > Jason > > [1] > https://gem5-review.googlesource.com/c/public/gem5/+/15857/1/tests/gem5/cpu_tests/benchmarks/FloatMM.c > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
