On 07/02/2019 15:28, Jason Lowe-Power wrote: > Hi all, > > Based off of the conversation on [1] I think we should have a broader > discussion of how to include test code. > > We have a number of different options with tradeoffs. I'd like to get this > right the first time and not have to go back and change things, so let's > try to hash this out now. :) > > There are multiple goals in mind when choosing testing workloads. In my > mind, there's the following, but I may be missing some other goals as well. > 1) Short. We want tests to run in seconds or minutes, not hours > 2) High coverage. Best case scenario is that we have a bunch of really > short tests that execute in seconds that each coveres a different (small) > part of gem5 and a couple of slightly longer tests that cover a larger part > of gem5. > 3) Unencumbered by licensing (for example, SPEC wouldn't work) > 4) Tests performance in an understandable way (relates to 2). Simple > workloads like matrix multiply are nice here. We know the memory footprint > so we should be able to predict the cache misses, for instance. Or predict > the number of FLOPs. > > # Option 1: Include simple code from other sources > This is what I've currently done. I went to the LLVM test suite ( > https://github.com/llvm-mirror/test-suite/) and grabbed a couple of > workloads. My thoughts with using the llvm test suite is that they would > have a number of workloads with different characteristics, be unencumbered, > and short. It turns out most of the workloads aren't that short, so I chose > a couple of the smallest ones I could find first.
There is a set of tests on http://linux-test-project.github.io/ that we should consider as well. They seem like a great source of tests for SE mode in particular. Unfortunately, they seem to be GPL, which means that they can't go in the main repo. Having them in a separate repo and compiling them on demand or distributing binaries should be fine though. > # Option 2: Roll our own > I don't think we have the developer time to do this generally. The RISC-V > tests are a good example of how we can do a great job when we do have time, > though. Agreed. This option doesn't scale in the general case. It would make sense for things like m5ops and Arm Semihosting though. > # Option 3: Choose other well-known benchmarks test suites > I'm open to other ideas. I just went with what I thought was quick and easy. > > # How to include the workloads > We also have to decide how to include the workloads. I feel strongly that > we should not include binaries in the repo, so I developed the > "DownloadedProgram" fixture for the new test infrastructure that will > automatically download the application when needed from gem5.org or any > other location. I completely agree with this. GIT really doesn't like binaries. > On the other hand, I'm less sure about where to include the code for the > tests. I don't *love* including it in the gem5 repo. However, it's really > nice to be able to see all of the code for a test in one place and not have > to search through multiple repos and our less than stellar documentation. > It's important to keep things simple in the tests, otherwise people won't > use it. > > I'm open to having a separate gem5-test-workloads repository if this is > something that others want (maybe with submodules pointing to other > projects like llvm-suite and linux). In the long term, I think this is the > best solution, but I had decided it wasn't worth the headache in the short > term. I would prefer this solution since it maintains a strong separation between the tests and gem5's core. I'm normally not a great fan of submodules, but I think they would be useful in this case. This is particularly important if/when we start using test cases that are GPL. It would also be really nice for the LLVM test suite since some of the tests don't include proper license/copyright headers (there are separate license files in the repo though). Cheers, Andreas IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
