Hi folks. I'm getting started integrating the systemc tests into the new testing framework, and I'd like to change how it finds tests to run. Currently it scans for files which starts or ends with test or tests, ends in .py, and which separate test or tests with a - or _ from the rest of the name. It looks for those files recursively relative to the current directory. That has a few problems.
First, once a bug is fixed I have a CL out for, the tests/main.py (may need a better name? Not obvious that's what you're supposed to run) can be run from outside the tests directory. That means that you may end up scanning files that you don't intend, or not scanning files that you do intend. I don't think the directories you look in should be based on your current directory, although I don't immediately have another proposal. I haven't put a ton of thought into it yet, suggestions welcome. Second, it's easy to have false positives. For instance, when I attempted to run main.py from the root of the checkout, so ./tests/main.py, it picked up a file in ext/googletest, specifically ext/googletest/googlemock/scripts/generator/cpp/gmock_class_test.py. It then dutifully executed that code to look for tests. In this case that file doesn't seem to do anything, but I don't think it's unrealistic that there could be accidental undesirable side effects from something which has an unfortunate name. I think if we're going to be executing things to see what tests they define (which is reasonable IMO), we should make sure their author intended for them to be used that way. In addition to accidental name collisions, there are definitely security implications from this scheme. Let's say I've got a gem5 tree checked out which no one else can modify, but for whatever reason they can still create files within. If they make a file with the right name, they can run arbitrary code as the owner of the tree just by tucking something away deep in the file hierarchy of gem5 somewhere when they go to run the tests. This is particularly dangerous when the paths that are scanned aren't well constrained/constant. Finally, the framework seems to be written with the intention that all tests would go in the tests directory. This has been gem5's tendency historically, but I think, outside of a few potential exceptions, it's a bad practice. Tests should be alongside the things that are being tested. For instance, if I get the systemc tests hooked into the new framework, that should be in src/systemc/tests (where the tests live), not in tests/systemc or similar. That makes it important to be able to be able to run the framework on directories outside the tests directory, minimally in the src directory, and have it do the "right thing", not run errantly named files, etc. Thoughts? Suggestions? Gabe _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
