Hey Gabe, I agree that the current implementation isn't the best. I'm happy to hear suggestions for better test discovery. Even a little yaml file or something like that in tests/ which lists all of the files to execute to look for tests would be fine with me. I don't love the idea of using SCons (plus it has all of the downsides you just mentioned ;)). @Sean Wilson <[email protected]>, do you have any recollection of what we were thinking when we implemented this?
Re: Name of tests/main.py. I agree that this is not a great long-term name. I'd like to put off renaming it until we delete all of the old testing framework, though. Cheers, Jason On Wed, Jan 16, 2019 at 3:30 PM Gabe Black <[email protected]> wrote: > Hi folks. I'm getting started integrating the systemc tests into the new > testing framework, and I'd like to change how it finds tests to run. > Currently it scans for files which starts or ends with test or tests, ends > in .py, and which separate test or tests with a - or _ from the rest of the > name. It looks for those files recursively relative to the current > directory. That has a few problems. > > First, once a bug is fixed I have a CL out for, the tests/main.py (may need > a better name? Not obvious that's what you're supposed to run) can be run > from outside the tests directory. That means that you may end up scanning > files that you don't intend, or not scanning files that you do intend. I > don't think the directories you look in should be based on your current > directory, although I don't immediately have another proposal. I haven't > put a ton of thought into it yet, suggestions welcome. > > Second, it's easy to have false positives. For instance, when I attempted > to run main.py from the root of the checkout, so ./tests/main.py, it picked > up a file in ext/googletest, > specifically > ext/googletest/googlemock/scripts/generator/cpp/gmock_class_test.py. > It then dutifully executed that code to look for tests. In this case that > file doesn't seem to do anything, but I don't think it's unrealistic that > there could be accidental undesirable side effects from something which has > an unfortunate name. I think if we're going to be executing things to see > what tests they define (which is reasonable IMO), we should make sure their > author intended for them to be used that way. > > In addition to accidental name collisions, there are definitely security > implications from this scheme. Let's say I've got a gem5 tree checked out > which no one else can modify, but for whatever reason they can still create > files within. If they make a file with the right name, they can run > arbitrary code as the owner of the tree just by tucking something away deep > in the file hierarchy of gem5 somewhere when they go to run the tests. This > is particularly dangerous when the paths that are scanned aren't well > constrained/constant. > > Finally, the framework seems to be written with the intention that all > tests would go in the tests directory. This has been gem5's tendency > historically, but I think, outside of a few potential exceptions, it's a > bad practice. Tests should be alongside the things that are being tested. > For instance, if I get the systemc tests hooked into the new framework, > that should be in src/systemc/tests (where the tests live), not in > tests/systemc or similar. > > That makes it important to be able to be able to run the framework on > directories outside the tests directory, minimally in the src directory, > and have it do the "right thing", not run errantly named files, etc. > > Thoughts? Suggestions? > Gabe > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
