Hi folks. I'm getting started integrating the systemc tests into the new
testing framework, and I'd like to change how it finds tests to run.
Currently it scans for files which starts or ends with test or tests, ends
in .py, and which separate test or tests with a - or _ from the rest of the
name. It looks for those files recursively relative to the current
directory. That has a few problems.

First, once a bug is fixed I have a CL out for, the tests/main.py (may need
a better name? Not obvious that's what you're supposed to run) can be run
from outside the tests directory. That means that you may end up scanning
files that you don't intend, or not scanning files that you do intend. I
don't think the directories you look in should be based on your current
directory, although I don't immediately have another proposal. I haven't
put a ton of thought into it yet, suggestions welcome.

Second, it's easy to have false positives. For instance, when I attempted
to run main.py from the root of the checkout, so ./tests/main.py, it picked
up a file in ext/googletest,
specifically 
ext/googletest/googlemock/scripts/generator/cpp/gmock_class_test.py.
It then dutifully executed that code to look for tests. In this case that
file doesn't seem to do anything, but I don't think it's unrealistic that
there could be accidental undesirable side effects from something which has
an unfortunate name. I think if we're going to be executing things to see
what tests they define (which is reasonable IMO), we should make sure their
author intended for them to be used that way.

In addition to accidental name collisions, there are definitely security
implications from this scheme. Let's say I've got a gem5 tree checked out
which no one else can modify, but for whatever reason they can still create
files within. If they make a file with the right name, they can run
arbitrary code as the owner of the tree just by tucking something away deep
in the file hierarchy of gem5 somewhere when they go to run the tests. This
is particularly dangerous when the paths that are scanned aren't well
constrained/constant.

Finally, the framework seems to be written with the intention that all
tests would go in the tests directory. This has been gem5's tendency
historically, but I think, outside of a few potential exceptions, it's a
bad practice. Tests should be alongside the things that are being tested.
For instance, if I get the systemc tests hooked into the new framework,
that should be in src/systemc/tests (where the tests live), not in
tests/systemc or similar.

That makes it important to be able to be able to run the framework on
directories outside the tests directory, minimally in the src directory,
and have it do the "right thing", not run errantly named files, etc.

Thoughts? Suggestions?
Gabe
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev

Reply via email to