Hey Gabe,

I agree that the current implementation isn't the best. I'm happy to hear
suggestions for better test discovery. Even a little yaml file or something
like that in tests/ which lists all of the files to execute to look for
tests would be fine with me. I don't love the idea of using SCons (plus it
has all of the downsides you just mentioned ;)). @Sean Wilson
<[email protected]>, do you have any recollection of what we were
thinking when we implemented this?

Re: Name of tests/main.py. I agree that this is not a great long-term name.
I'd like to put off renaming it until we delete all of the old testing
framework, though.

Cheers,
Jason

On Wed, Jan 16, 2019 at 3:30 PM Gabe Black <[email protected]> wrote:

> Hi folks. I'm getting started integrating the systemc tests into the new
> testing framework, and I'd like to change how it finds tests to run.
> Currently it scans for files which starts or ends with test or tests, ends
> in .py, and which separate test or tests with a - or _ from the rest of the
> name. It looks for those files recursively relative to the current
> directory. That has a few problems.
>
> First, once a bug is fixed I have a CL out for, the tests/main.py (may need
> a better name? Not obvious that's what you're supposed to run) can be run
> from outside the tests directory. That means that you may end up scanning
> files that you don't intend, or not scanning files that you do intend. I
> don't think the directories you look in should be based on your current
> directory, although I don't immediately have another proposal. I haven't
> put a ton of thought into it yet, suggestions welcome.
>
> Second, it's easy to have false positives. For instance, when I attempted
> to run main.py from the root of the checkout, so ./tests/main.py, it picked
> up a file in ext/googletest,
> specifically
> ext/googletest/googlemock/scripts/generator/cpp/gmock_class_test.py.
> It then dutifully executed that code to look for tests. In this case that
> file doesn't seem to do anything, but I don't think it's unrealistic that
> there could be accidental undesirable side effects from something which has
> an unfortunate name. I think if we're going to be executing things to see
> what tests they define (which is reasonable IMO), we should make sure their
> author intended for them to be used that way.
>
> In addition to accidental name collisions, there are definitely security
> implications from this scheme. Let's say I've got a gem5 tree checked out
> which no one else can modify, but for whatever reason they can still create
> files within. If they make a file with the right name, they can run
> arbitrary code as the owner of the tree just by tucking something away deep
> in the file hierarchy of gem5 somewhere when they go to run the tests. This
> is particularly dangerous when the paths that are scanned aren't well
> constrained/constant.
>
> Finally, the framework seems to be written with the intention that all
> tests would go in the tests directory. This has been gem5's tendency
> historically, but I think, outside of a few potential exceptions, it's a
> bad practice. Tests should be alongside the things that are being tested.
> For instance, if I get the systemc tests hooked into the new framework,
> that should be in src/systemc/tests (where the tests live), not in
> tests/systemc or similar.
>
> That makes it important to be able to be able to run the framework on
> directories outside the tests directory, minimally in the src directory,
> and have it do the "right thing", not run errantly named files, etc.
>
> Thoughts? Suggestions?
> Gabe
> _______________________________________________
> gem5-dev mailing list
> [email protected]
> http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev

Reply via email to