Right, I agree we should avoid involving scons since we might want to run
tests that don't involve scons for whatever reason, or run it as a substep
which might be confusing, etc. It does make a nice collection point for
setting up these sorts of things, but I think the drawbacks make it
something worth avoiding.

It's also reasonable to put off renaming main.py, although it's something
we should do at some point.

Gabe

On Thu, Jan 17, 2019 at 10:31 AM Jason Lowe-Power <[email protected]>
wrote:

> Hey Gabe,
>
> I agree that the current implementation isn't the best. I'm happy to hear
> suggestions for better test discovery. Even a little yaml file or something
> like that in tests/ which lists all of the files to execute to look for
> tests would be fine with me. I don't love the idea of using SCons (plus it
> has all of the downsides you just mentioned ;)). @Sean Wilson
> <[email protected]>, do you have any recollection of what we were
> thinking when we implemented this?
>
> Re: Name of tests/main.py. I agree that this is not a great long-term name.
> I'd like to put off renaming it until we delete all of the old testing
> framework, though.
>
> Cheers,
> Jason
>
> On Wed, Jan 16, 2019 at 3:30 PM Gabe Black <[email protected]> wrote:
>
> > Hi folks. I'm getting started integrating the systemc tests into the new
> > testing framework, and I'd like to change how it finds tests to run.
> > Currently it scans for files which starts or ends with test or tests,
> ends
> > in .py, and which separate test or tests with a - or _ from the rest of
> the
> > name. It looks for those files recursively relative to the current
> > directory. That has a few problems.
> >
> > First, once a bug is fixed I have a CL out for, the tests/main.py (may
> need
> > a better name? Not obvious that's what you're supposed to run) can be run
> > from outside the tests directory. That means that you may end up scanning
> > files that you don't intend, or not scanning files that you do intend. I
> > don't think the directories you look in should be based on your current
> > directory, although I don't immediately have another proposal. I haven't
> > put a ton of thought into it yet, suggestions welcome.
> >
> > Second, it's easy to have false positives. For instance, when I attempted
> > to run main.py from the root of the checkout, so ./tests/main.py, it
> picked
> > up a file in ext/googletest,
> > specifically
> > ext/googletest/googlemock/scripts/generator/cpp/gmock_class_test.py.
> > It then dutifully executed that code to look for tests. In this case that
> > file doesn't seem to do anything, but I don't think it's unrealistic that
> > there could be accidental undesirable side effects from something which
> has
> > an unfortunate name. I think if we're going to be executing things to see
> > what tests they define (which is reasonable IMO), we should make sure
> their
> > author intended for them to be used that way.
> >
> > In addition to accidental name collisions, there are definitely security
> > implications from this scheme. Let's say I've got a gem5 tree checked out
> > which no one else can modify, but for whatever reason they can still
> create
> > files within. If they make a file with the right name, they can run
> > arbitrary code as the owner of the tree just by tucking something away
> deep
> > in the file hierarchy of gem5 somewhere when they go to run the tests.
> This
> > is particularly dangerous when the paths that are scanned aren't well
> > constrained/constant.
> >
> > Finally, the framework seems to be written with the intention that all
> > tests would go in the tests directory. This has been gem5's tendency
> > historically, but I think, outside of a few potential exceptions, it's a
> > bad practice. Tests should be alongside the things that are being tested.
> > For instance, if I get the systemc tests hooked into the new framework,
> > that should be in src/systemc/tests (where the tests live), not in
> > tests/systemc or similar.
> >
> > That makes it important to be able to be able to run the framework on
> > directories outside the tests directory, minimally in the src directory,
> > and have it do the "right thing", not run errantly named files, etc.
> >
> > Thoughts? Suggestions?
> > Gabe
> > _______________________________________________
> > gem5-dev mailing list
> > [email protected]
> > http://m5sim.org/mailman/listinfo/gem5-dev
> _______________________________________________
> gem5-dev mailing list
> [email protected]
> http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev

Reply via email to