This looks like a good start... I agree with you and Brad both that
explicitly enumerating the tests and making sure we can add tests via
EXTRAS are key features.
My inclination is to push it a little farther in terms of abstraction
though... all we really need for a test is a command to run and a
reference output directory to check against, so do we need anything
else? In particular, the OS thing was just to distinguish between the
tru64 and linux versions of tests in Alpha, so I'd hate to see that
codified as a first-class requirement. (Practically speaking, do we
support anything other than linux on any other ISA?) Similary with
the config names; those are just the names of scripts in
tests/configs, and as we've recently discussed, we want to get away
from those and start using the configs/example scripts anyway.
Maybe for simplicity we should put the command script in the reference
directory also, so you just need the direcory... there would be some
duplication (since it would be harder to share a common script across
tests) but in reality I'm not sure that ever bought us much.
So we could have something more like this (just using current paths
for backward compatibility):
Test('quick/00.hello/ref/%s/linux/simple-atomic' % env['TARGET_ISA'])
if env['TARGET_ISA'] == 'alpha':
Test('quick/00.hello/ref/alpha/tru64/simple-atomic')
If you wanted to get crazier, you could use something like a dict to
handle exceptional cases a little better (e.g., each ISA name maps to
a list of configs that have 00.hello tests, or something like that).
The nice thing here is that even though I haven't changed the paths,
the paths themselves have no inherent semantics; I'm leveraging their
current structure to simplify the SConscript where they get
enumerated, but they can easily be changed, and different groups can
have different structures.
The one thing I can think of that would be nice to support more
automatically is checking for binaries, disk images, etc. ... since
those are downloaded separately, it would be nice to cleanly skip
tests for which those aren't found (with a warning message, I think).
Also we need to make sure the paths for finding those things stay
flexible (e.g., using environment vars).
As far as groups, is there anything other than "quick" and "all" that
we ever want to use them for? If not, I'd suggest a more numerical
approach, where we assign each test a score (could be an approximate
runtime, or some sort of priority), and then I say "run all tests
shorter than 5 minutes" or "run all tests priority 2 and higher".
That's more scalable than having to assign individual tests to groups;
I'm not sure how that assignment idea would work with tests added via
EXTRAS, for example.
These are all just brainstorming thoughts, none are ideas I'm really
wedded to... if we put all our ideas together I'm sure we'll come up
with a good solution.
Steve
On Tue, Mar 2, 2010 at 9:53 PM, nathan binkert <[email protected]> wrote:
> Ok, so I'm going to try to make some progress on the tests stuff, and
> one of the issues is trying to name/specify tests. Right now, we have
> tests/(long|quick)/[0-9]{2}\.*/test.py that specifies the basics of
> the test. We also have directories like
> tests/long/00.hello/ref/arm/linux/simple-atomic. The key parts to a
> test are the test itself, the isa, the opsys, and the config. A test
> exists simply if a test.py exists in the right place and there is a
> ref directory for a particular test. My goal is to involve SCons a
> bit more and actually have users specify the existence of a test via a
> statement like:
> def Test(name, start, *args, **kwargs)
>
> start could be a filename/object which would be evaluated in a new
> environment and the function would be called with the various
> parameters and would be expected to return a unittest object.
> Alternatively, start could simply be a function which *args and
> **kwargs being called, or start could be a unittest object.
>
> _00_hello_arm_linux_simple_atomic = Test('00.hello',
> '00.hello.py:make_test', 'arm', 'linux', 'simple-atomic')
>
> I'd also like to have test groups like this:
> TestGroup('quick', [_00_hello_arm_linux_simple_atomic, ...])
>
> So, this leaves me with two questions, what should the directory
> structure look like, and where should the refs go? Should I just make
> it so that the refs directory is a parameter to Test, or should I make
> it part of some UnitTest subclass? I don't want to go with the
> current system because I'd really like to add our m5/unittest
> directory to the test system, and I'd also like to be able to put
> tests in EXTRAS.
>
> Thoughts? Steve? I'm happy to try to hash this out over the phone.
>
> Nate
> _______________________________________________
> m5-dev mailing list
> [email protected]
> http://m5sim.org/mailman/listinfo/m5-dev
>
_______________________________________________
m5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/m5-dev
