> My inclination is to push it a little farther in terms of abstraction
> though... all we really need for a test is a command to run and a
> reference output directory to check against, so do we need anything
> else?
I don't think so. My intent is to actually be very generic and more
or less support the basic UnitTest features. I'd like to be able to
turn all of our stuff in the src/unittest directory into real tests.
I'd like to encourage us to implement subsystem tests. I'd also like
to have dependencies between tests so we can have a sequence of tests
for doing things like checkpointing and perhaps future sampling tests.
> In particular, the OS thing was just to distinguish between the
> tru64 and linux versions of tests in Alpha, so I'd hate to see that
> codified as a first-class requirement. (Practically speaking, do we
> support anything other than linux on any other ISA?) Similary with
> the config names; those are just the names of scripts in
> tests/configs, and as we've recently discussed, we want to get away
> from those and start using the configs/example scripts anyway.
I didn't intend to codify anything rigid, anything like this would be
convenience stuff on top of the basic UnitTest support. To make it
easier to define a bunch of tests with one line of code in the
SConscript file.
> Maybe for simplicity we should put the command script in the reference
> directory also, so you just need the direcory... there would be some
> duplication (since it would be harder to share a common script across
> tests) but in reality I'm not sure that ever bought us much.
I think that we should support this, but not require it. Something
like UnitTest('test/path', script=foo). I'd like to even support
putting the SConscript into the test directory itself.
> So we could have something more like this (just using current paths
> for backward compatibility):
>
> Test('quick/00.hello/ref/%s/linux/simple-atomic' % env['TARGET_ISA'])
>
> if env['TARGET_ISA'] == 'alpha':
> Test('quick/00.hello/ref/alpha/tru64/simple-atomic')
I think we have to avoid this because currently TARGET_ISA is inferred
(something I'd like to fix). Since it is inferred, there's no easy
way to force dependencies to be checked and built.
> If you wanted to get crazier, you could use something like a dict to
> handle exceptional cases a little better (e.g., each ISA name maps to
> a list of configs that have 00.hello tests, or something like that).
> The nice thing here is that even though I haven't changed the paths,
> the paths themselves have no inherent semantics; I'm leveraging their
> current structure to simplify the SConscript where they get
> enumerated, but they can easily be changed, and different groups can
> have different structures.
I'm not quite sure what you're suggesting, but I think we need to
figure out an alternative for the above idea.
> The one thing I can think of that would be nice to support more
> automatically is checking for binaries, disk images, etc. ... since
> those are downloaded separately, it would be nice to cleanly skip
> tests for which those aren't found (with a warning message, I think).
> Also we need to make sure the paths for finding those things stay
> flexible (e.g., using environment vars).
Absolutely.
> As far as groups, is there anything other than "quick" and "all" that
> we ever want to use them for? If not, I'd suggest a more numerical
> approach, where we assign each test a score (could be an approximate
> runtime, or some sort of priority), and then I say "run all tests
> shorter than 5 minutes" or "run all tests priority 2 and higher".
> That's more scalable than having to assign individual tests to groups;
> I'm not sure how that assignment idea would work with tests added via
> EXTRAS, for example.
I'd rather not constrain things. Groups should be easy to implement
and I want to make it simple to add groups. Let's say you're working
on something specific and you have a set of tests that have been
failing. It would be nice to specify a temporary group to build a
certain set of tests.
> These are all just brainstorming thoughts, none are ideas I'm really
> wedded to... if we put all our ideas together I'm sure we'll come up
> with a good solution.
Thanks for the help. Let's just hope I can get it all done.
Nate
_______________________________________________
m5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/m5-dev
