Hi Gabe,

Thanks for the feedback, and we look forward to hearing your thoughts on
the code.

Two replies to your comment:
1. I would argue that our current infrastructure isn't much better than
nothing. It sounds like internally at ARM and AMD (maybe other companies?),
they have a customized version that tests what is important to them.
However, for all of us not at a company, it's *very* hard to run the tests.
So hard, in fact, that I personally never run them, and I have 7 years of
gem5 experience! I've talked to a lot of gem5 users, and as it stands very
few feel comfortable executing and understanding the output of the current

2. The above doesn't really matter. It was just a rant :). We plan on
porting over most of the current tests before pushing the new framework.
Once we feel comfortable that the community is accepting of the new
framework, we'll start working on porting more tests.

We were thinking of keeping most of the tests that execute unique things.
Importantly, we were going to drop all SPEC tests. Hopefully we will have a
replacement for them, but probably not in the very short term. The only
time SPEC tests were run was on zizzer, so it was very difficult for our
users to get feedback about how their change affected those tests.

It would be great if we had more tests like insttest for RISC-V that
actually tests the functionality of specific simulator components. We are
also hoping the increased flexibility of the new infrastructure will
encourage the creation of unittests and other tests that are not full
integration tests.


On Fri, Aug 4, 2017 at 7:20 PM Gabe Black <gabebl...@google.com> wrote:

> I haven't had a chance to look at any of the documentation you referenced
> and am generally in favor of improvements to our testing setup, but it
> might be a bit premature to delete the old system if the new one is still a
> work in progress. That would take us from a not that great testing system
> to essentially not having one...
> Gabe
> On Fri, Aug 4, 2017 at 2:31 PM, Sean Wilson <spwils...@wisc.edu> wrote:
> > Hi all,
> >
> > I’ve just pushed a patchset which removes the current testing
> > implementation and replaces it with a work in progress test framework.
> This
> > framework is meant to be a strong starting point for a new testing
> > infrastructure which would allow for more test coverage and an
> > easier/better user experience. The new framework is completely functional
> > as it stands, but I am open to suggestions, criticism and change requests
> > that might improve it further.
> >
> >
> > Just to briefly motivate this patch, there are quite a few reasons I
> > believe the current testing framework is not suitable and is worth
> > completely replacing. The main reason is that there are too many
> different
> > ways to access the testing system, each of which is documented
> separately.
> > The biggest cause of this is the reliance on SCons to enumerate and
> provide
> > the dependencies for the tests. This proposed framework removes that
> > dependency and provides its own method for tracking dependencies -
> > fixtures. (These are described in more detail in the documentation.)
> >
> >
> > In addition to the patch directly applied to gem5 found
> > https://gem5-review.googlesource.com/c/4420. You can find the library in
> > its own development branch on github here https://github.com/spwilson2/
> > whimsy. The documentation has already been generated and can be browsed
> > here https://spwilson2.github.io/whimsy/. In case the paragraph above
> > wasn’t enough, the first few sections of the documentation provide some
> > more motivation for this library before introducing the framework in
> full.
> >
> > Finally, I only have a week left that I will have dedicated to
> development
> > on this project, so I appreciate hearing comments sooner rather than
> later!
> >
> > Thanks all!
> >
> >
> > - Sean Wilson
> >
> >
> > _______________________________________________
> > gem5-dev mailing list
> > gem5-dev@gem5.org
> > http://m5sim.org/mailman/listinfo/gem5-dev
> _______________________________________________
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
gem5-dev mailing list

Reply via email to