Re: [gem5-dev] New testing infrastructure

[Jason Lowe-Power]

Hi Gabe,

Thanks for the feedback, and we look forward to hearing your thoughts on
the code.

Two replies to your comment:
1. I would argue that our current infrastructure isn't much better than
nothing. It sounds like internally at ARM and AMD (maybe other companies?),
they have a customized version that tests what is important to them.
However, for all of us not at a company, it's *very* hard to run the tests.
So hard, in fact, that I personally never run them, and I have 7 years of
gem5 experience! I've talked to a lot of gem5 users, and as it stands very
few feel comfortable executing and understanding the output of the current
framework.

2. The above doesn't really matter. It was just a rant :). We plan on
porting over most of the current tests before pushing the new framework.
Once we feel comfortable that the community is accepting of the new
framework, we'll start working on porting more tests.

We were thinking of keeping most of the tests that execute unique things.
Importantly, we were going to drop all SPEC tests. Hopefully we will have a
replacement for them, but probably not in the very short term. The only
time SPEC tests were run was on zizzer, so it was very difficult for our
users to get feedback about how their change affected those tests.

It would be great if we had more tests like insttest for RISC-V that
actually tests the functionality of specific simulator components. We are
also hoping the increased flexibility of the new infrastructure will
encourage the creation of unittests and other tests that are not full
integration tests.

Cheers,
Jason

On Fri, Aug 4, 2017 at 7:20 PM Gabe Black  wrote:

> I haven't had a chance to look at any of the documentation you referenced
> and am generally in favor of improvements to our testing setup, but it
> might be a bit premature to delete the old system if the new one is still a
> work in progress. That would take us from a not that great testing system
> to essentially not having one...
>
> Gabe
>
> On Fri, Aug 4, 2017 at 2:31 PM, Sean Wilson  wrote:
>
> > Hi all,
> >
> > I’ve just pushed a patchset which removes the current testing
> > implementation and replaces it with a work in progress test framework.
> This
> > framework is meant to be a strong starting point for a new testing
> > infrastructure which would allow for more test coverage and an
> > easier/better user experience. The new framework is completely functional
> > as it stands, but I am open to suggestions, criticism and change requests
> > that might improve it further.
> >
> >
> > Just to briefly motivate this patch, there are quite a few reasons I
> > believe the current testing framework is not suitable and is worth
> > completely replacing. The main reason is that there are too many
> different
> > ways to access the testing system, each of which is documented
> separately.
> > The biggest cause of this is the reliance on SCons to enumerate and
> provide
> > the dependencies for the tests. This proposed framework removes that
> > dependency and provides its own method for tracking dependencies -
> > fixtures. (These are described in more detail in the documentation.)
> >
> >
> > In addition to the patch directly applied to gem5 found
> > https://gem5-review.googlesource.com/c/4420. You can find the library in
> > its own development branch on github here https://github.com/spwilson2/
> > whimsy. The documentation has already been generated and can be browsed
> > here https://spwilson2.github.io/whimsy/. In case the paragraph above
> > wasn’t enough, the first few sections of the documentation provide some
> > more motivation for this library before introducing the framework in
> full.
> >
> > Finally, I only have a week left that I will have dedicated to
> development
> > on this project, so I appreciate hearing comments sooner rather than
> later!
> >
> > Thanks all!
> >
> >
> > - Sean Wilson
> >
> >
> > ___
> > gem5-dev mailing list
> > gem5-dev@gem5.org
> > http://m5sim.org/mailman/listinfo/gem5-dev
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] New testing infrastructure

[Gabe Black]

I haven't had a chance to look at any of the documentation you referenced
and am generally in favor of improvements to our testing setup, but it
might be a bit premature to delete the old system if the new one is still a
work in progress. That would take us from a not that great testing system
to essentially not having one...

Gabe

On Fri, Aug 4, 2017 at 2:31 PM, Sean Wilson  wrote:

> Hi all,
>
> I’ve just pushed a patchset which removes the current testing
> implementation and replaces it with a work in progress test framework. This
> framework is meant to be a strong starting point for a new testing
> infrastructure which would allow for more test coverage and an
> easier/better user experience. The new framework is completely functional
> as it stands, but I am open to suggestions, criticism and change requests
> that might improve it further.
>
>
> Just to briefly motivate this patch, there are quite a few reasons I
> believe the current testing framework is not suitable and is worth
> completely replacing. The main reason is that there are too many different
> ways to access the testing system, each of which is documented separately.
> The biggest cause of this is the reliance on SCons to enumerate and provide
> the dependencies for the tests. This proposed framework removes that
> dependency and provides its own method for tracking dependencies -
> fixtures. (These are described in more detail in the documentation.)
>
>
> In addition to the patch directly applied to gem5 found
> https://gem5-review.googlesource.com/c/4420. You can find the library in
> its own development branch on github here https://github.com/spwilson2/
> whimsy. The documentation has already been generated and can be browsed
> here https://spwilson2.github.io/whimsy/. In case the paragraph above
> wasn’t enough, the first few sections of the documentation provide some
> more motivation for this library before introducing the framework in full.
>
> Finally, I only have a week left that I will have dedicated to development
> on this project, so I appreciate hearing comments sooner rather than later!
>
> Thanks all!
>
>
> - Sean Wilson
>
>
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

[gem5-dev] New testing infrastructure

[Sean Wilson]

Hi all,

I’ve just pushed a patchset which removes the current testing implementation 
and replaces it with a work in progress test framework. This framework is meant 
to be a strong starting point for a new testing infrastructure which would 
allow for more test coverage and an easier/better user experience. The new 
framework is completely functional as it stands, but I am open to suggestions, 
criticism and change requests that might improve it further.


Just to briefly motivate this patch, there are quite a few reasons I believe 
the current testing framework is not suitable and is worth completely 
replacing. The main reason is that there are too many different ways to access 
the testing system, each of which is documented separately. The biggest cause 
of this is the reliance on SCons to enumerate and provide the dependencies for 
the tests. This proposed framework removes that dependency and provides its own 
method for tracking dependencies - fixtures. (These are described in more 
detail in the documentation.)


In addition to the patch directly applied to gem5 found 
https://gem5-review.googlesource.com/c/4420. You can find the library in its 
own development branch on github here https://github.com/spwilson2/whimsy. The 
documentation has already been generated and can be browsed here 
https://spwilson2.github.io/whimsy/. In case the paragraph above wasn’t enough, 
the first few sections of the documentation provide some more motivation for 
this library before introducing the framework in full.

Finally, I only have a week left that I will have dedicated to development on 
this project, so I appreciate hearing comments sooner rather than later!

Thanks all!


- Sean Wilson


___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev