I think there are two important aspects of this issue.
1. Using regression tests we can't distribute freely has some
important limitations. It would be nice to replace them with ones we
can.
2. The majority of the regression tests we have now are really
benchmarks which provide basic coverage by working/not working and not
changing behavior unexpectedly. That's an important element to have
since it's a practical reality check and probably hits things we
wouldn't think to test. They have significant limitations, though,
since they take a long time to run and tend to exercise the same
simulator functionality over and over. For instance, gcc may generate
code that always has the same type of backward branch for a for loop.
Using gzip as a test will verify that that branch works, but possibly
not the slightly different variant that may, for instance, use a large
branch displacement. Even when writing code in x86 assembly it can be
impossible to predict which of the possibly many redundant instruction
encodings the assembler might pick.
So, in everyone's infinite free time, I think we should replace our
benchmark based regressions with a smaller set of freely distributable
regressions/inputs, and augment them with shorter, targeted tests that
exercise particular mechanisms, circumstances, instructions, etc.
Instead of replacing our existing benchmarks which are useful as
actual benchmarks and are good to keep working, we could build up this
second set of tests in parallel.
Gabe
Quoting "Beckmann, Brad" <brad.beckm...@amd.com>:
Hi Nilay,
I understand your confusion. This is an example of where the wiki
needs to be updated. I believe the wiki only mentions the
encumbered tar ball and doesn't mention the encumbered hg repo on
repo.m5sim.org. As far as the anagram test program goes, I remember
Lisa and I encountered the same issue a while back and to resolve it
I believe Lisa copied that test along with several other regression
tester programs from Michigan to AMD.
I can provide you those regression tester programs, but at a higher
level, I think this is a good time to ask the question on how we
want to provide external users all the files necessary to run the
regression tester? As Nilay points out, the encumbered repo has
some, but not all of the necessary files. I believe, one also needs
another set of regression tester programs which include both the
anagram files, as well as the SPECCPU files for the long regression
tester runs.
Thoughts?
Brad
-----Original Message-----
From: m5-dev-boun...@m5sim.org [mailto:m5-dev-boun...@m5sim.org] On
Behalf Of Nilay Vaish
Sent: Monday, January 17, 2011 1:55 PM
To: M5 Developer List
Subject: Re: [m5-dev] EIO Regression Tests
I figured that out, but there is no anagram directory in tests/test-
progs.
I, therefore, receive the following error:
gzip: tests/test-progs/anagram/bin/alpha/eio/anagram-vshort.eio.gz: No
such file or directory
--
Nilay
On Mon, 17 Jan 2011, Steve Reinhardt wrote:
> The one where the EIO code lives. That's it's name, at
> http://repo.m5sim.org.
>
> On Mon, Jan 17, 2011 at 12:59 PM, Nilay Vaish <ni...@cs.wisc.edu>
wrote:
>
>> What do you mean by the encumbered repository?
>>
>>
>> On Mon, 17 Jan 2011, Steve Reinhardt wrote:
>>
>> Yes, it should be a concern... it should work. Did you do a pull
on the
>>> encumbered repository? There were some changes there needed to
maintain
>>> compatibility with the latest m5 dev repo.
>>>
>>> Otherwise you'll need to provide more detail about how things
failed.
>>>
>>> Steve
>>>
>>> On Mon, Jan 17, 2011 at 10:21 AM, Nilay Vaish <ni...@cs.wisc.edu>
wrote:
>>>
>>> I just ran the regression tests for the patch (deals with SLICC
and cache
>>>> coherence protocols) that I need to commit. The EIO tests fail.
Should
>>>> this
>>>> be a concern?
>>>>
>>>> --
>>>> Nilay
>>>> _______________________________________________
>>>> m5-dev mailing list
>>>> m5-dev@m5sim.org
>>>> http://m5sim.org/mailman/listinfo/m5-dev
>>>>
>>>>
>>> _______________________________________________
>> m5-dev mailing list
>> m5-dev@m5sim.org
>> http://m5sim.org/mailman/listinfo/m5-dev
>>
>
_______________________________________________
m5-dev mailing list
m5-dev@m5sim.org
http://m5sim.org/mailman/listinfo/m5-dev
_______________________________________________
m5-dev mailing list
m5-dev@m5sim.org
http://m5sim.org/mailman/listinfo/m5-dev
_______________________________________________
m5-dev mailing list
m5-dev@m5sim.org
http://m5sim.org/mailman/listinfo/m5-dev
