Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ian Romanick]

There is at least one bigger bug in this series, and fixing it will
require changes to shader-runner too.  I had some e-mail either from one
of the guys at VMware or one of the guys at NVIDIA, but I can't find it.

My recollection is that there is some case in the generator (and
shader-runner in general) that expects non-base elements of arrays to be
advertised, but the spec says they should not be advertised.  I think
the case was that for a UBO like

struct S {
int i;
float f;
};

uniform U {
S s[4];
};

only s[0].i and s[0].f would be reported, but shader-runner expects all
elements of s[] to be reported.

This will be annoying to fix because shader-runner relies on this to
determine the offsets s[1].i and friends.

It should be relatively easy to figure out what the failure case is.  I
think there's already a test in piglit that misuses the feature, and
that test fails on NVIDIA's closed-source driver.

On 01/12/2016 03:09 PM, Ilia Mirkin wrote:
> My fixes are one-liners (might just be one, I forget). At this point I
> would very much like to see *anything* checked in, and we can improve
> from there.
> 
> On Tue, Jan 12, 2016 at 6:04 PM, Dylan Baker  wrote:
>> That sounds like a plan. Would you like to get Ian's original scripts
>> merged, and I'll update the cleanups I have for his scripts and send them
>> out?
>>
>> Dylan
>>
>> On Tue, Jan 12, 2016 at 2:41 PM, Timothy Arceri 
>> wrote:
>>>
>>> On Tue, 2015-11-10 at 16:46 -0800, Dylan Baker wrote:
 Since gravy is so delicious,

 https://github.com/dcbaker/piglit.git wip/ubo-fuzzer

 bin/piglit run ubo-fuzzer output

 Obviously it's not at all feature complete, it's more proof of
 concept
 than anything else, but gravy.
>>>
>>> Hi Dylan,
>>>
>>> I took a quick look at your branch and I'd like to suggest a way
>>> forward on this.
>>>
>>> It looks like you made a bunch of small/fixes tidy ups to Ian's
>>> scripts, how about we get those sent out for review/comment then commit
>>> the script to the main piglit repo.
>>>
>>> Ilia and I can then rebase our modifications, resend for review and
>>> commit those.
>>>
>>> Then we can worry about gravy once we have all the work in one place.
>>>
>>> How does that sound?
>>>
>>> Tim
>>>
 On Tue, Nov 10, 2015 at 03:01 11AM -0500, Ilia Mirkin wrote:
> That's a full step ahead of my suggestion. I just want the thing in
> a
> shared place so that I can run it when doing dodgy things to code I
> don't understand... if there's some automated process running it,
> that's just gravy on top.
>
 [snip]
>>
> ___
> Piglit mailing list
> Piglit@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit

___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Dylan Baker]

I seem to remember this, it was nvidia I think, if that helps you look for
the email.

On Wed, Jan 13, 2016 at 11:39 AM, Ian Romanick  wrote:

> There is at least one bigger bug in this series, and fixing it will
> require changes to shader-runner too.  I had some e-mail either from one
> of the guys at VMware or one of the guys at NVIDIA, but I can't find it.
>
> My recollection is that there is some case in the generator (and
> shader-runner in general) that expects non-base elements of arrays to be
> advertised, but the spec says they should not be advertised.  I think
> the case was that for a UBO like
>
> struct S {
> int i;
> float f;
> };
>
> uniform U {
> S s[4];
> };
>
> only s[0].i and s[0].f would be reported, but shader-runner expects all
> elements of s[] to be reported.
>
> This will be annoying to fix because shader-runner relies on this to
> determine the offsets s[1].i and friends.
>
> It should be relatively easy to figure out what the failure case is.  I
> think there's already a test in piglit that misuses the feature, and
> that test fails on NVIDIA's closed-source driver.
>
> On 01/12/2016 03:09 PM, Ilia Mirkin wrote:
> > My fixes are one-liners (might just be one, I forget). At this point I
> > would very much like to see *anything* checked in, and we can improve
> > from there.
> >
> > On Tue, Jan 12, 2016 at 6:04 PM, Dylan Baker 
> wrote:
> >> That sounds like a plan. Would you like to get Ian's original scripts
> >> merged, and I'll update the cleanups I have for his scripts and send
> them
> >> out?
> >>
> >> Dylan
> >>
> >> On Tue, Jan 12, 2016 at 2:41 PM, Timothy Arceri 
> >> wrote:
> >>>
> >>> On Tue, 2015-11-10 at 16:46 -0800, Dylan Baker wrote:
>  Since gravy is so delicious,
> 
>  https://github.com/dcbaker/piglit.git wip/ubo-fuzzer
> 
>  bin/piglit run ubo-fuzzer output
> 
>  Obviously it's not at all feature complete, it's more proof of
>  concept
>  than anything else, but gravy.
> >>>
> >>> Hi Dylan,
> >>>
> >>> I took a quick look at your branch and I'd like to suggest a way
> >>> forward on this.
> >>>
> >>> It looks like you made a bunch of small/fixes tidy ups to Ian's
> >>> scripts, how about we get those sent out for review/comment then commit
> >>> the script to the main piglit repo.
> >>>
> >>> Ilia and I can then rebase our modifications, resend for review and
> >>> commit those.
> >>>
> >>> Then we can worry about gravy once we have all the work in one place.
> >>>
> >>> How does that sound?
> >>>
> >>> Tim
> >>>
>  On Tue, Nov 10, 2015 at 03:01 11AM -0500, Ilia Mirkin wrote:
> > That's a full step ahead of my suggestion. I just want the thing in
> > a
> > shared place so that I can run it when doing dodgy things to code I
> > don't understand... if there's some automated process running it,
> > that's just gravy on top.
> >
>  [snip]
> >>
> > ___
> > Piglit mailing list
> > Piglit@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/piglit
>
>
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ian Romanick]

On 01/13/2016 01:34 PM, Dylan Baker wrote:
> I seem to remember this, it was nvidia I think, if that helps you look
> for the email.

Looking at old messages in the thread... it looks like this was fixed in
shader-runner in
http://cgit.freedesktop.org/piglit/commit/?id=2b94faec18dc1f8f0d9241ec731408959320cd7c.

> On Wed, Jan 13, 2016 at 11:39 AM, Ian Romanick  > wrote:
> 
> There is at least one bigger bug in this series, and fixing it will
> require changes to shader-runner too.  I had some e-mail either from one
> of the guys at VMware or one of the guys at NVIDIA, but I can't find it.
> 
> My recollection is that there is some case in the generator (and
> shader-runner in general) that expects non-base elements of arrays to be
> advertised, but the spec says they should not be advertised.  I think
> the case was that for a UBO like
> 
> struct S {
> int i;
> float f;
> };
> 
> uniform U {
> S s[4];
> };
> 
> only s[0].i and s[0].f would be reported, but shader-runner expects all
> elements of s[] to be reported.
> 
> This will be annoying to fix because shader-runner relies on this to
> determine the offsets s[1].i and friends.
> 
> It should be relatively easy to figure out what the failure case is.  I
> think there's already a test in piglit that misuses the feature, and
> that test fails on NVIDIA's closed-source driver.
> 
> On 01/12/2016 03:09 PM, Ilia Mirkin wrote:
> > My fixes are one-liners (might just be one, I forget). At this point I
> > would very much like to see *anything* checked in, and we can improve
> > from there.
> >
> > On Tue, Jan 12, 2016 at 6:04 PM, Dylan Baker
> > wrote:
> >> That sounds like a plan. Would you like to get Ian's original scripts
> >> merged, and I'll update the cleanups I have for his scripts and
> send them
> >> out?
> >>
> >> Dylan
> >>
> >> On Tue, Jan 12, 2016 at 2:41 PM, Timothy Arceri
> >
> >> wrote:
> >>>
> >>> On Tue, 2015-11-10 at 16:46 -0800, Dylan Baker wrote:
>  Since gravy is so delicious,
> 
>  https://github.com/dcbaker/piglit.git wip/ubo-fuzzer
> 
>  bin/piglit run ubo-fuzzer output
> 
>  Obviously it's not at all feature complete, it's more proof of
>  concept
>  than anything else, but gravy.
> >>>
> >>> Hi Dylan,
> >>>
> >>> I took a quick look at your branch and I'd like to suggest a way
> >>> forward on this.
> >>>
> >>> It looks like you made a bunch of small/fixes tidy ups to Ian's
> >>> scripts, how about we get those sent out for review/comment then
> commit
> >>> the script to the main piglit repo.
> >>>
> >>> Ilia and I can then rebase our modifications, resend for review and
> >>> commit those.
> >>>
> >>> Then we can worry about gravy once we have all the work in one
> place.
> >>>
> >>> How does that sound?
> >>>
> >>> Tim
> >>>
>  On Tue, Nov 10, 2015 at 03:01 11AM -0500, Ilia Mirkin wrote:
> > That's a full step ahead of my suggestion. I just want the
> thing in
> > a
> > shared place so that I can run it when doing dodgy things to
> code I
> > don't understand... if there's some automated process running it,
> > that's just gravy on top.
> >
>  [snip]
> >>
> > ___
> > Piglit mailing list
> > Piglit@lists.freedesktop.org 
> > http://lists.freedesktop.org/mailman/listinfo/piglit
> 
> 

___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Timothy Arceri]

On Tue, 2015-11-10 at 16:46 -0800, Dylan Baker wrote:
> Since gravy is so delicious,
> 
> https://github.com/dcbaker/piglit.git wip/ubo-fuzzer
> 
> bin/piglit run ubo-fuzzer output
> 
> Obviously it's not at all feature complete, it's more proof of
> concept
> than anything else, but gravy.

Hi Dylan,

I took a quick look at your branch and I'd like to suggest a way
forward on this.

It looks like you made a bunch of small/fixes tidy ups to Ian's
scripts, how about we get those sent out for review/comment then commit
the script to the main piglit repo. 

Ilia and I can then rebase our modifications, resend for review and
commit those.

Then we can worry about gravy once we have all the work in one place.

How does that sound?

Tim


> 
> On Tue, Nov 10, 2015 at 03:01 11AM -0500, Ilia Mirkin wrote:
> > That's a full step ahead of my suggestion. I just want the thing in
> > a
> > shared place so that I can run it when doing dodgy things to code I
> > don't understand... if there's some automated process running it,
> > that's just gravy on top.
> > 
> [snip]
> ___
> Piglit mailing list
> Piglit@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Dylan Baker]

That sounds like a plan. Would you like to get Ian's original scripts
merged, and I'll update the cleanups I have for his scripts and send them
out?

Dylan

On Tue, Jan 12, 2016 at 2:41 PM, Timothy Arceri 
wrote:

> On Tue, 2015-11-10 at 16:46 -0800, Dylan Baker wrote:
> > Since gravy is so delicious,
> >
> > https://github.com/dcbaker/piglit.git wip/ubo-fuzzer
> >
> > bin/piglit run ubo-fuzzer output
> >
> > Obviously it's not at all feature complete, it's more proof of
> > concept
> > than anything else, but gravy.
>
> Hi Dylan,
>
> I took a quick look at your branch and I'd like to suggest a way
> forward on this.
>
> It looks like you made a bunch of small/fixes tidy ups to Ian's
> scripts, how about we get those sent out for review/comment then commit
> the script to the main piglit repo.
>
> Ilia and I can then rebase our modifications, resend for review and
> commit those.
>
> Then we can worry about gravy once we have all the work in one place.
>
> How does that sound?
>
> Tim
>
>
> >
> > On Tue, Nov 10, 2015 at 03:01 11AM -0500, Ilia Mirkin wrote:
> > > That's a full step ahead of my suggestion. I just want the thing in
> > > a
> > > shared place so that I can run it when doing dodgy things to code I
> > > don't understand... if there's some automated process running it,
> > > that's just gravy on top.
> > >
> > [snip]
> > ___
> > Piglit mailing list
> > Piglit@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/piglit
>
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ilia Mirkin]

My fixes are one-liners (might just be one, I forget). At this point I
would very much like to see *anything* checked in, and we can improve
from there.

On Tue, Jan 12, 2016 at 6:04 PM, Dylan Baker  wrote:
> That sounds like a plan. Would you like to get Ian's original scripts
> merged, and I'll update the cleanups I have for his scripts and send them
> out?
>
> Dylan
>
> On Tue, Jan 12, 2016 at 2:41 PM, Timothy Arceri 
> wrote:
>>
>> On Tue, 2015-11-10 at 16:46 -0800, Dylan Baker wrote:
>> > Since gravy is so delicious,
>> >
>> > https://github.com/dcbaker/piglit.git wip/ubo-fuzzer
>> >
>> > bin/piglit run ubo-fuzzer output
>> >
>> > Obviously it's not at all feature complete, it's more proof of
>> > concept
>> > than anything else, but gravy.
>>
>> Hi Dylan,
>>
>> I took a quick look at your branch and I'd like to suggest a way
>> forward on this.
>>
>> It looks like you made a bunch of small/fixes tidy ups to Ian's
>> scripts, how about we get those sent out for review/comment then commit
>> the script to the main piglit repo.
>>
>> Ilia and I can then rebase our modifications, resend for review and
>> commit those.
>>
>> Then we can worry about gravy once we have all the work in one place.
>>
>> How does that sound?
>>
>> Tim
>>
>>
>> >
>> > On Tue, Nov 10, 2015 at 03:01 11AM -0500, Ilia Mirkin wrote:
>> > > That's a full step ahead of my suggestion. I just want the thing in
>> > > a
>> > > shared place so that I can run it when doing dodgy things to code I
>> > > don't understand... if there's some automated process running it,
>> > > that's just gravy on top.
>> > >
>> > [snip]
>> > ___
>> > Piglit mailing list
>> > Piglit@lists.freedesktop.org
>> > http://lists.freedesktop.org/mailman/listinfo/piglit
>
>
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Dylan Baker]

That is the plan I think? Timothy will merge Ian's scripts, and then we can
rebase cleanups and send them out for review?

On Tue, Jan 12, 2016 at 3:09 PM, Ilia Mirkin  wrote:

> My fixes are one-liners (might just be one, I forget). At this point I
> would very much like to see *anything* checked in, and we can improve
> from there.
>
> On Tue, Jan 12, 2016 at 6:04 PM, Dylan Baker 
> wrote:
> > That sounds like a plan. Would you like to get Ian's original scripts
> > merged, and I'll update the cleanups I have for his scripts and send them
> > out?
> >
> > Dylan
> >
> > On Tue, Jan 12, 2016 at 2:41 PM, Timothy Arceri 
> > wrote:
> >>
> >> On Tue, 2015-11-10 at 16:46 -0800, Dylan Baker wrote:
> >> > Since gravy is so delicious,
> >> >
> >> > https://github.com/dcbaker/piglit.git wip/ubo-fuzzer
> >> >
> >> > bin/piglit run ubo-fuzzer output
> >> >
> >> > Obviously it's not at all feature complete, it's more proof of
> >> > concept
> >> > than anything else, but gravy.
> >>
> >> Hi Dylan,
> >>
> >> I took a quick look at your branch and I'd like to suggest a way
> >> forward on this.
> >>
> >> It looks like you made a bunch of small/fixes tidy ups to Ian's
> >> scripts, how about we get those sent out for review/comment then commit
> >> the script to the main piglit repo.
> >>
> >> Ilia and I can then rebase our modifications, resend for review and
> >> commit those.
> >>
> >> Then we can worry about gravy once we have all the work in one place.
> >>
> >> How does that sound?
> >>
> >> Tim
> >>
> >>
> >> >
> >> > On Tue, Nov 10, 2015 at 03:01 11AM -0500, Ilia Mirkin wrote:
> >> > > That's a full step ahead of my suggestion. I just want the thing in
> >> > > a
> >> > > shared place so that I can run it when doing dodgy things to code I
> >> > > don't understand... if there's some automated process running it,
> >> > > that's just gravy on top.
> >> > >
> >> > [snip]
> >> > ___
> >> > Piglit mailing list
> >> > Piglit@lists.freedesktop.org
> >> > http://lists.freedesktop.org/mailman/listinfo/piglit
> >
> >
>
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ilia Mirkin]

That's a full step ahead of my suggestion. I just want the thing in a
shared place so that I can run it when doing dodgy things to code I
don't understand... if there's some automated process running it,
that's just gravy on top.

On Tue, Nov 10, 2015 at 1:26 AM, Dylan Baker  wrote:
> Here's my idea.
>
> We add ubo-fuzzer as a profile. We generate each test file on demand, and
> write it out to a temporary file before running it. Then if the test fails
> we can record the test into the results file for the developer to come back
> and look at later to decide if it's worth keeping. Does this sound like a
> reasonable approach?
>
> On Mon, Nov 9, 2015 at 5:08 PM, Ian Romanick  wrote:
>>
>> On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
>> > On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick 
>> > wrote:
>> >> On 09/24/2014 09:47 AM, Ian Romanick wrote:
>> >>> So, here it is.  Finally.
>> >>>
>> >>> The first two patches provide the infrastructure for generating
>> >>> randomized UBO tests.  I think these are pretty solid, but there are
>> >>> probably ways to impove the Python, etc.
>> >>>
>> >>> The remaining three patches are examples of ways the infrastructure
>> >>> can
>> >>> be used.  Here is where I am not sure what we should do.  I know that
>> >>> we
>> >>> don't want to make the "forever" test in patch 4 part of regular
>> >>> piglit
>> >>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that
>> >>> I
>> >>> have tested.
>> >>>
>> >>> I'm also unsure about the random tests generated by patch 3.  Do we
>> >>> want
>> >>> actual random tests in regular piglit runs?  What do we do for tests
>> >>> for
>> >>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
>> >>> #extension?
>> >>>
>> >>> In any case, I know that folks are hard at work on fp64 support, so
>> >>> using the various random runners here should help that effort.  Sorry
>> >>> for all the delays.
>> >>>
>> >>> One last thing... I'm presenting a bunch of information about this
>> >>> work
>> >>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the
>> >>> more
>> >>> difficult details until then.  Dunno.
>> >>
>> >> I've pushed updated version of this series to the ubo-lolz branch of my
>> >> fd.o piglit repo.
>> >
>> > It looks like this didn't end up going anywhere... on several
>> > occasions I've either used this script (like for fp64), or recommended
>> > it to others (like for ssbo, and will do so for ARB_enhanced_layouts
>> > when that conversation comes up).
>> >
>> > I think it'd benefit greatly from being in a shared and updated
>> > location as features are added, bugs are fixed, etc. However running
>> > it as part of piglit may not be a great idea. Perhaps we can find a
>> > place in the repo where we can store it? Or maybe even a different
>> > repo?
>> >
>> > How about tests/fuzzing in piglit? Any objections?
>>
>> Having it actually live somewhere is a good idea.  There are definitely
>> some bugs in it... and some of the tests that failed on other
>> implementations may be expecting things the spec doesn't allow.  I need
>> to dig back through my e-mail, but some guys from NVIDIA had convinced
>> me that there was something wrong... but I don't recall what.
>>
>> When I presented this at XDC in 2014, I think the consensus was that
>> tests that actually found a bug should be added to the "right" place in
>> the repo, but we don't want to run 47,000,000 random tests on regular
>> piglit runs.  Someone should have a system somewhere that just runs
>> these (and other) random, fuzzing tests 24/7.
>>
>> > Cheers,
>> >
>> >   -ilia
>>
>> ___
>> Piglit mailing list
>> Piglit@lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/piglit
>
>
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Arthur Huillet]

On 2015-11-10 2:08, Ian Romanick wrote:

On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick  
wrote:

On 09/24/2014 09:47 AM, Ian Romanick wrote:

So, here it is.  Finally.

The first two patches provide the infrastructure for generating
randomized UBO tests.  I think these are pretty solid, but there are
probably ways to impove the Python, etc.

The remaining three patches are examples of ways the infrastructure 
can
be used.  Here is where I am not sure what we should do.  I know 
that we
don't want to make the "forever" test in patch 4 part of regular 
piglit
runs.  However, it has found a LOT of bugs in EVERY OpenGL driver 
that I

have tested.

I'm also unsure about the random tests generated by patch 3.  Do we 
want
actual random tests in regular piglit runs?  What do we do for tests 
for
GLSL 1.40?  Generate the "same" tests, but use #version 140 instead 
of

#extension?

In any case, I know that folks are hard at work on fp64 support, so
using the various random runners here should help that effort.  
Sorry

for all the delays.

One last thing... I'm presenting a bunch of information about this 
work
at XDC in a couple weeks.  Maybe we want to wait to hammer out the 
more

difficult details until then.  Dunno.


I've pushed updated version of this series to the ubo-lolz branch of 
my

fd.o piglit repo.


It looks like this didn't end up going anywhere... on several
occasions I've either used this script (like for fp64), or recommended
it to others (like for ssbo, and will do so for ARB_enhanced_layouts
when that conversation comes up).

I think it'd benefit greatly from being in a shared and updated
location as features are added, bugs are fixed, etc. However running
it as part of piglit may not be a great idea. Perhaps we can find a
place in the repo where we can store it? Or maybe even a different
repo?

How about tests/fuzzing in piglit? Any objections?


Having it actually live somewhere is a good idea.  There are definitely
some bugs in it... and some of the tests that failed on other
implementations may be expecting things the spec doesn't allow.  I need
to dig back through my e-mail, but some guys from NVIDIA had convinced
me that there was something wrong... but I don't recall what.


I am "some guys" (or at least one of them). IIRC, the bug was in 
shader-runner, not in your fuzzer per se, and got fixed in 
http://cgit.freedesktop.org/piglit/commit/?id=2b94faec18dc1f8f0d9241ec731408959320cd7c


I don't recall finding other bugs, there was one in the NVIDIA driver 
that it exposed and we fixed it. (Thanks!)



When I presented this at XDC in 2014, I think the consensus was that
tests that actually found a bug should be added to the "right" place in
the repo, but we don't want to run 47,000,000 random tests on regular
piglit runs.


I think it makes sense to only store the tests that are, or used to be, 
red, and not store the tests that always passed everywhere, as these 
bring no particular value.


--
Greetings,
A. Huillet
NVIDIA Linux graphics
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ilia Mirkin]

On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick  wrote:
> On 09/24/2014 09:47 AM, Ian Romanick wrote:
>> So, here it is.  Finally.
>>
>> The first two patches provide the infrastructure for generating
>> randomized UBO tests.  I think these are pretty solid, but there are
>> probably ways to impove the Python, etc.
>>
>> The remaining three patches are examples of ways the infrastructure can
>> be used.  Here is where I am not sure what we should do.  I know that we
>> don't want to make the "forever" test in patch 4 part of regular piglit
>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
>> have tested.
>>
>> I'm also unsure about the random tests generated by patch 3.  Do we want
>> actual random tests in regular piglit runs?  What do we do for tests for
>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
>> #extension?
>>
>> In any case, I know that folks are hard at work on fp64 support, so
>> using the various random runners here should help that effort.  Sorry
>> for all the delays.
>>
>> One last thing... I'm presenting a bunch of information about this work
>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the more
>> difficult details until then.  Dunno.
>
> I've pushed updated version of this series to the ubo-lolz branch of my
> fd.o piglit repo.

It looks like this didn't end up going anywhere... on several
occasions I've either used this script (like for fp64), or recommended
it to others (like for ssbo, and will do so for ARB_enhanced_layouts
when that conversation comes up).

I think it'd benefit greatly from being in a shared and updated
location as features are added, bugs are fixed, etc. However running
it as part of piglit may not be a great idea. Perhaps we can find a
place in the repo where we can store it? Or maybe even a different
repo?

How about tests/fuzzing in piglit? Any objections?

Cheers,

  -ilia
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Dylan Baker]

Why don't we make a ubo-fuzzing profile that Sandra the rest as part of the
output? If be happy to do that if others think it's useful
On Nov 9, 2015 05:19, "Ilia Mirkin"  wrote:

> On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick  wrote:
> > On 09/24/2014 09:47 AM, Ian Romanick wrote:
> >> So, here it is.  Finally.
> >>
> >> The first two patches provide the infrastructure for generating
> >> randomized UBO tests.  I think these are pretty solid, but there are
> >> probably ways to impove the Python, etc.
> >>
> >> The remaining three patches are examples of ways the infrastructure can
> >> be used.  Here is where I am not sure what we should do.  I know that we
> >> don't want to make the "forever" test in patch 4 part of regular piglit
> >> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
> >> have tested.
> >>
> >> I'm also unsure about the random tests generated by patch 3.  Do we want
> >> actual random tests in regular piglit runs?  What do we do for tests for
> >> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
> >> #extension?
> >>
> >> In any case, I know that folks are hard at work on fp64 support, so
> >> using the various random runners here should help that effort.  Sorry
> >> for all the delays.
> >>
> >> One last thing... I'm presenting a bunch of information about this work
> >> at XDC in a couple weeks.  Maybe we want to wait to hammer out the more
> >> difficult details until then.  Dunno.
> >
> > I've pushed updated version of this series to the ubo-lolz branch of my
> > fd.o piglit repo.
>
> It looks like this didn't end up going anywhere... on several
> occasions I've either used this script (like for fp64), or recommended
> it to others (like for ssbo, and will do so for ARB_enhanced_layouts
> when that conversation comes up).
>
> I think it'd benefit greatly from being in a shared and updated
> location as features are added, bugs are fixed, etc. However running
> it as part of piglit may not be a great idea. Perhaps we can find a
> place in the repo where we can store it? Or maybe even a different
> repo?
>
> How about tests/fuzzing in piglit? Any objections?
>
> Cheers,
>
>   -ilia
> ___
> Piglit mailing list
> Piglit@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
>
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Dylan Baker]

Here's my idea.

We add ubo-fuzzer as a profile. We generate each test file on demand, and
write it out to a temporary file before running it. Then if the test fails
we can record the test into the results file for the developer to come back
and look at later to decide if it's worth keeping. Does this sound like a
reasonable approach?

On Mon, Nov 9, 2015 at 5:08 PM, Ian Romanick  wrote:

> On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
> > On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick 
> wrote:
> >> On 09/24/2014 09:47 AM, Ian Romanick wrote:
> >>> So, here it is.  Finally.
> >>>
> >>> The first two patches provide the infrastructure for generating
> >>> randomized UBO tests.  I think these are pretty solid, but there are
> >>> probably ways to impove the Python, etc.
> >>>
> >>> The remaining three patches are examples of ways the infrastructure can
> >>> be used.  Here is where I am not sure what we should do.  I know that
> we
> >>> don't want to make the "forever" test in patch 4 part of regular piglit
> >>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that
> I
> >>> have tested.
> >>>
> >>> I'm also unsure about the random tests generated by patch 3.  Do we
> want
> >>> actual random tests in regular piglit runs?  What do we do for tests
> for
> >>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
> >>> #extension?
> >>>
> >>> In any case, I know that folks are hard at work on fp64 support, so
> >>> using the various random runners here should help that effort.  Sorry
> >>> for all the delays.
> >>>
> >>> One last thing... I'm presenting a bunch of information about this work
> >>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the more
> >>> difficult details until then.  Dunno.
> >>
> >> I've pushed updated version of this series to the ubo-lolz branch of my
> >> fd.o piglit repo.
> >
> > It looks like this didn't end up going anywhere... on several
> > occasions I've either used this script (like for fp64), or recommended
> > it to others (like for ssbo, and will do so for ARB_enhanced_layouts
> > when that conversation comes up).
> >
> > I think it'd benefit greatly from being in a shared and updated
> > location as features are added, bugs are fixed, etc. However running
> > it as part of piglit may not be a great idea. Perhaps we can find a
> > place in the repo where we can store it? Or maybe even a different
> > repo?
> >
> > How about tests/fuzzing in piglit? Any objections?
>
> Having it actually live somewhere is a good idea.  There are definitely
> some bugs in it... and some of the tests that failed on other
> implementations may be expecting things the spec doesn't allow.  I need
> to dig back through my e-mail, but some guys from NVIDIA had convinced
> me that there was something wrong... but I don't recall what.
>
> When I presented this at XDC in 2014, I think the consensus was that
> tests that actually found a bug should be added to the "right" place in
> the repo, but we don't want to run 47,000,000 random tests on regular
> piglit runs.  Someone should have a system somewhere that just runs
> these (and other) random, fuzzing tests 24/7.
>
> > Cheers,
> >
> >   -ilia
>
> ___
> Piglit mailing list
> Piglit@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
>
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ian Romanick]

On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
> On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick  wrote:
>> On 09/24/2014 09:47 AM, Ian Romanick wrote:
>>> So, here it is.  Finally.
>>>
>>> The first two patches provide the infrastructure for generating
>>> randomized UBO tests.  I think these are pretty solid, but there are
>>> probably ways to impove the Python, etc.
>>>
>>> The remaining three patches are examples of ways the infrastructure can
>>> be used.  Here is where I am not sure what we should do.  I know that we
>>> don't want to make the "forever" test in patch 4 part of regular piglit
>>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
>>> have tested.
>>>
>>> I'm also unsure about the random tests generated by patch 3.  Do we want
>>> actual random tests in regular piglit runs?  What do we do for tests for
>>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
>>> #extension?
>>>
>>> In any case, I know that folks are hard at work on fp64 support, so
>>> using the various random runners here should help that effort.  Sorry
>>> for all the delays.
>>>
>>> One last thing... I'm presenting a bunch of information about this work
>>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the more
>>> difficult details until then.  Dunno.
>>
>> I've pushed updated version of this series to the ubo-lolz branch of my
>> fd.o piglit repo.
> 
> It looks like this didn't end up going anywhere... on several
> occasions I've either used this script (like for fp64), or recommended
> it to others (like for ssbo, and will do so for ARB_enhanced_layouts
> when that conversation comes up).
> 
> I think it'd benefit greatly from being in a shared and updated
> location as features are added, bugs are fixed, etc. However running
> it as part of piglit may not be a great idea. Perhaps we can find a
> place in the repo where we can store it? Or maybe even a different
> repo?
> 
> How about tests/fuzzing in piglit? Any objections?

Having it actually live somewhere is a good idea.  There are definitely
some bugs in it... and some of the tests that failed on other
implementations may be expecting things the spec doesn't allow.  I need
to dig back through my e-mail, but some guys from NVIDIA had convinced
me that there was something wrong... but I don't recall what.

When I presented this at XDC in 2014, I think the consensus was that
tests that actually found a bug should be added to the "right" place in
the repo, but we don't want to run 47,000,000 random tests on regular
piglit runs.  Someone should have a system somewhere that just runs
these (and other) random, fuzzing tests 24/7.

> Cheers,
> 
>   -ilia

___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ian Romanick]

On 09/24/2014 09:47 AM, Ian Romanick wrote:
 So, here it is.  Finally.
 
 The first two patches provide the infrastructure for generating
 randomized UBO tests.  I think these are pretty solid, but there are
 probably ways to impove the Python, etc.
 
 The remaining three patches are examples of ways the infrastructure can
 be used.  Here is where I am not sure what we should do.  I know that we
 don't want to make the forever test in patch 4 part of regular piglit
 runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
 have tested.
 
 I'm also unsure about the random tests generated by patch 3.  Do we want
 actual random tests in regular piglit runs?  What do we do for tests for
 GLSL 1.40?  Generate the same tests, but use #version 140 instead of
 #extension?
 
 In any case, I know that folks are hard at work on fp64 support, so
 using the various random runners here should help that effort.  Sorry
 for all the delays.
 
 One last thing... I'm presenting a bunch of information about this work
 at XDC in a couple weeks.  Maybe we want to wait to hammer out the more
 difficult details until then.  Dunno.

I've pushed updated version of this series to the ubo-lolz branch of my
fd.o piglit repo.

 ___
 Piglit mailing list
 Piglit@lists.freedesktop.org
 http://lists.freedesktop.org/mailman/listinfo/piglit

___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Matt Turner]

On Wed, Sep 24, 2014 at 9:47 AM, Ian Romanick i...@freedesktop.org wrote:
 So, here it is.  Finally.

 The first two patches provide the infrastructure for generating
 randomized UBO tests.  I think these are pretty solid, but there are
 probably ways to impove the Python, etc.

 The remaining three patches are examples of ways the infrastructure can
 be used.  Here is where I am not sure what we should do.  I know that we
 don't want to make the forever test in patch 4 part of regular piglit
 runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
 have tested.

 I'm also unsure about the random tests generated by patch 3.  Do we want
 actual random tests in regular piglit runs?

I think you know the answer to that. :)

 What do we do for tests for
 GLSL 1.40?  Generate the same tests, but use #version 140 instead of
 #extension?

I don't see any benefit to running the same execution tests under
different GLSL versions.
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ian Romanick]

On 09/24/2014 10:24 AM, Matt Turner wrote:
 On Wed, Sep 24, 2014 at 9:47 AM, Ian Romanick i...@freedesktop.org wrote:
 So, here it is.  Finally.

 The first two patches provide the infrastructure for generating
 randomized UBO tests.  I think these are pretty solid, but there are
 probably ways to impove the Python, etc.

 The remaining three patches are examples of ways the infrastructure can
 be used.  Here is where I am not sure what we should do.  I know that we
 don't want to make the forever test in patch 4 part of regular piglit
 runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
 have tested.

 I'm also unsure about the random tests generated by patch 3.  Do we want
 actual random tests in regular piglit runs?
 
 I think you know the answer to that. :)

I actually don't. :( Running the same set of tests repeatedly isn't
going to find additional bugs.  If we're at a point where we think we've
fixed all the bugs, running slightly different
array-of-structures-containing-a-structure test won't have the
irritating oscillating behavior that, say, the front-buffer rendering
tests or timer query tests have.  The tests are generated at build-time,
so the results will be stable across runs regardless.

Are there more bugs to be found by this generator in our implementation?
 I don't know.  The probability diminishes as more things get fixed in
the implementation.  As we add more functionality, the probability goes
back up.

What method should we use to search for them?  I also don't know.  Last
week I ran the script in patch 4 for about 8 hours on a single CPU
without encountering a failure.  It ran around 8,000 tests.

This is part of the reason that it has taken so long to get this posted.
 That and fixing the bugs that it found...

 What do we do for tests for
 GLSL 1.40?  Generate the same tests, but use #version 140 instead of
 #extension?
 
 I don't see any benefit to running the same execution tests under
 different GLSL versions.

Certainly not the same.  Almost certainly not fuzzy the same either.
We usually have tests for an extension and tests for the core version
(with varying degrees of success).  It looks like the only
implementation (possibly existing in the world) that has UBOs and not
OpenGL 3.1 is swrast.  If we don't care about abusing swrast, we could
just use these tests on 3.1.
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit