Re: [Piglit] [PATCH] framework: Handle tests with subtests crashing in totals

On Fri, 2018-01-19 at 15:26 -0800, Dylan Baker wrote: > I mean to CC Fabien as well... I think you did, at least the email looks correct. I'm not sure how to test this change. After reverting Fabien's patch things got back to normal. the tests on Jan 17 and 18 ran with Fabien's change, the rest

Re: [Piglit] [RFC 1/9] util: Add a function for enumerating subtests

I meant to add a cover letter to this, also, there's only 8 patches. This is an RFC, I'm looking for input into fixing the subtest reporting problem. This approach is going to be more complete, but it's going to be pretty involved. If anyone has a better approach I'd be happy to hear it. Quoting

[Piglit] [PATCH 5/9] tests/fbo-storage-formats: Always print the same number of subtests

!skip -> skip won't show up in the regressions/fixes/etc lists anyway, and this means that the output will always be the same. --- tests/fbo/fbo-storage-formats.c | 6 ++ 1 file changed, 6 insertions(+) diff --git a/tests/fbo/fbo-storage-formats.c b/tests/fbo/fbo-storage-formats.c index

[Piglit] [PATCH 3/9] framework: add support for parsing subtest enumeration

This adds support for enumerating subtests to the python layer. When it sees this it sets each subtest to notrun. This allows the python framework to report that tests didn't run when they were expected to. Signed-off-by: Dylan Baker --- framework/test/piglit_test.py |

[Piglit] [PATCH 8/9] tests/fbo-storage-formats: enumerate subtests

--- tests/fbo/fbo-storage-formats.c | 22 ++ 1 file changed, 22 insertions(+) diff --git a/tests/fbo/fbo-storage-formats.c b/tests/fbo/fbo-storage-formats.c index 4db990d..a11335b 100644 --- a/tests/fbo/fbo-storage-formats.c +++ b/tests/fbo/fbo-storage-formats.c @@ -243,9

[Piglit] [PATCH 2/9] tests: enumerate subtests in gl-2.1-gbo test

This gives us something to test the python part against. Signed-off-by: Dylan Baker --- tests/spec/gl-2.1/pbo.c | 6 ++ 1 file changed, 6 insertions(+) diff --git a/tests/spec/gl-2.1/pbo.c b/tests/spec/gl-2.1/pbo.c index 83dc1c4..1a59733 100644 ---

Re: [Piglit] [PATCH] framework: Handle tests with subtests crashing in totals

I mean to CC Fabien as well... Quoting Dylan Baker (2018-01-19 14:04:13) > Currently piglit doesn't account for a test with subtests crashing when > it calculates the total number of tests of each status. The result is > that if a test with subtests runs no tests before crashing it is handled >

[Piglit] [PATCH 7/9] tests/fbo-storage-formats: print subtest result for skip too

Because we always want to have the same number of subtests printed, or they'll show up as "NOTRUN" instead of "SKIP" in the summary. Signed-off-by: Dylan Baker --- tests/fbo/fbo-storage-formats.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git

[Piglit] [PATCH 6/9] tests/fbo-storage-formats: Make subtest names predictable

Currently the name of the subtest relies on checking the GL state of the command to know whether the framebuffer is completely or incomplete. That is a problem for enumerating subtests, we would need to know ahead of time whether the framebuffer is complete or not. Instead print the completeness

[Piglit] [PATCH 4/9] tests: enumerate subtests in fbo-incomplete

Signed-off-by: Dylan Baker --- tests/fbo/fbo-incomplete.cpp | 7 +++ 1 file changed, 7 insertions(+) diff --git a/tests/fbo/fbo-incomplete.cpp b/tests/fbo/fbo-incomplete.cpp index 8cde6d2..0354615 100644 --- a/tests/fbo/fbo-incomplete.cpp +++

[Piglit] [PATCH 1/9] util: Add a function for enumerating subtests

This function takes one or more subtests as strings and returns a JSON structure that the python framework can consume. Signed-off-by: Dylan Baker --- tests/util/piglit-util.c | 29 + tests/util/piglit-util.h | 2 ++ 2 files changed, 31

[Piglit] [PATCH] framework: Handle tests with subtests crashing in totals

Currently piglit doesn't account for a test with subtests crashing when it calculates the total number of tests of each status. The result is that if a test with subtests runs no tests before crashing it is handled correctly (since it goes down the non-subtest path), but if one or more subtests

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #9 from Dylan Baker --- I've sent a patch for this, I cc'd both Jan and Fabian -- You are receiving this mail because: You are the QA Contact for the bug.___

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #8 from Dylan Baker --- Fabian, it's the totalling code that's wrong. It doesn't account for crashes in tests with subtests. I'm looking at it now. -- You are receiving this mail because: You are the QA

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #7 from Fabian Bieler --- (In reply to Jan Vesely from comment #5) > (In reply to Fabian Bieler from comment #3) > > Strange, I can't seem to reproduce that. > > I reverted 938ec48e2, ran

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #6 from Dylan Baker --- I think there's actually a bug in the summary code. I reverted this patch, and added the Fabian's patch to the data-sync test. The raw JSON contains what I expect, but `piglit summary

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #5 from Jan Vesely --- (In reply to Fabian Bieler from comment #3) > Strange, I can't seem to reproduce that. > I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once > with a call to

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #4 from Dylan Baker --- the behavior Jan sees is what the framework is *supposed* to do. This is pretty frustrating actually because it makes it impossible for us to track regressions in CI because the

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #3 from Fabian Bieler --- Strange, I can't seem to reproduce that. I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once with a call to abort() after the first call to

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #2 from Jan Vesely --- That's no entirely accurate. The previous behavior would report crash of the entire test. So you could see: test1 pass -> crash subtest0 pass -> notrun subtest1 pass -> notrun the

Re: [Piglit] [PATCH] cl: Add test for MUBUF access with a negative vaddr

> On Jan 18, 2018, at 15:02, Jan Vesely wrote: > > Why is this necessary? can't you just pass the offset argument as a > kernel input? > > Jan It needs to specifically be in a VGPR___ Piglit mailing list

[Piglit] [Bug 104700] Crashing tests with subtests don't show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 Fabian Bieler changed: What|Removed |Added Summary|Crashing tests with |Crashing

[Piglit] [Bug 104700] Crashing tests with subtests no longer show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 --- Comment #1 from Fabian Bieler --- Created attachment 136858 --> https://bugs.freedesktop.org/attachment.cgi?id=136858=edit Framework/summary: Include crashing subtests in fixes and regressions. If a subtest

Re: [Piglit] [PATCH] gl-1.0-blend-func: skip some blend tests when using LLVM 3.8

Am 19.01.2018 um 06:35 schrieb Eric Anholt: > Brian Paul writes: > >> On 01/18/2018 01:27 PM, Eric Anholt wrote: >>> Brian Paul writes: >>> To avoid an infinite loop. See code comments for details. >>> >>> Skipping a failing test and returning pass is

[Piglit] [Bug 104700] New: Crashing tests with subtests no longer show in a list of regressions

https://bugs.freedesktop.org/show_bug.cgi?id=104700 Bug ID: 104700 Summary: Crashing tests with subtests no longer show in a list of regressions Product: piglit Version: unspecified Hardware: Other OS: All