Actually, branches do have a pass fail status, it's the worst status of that branch's children.
I've been thinking of using a Tree structure to represent the tests in Summary. The problem with the current architecture (as it pertains to this problem) is that the the statuses of each group are auto-generated from the 'worst' status in their children, and the only way to set a group status is after the entire tree has been generated. If you write a patch to add a fake subtest I'll be sure to look at it. On Monday, April 07, 2014 06:44:04 PM Ilia Mirkin wrote: > Before rewriting (fun though it is!), we should figure out what we'd > want to see, in a perfect world. Even that much isn't clear to me. Do > we want to promote test tree structure to first-class status, thus > being able to accurate express the situation? That would require HTML > changes as well, I think, since only leaves can have pass/fail status > right now, and everything else is just a n/m number. My hack should > make it clear what happened while also minimizing the amount of > change. I don't think that this is a common scenario (passing subtests > followed by crash), so just making it visible may be good enough. > > -ilia > > On Mon, Apr 7, 2014 at 6:34 PM, Dylan Baker <[email protected]> wrote: > > Ilia, > > > > After looking at the code and what you said I don't think this can be > > fixed > > without your hack or a complete re-architecture of the summary code. I've > > been itching for an excuse to rewrite that code again (what I wrote was > > better than what was there before, which should say something about what > > was there before) > > > > Does that sound right to you? > > > > On Monday, April 07, 2014 09:34:13 AM Ilia Mirkin wrote: > >> This took me a little bit to figure out what was going on... so to > >> save someone else some time: the problem is that the summary logic > >> branches into two -- individual results, and subtest results. For the > >> subtest results, the "overall" test result isn't kept anywhere. So the > >> fact that the test crashed is completely lost, since the individual > >> subtests completed. > >> > >> The reported data is accurate -- those subtests did all pass; however > >> it's incomplete. Perhaps if the test result is a crash, and there are > >> subtests, then create a new "fake" subtest that shows that the overall > >> test actually crashed? > >> > >> -ilia > >> > >> On Mon, Apr 7, 2014 at 7:27 AM, Marek Olšák <[email protected]> wrote: > >> > Hi, > >> > > >> > There is a serious issue with piglit that crashing tests just > >> > disappear from the HTML summary. Let's say I apply this: > >> > > >> > diff --git a/tests/fbo/fbo-generatemipmap-formats.c > >> > b/tests/fbo/fbo-generatemipmap-formats.c > >> > index 666030f..0d4a06b 100644 > >> > --- a/tests/fbo/fbo-generatemipmap-formats.c > >> > +++ b/tests/fbo/fbo-generatemipmap-formats.c > >> > @@ -431,6 +431,8 @@ test_format(const struct format_desc *format, > >> > GLenum basetype) > >> > > >> > GLboolean pass = GL_TRUE; > >> > > >> > printf("Testing %s%s\n", format->name, tex_width == 256 ? "" : > >> > " (NPOT)"); > >> > + if (format->internalformat == GL_RG16) > >> > + *(int*)NULL = 0; > >> > > >> > tex = create_tex(format->internalformat, > >> > format->base_internal_format, > >> > > >> > basetype); > >> > > >> > Then, the output of the test is: > >> > > >> > $ bin/fbo-generatemipmap-formats GL_ARB_texture_rg -auto > >> > Using test set: GL_ARB_texture_rg > >> > Testing GL_R8 > >> > PIGLIT:subtest {'GL_R8' : 'pass'} > >> > Testing GL_R16 > >> > PIGLIT:subtest {'GL_R16' : 'pass'} > >> > Testing GL_RG > >> > PIGLIT:subtest {'GL_RG' : 'pass'} > >> > Testing GL_RG8 > >> > PIGLIT:subtest {'GL_RG8' : 'pass'} > >> > Testing GL_RG16 > >> > Segmentation fault (core dumped) > >> > > >> > Let's run piglit-run.py with "-t > >> > ARB_texture_rg/fbo-generatemipmap-formats$ -v": > >> > > >> > running :: spec/ARB_texture_rg/fbo-generatemipmap-formats > >> > crash :: spec/ARB_texture_rg/fbo-generatemipmap-formats > >> > [1/1] crash: 1 > >> > > >> > Looks good, right? But there is no crash in the HTML summary: > >> > > >> > http://people.freedesktop.org/~mareko/summary/ > >> > > >> > Do you have any idea how to fix this? > >> > > >> > Marek > >> > _______________________________________________ > >> > Piglit mailing list > >> > [email protected] > >> > http://lists.freedesktop.org/mailman/listinfo/piglit > >> > >> _______________________________________________ > >> Piglit mailing list > >> [email protected] > >> http://lists.freedesktop.org/mailman/listinfo/piglit
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
