Before rewriting (fun though it is!), we should figure out what we'd want to see, in a perfect world. Even that much isn't clear to me. Do we want to promote test tree structure to first-class status, thus being able to accurate express the situation? That would require HTML changes as well, I think, since only leaves can have pass/fail status right now, and everything else is just a n/m number. My hack should make it clear what happened while also minimizing the amount of change. I don't think that this is a common scenario (passing subtests followed by crash), so just making it visible may be good enough.
-ilia On Mon, Apr 7, 2014 at 6:34 PM, Dylan Baker <[email protected]> wrote: > Ilia, > > After looking at the code and what you said I don't think this can be fixed > without your hack or a complete re-architecture of the summary code. I've been > itching for an excuse to rewrite that code again (what I wrote was better than > what was there before, which should say something about what was there before) > > Does that sound right to you? > > On Monday, April 07, 2014 09:34:13 AM Ilia Mirkin wrote: >> This took me a little bit to figure out what was going on... so to >> save someone else some time: the problem is that the summary logic >> branches into two -- individual results, and subtest results. For the >> subtest results, the "overall" test result isn't kept anywhere. So the >> fact that the test crashed is completely lost, since the individual >> subtests completed. >> >> The reported data is accurate -- those subtests did all pass; however >> it's incomplete. Perhaps if the test result is a crash, and there are >> subtests, then create a new "fake" subtest that shows that the overall >> test actually crashed? >> >> -ilia >> >> On Mon, Apr 7, 2014 at 7:27 AM, Marek Olšák <[email protected]> wrote: >> > Hi, >> > >> > There is a serious issue with piglit that crashing tests just >> > disappear from the HTML summary. Let's say I apply this: >> > >> > diff --git a/tests/fbo/fbo-generatemipmap-formats.c >> > b/tests/fbo/fbo-generatemipmap-formats.c >> > index 666030f..0d4a06b 100644 >> > --- a/tests/fbo/fbo-generatemipmap-formats.c >> > +++ b/tests/fbo/fbo-generatemipmap-formats.c >> > @@ -431,6 +431,8 @@ test_format(const struct format_desc *format, >> > GLenum basetype) >> > >> > GLboolean pass = GL_TRUE; >> > >> > printf("Testing %s%s\n", format->name, tex_width == 256 ? "" : >> > " (NPOT)"); >> > + if (format->internalformat == GL_RG16) >> > + *(int*)NULL = 0; >> > >> > tex = create_tex(format->internalformat, >> > format->base_internal_format, >> > >> > basetype); >> > >> > Then, the output of the test is: >> > >> > $ bin/fbo-generatemipmap-formats GL_ARB_texture_rg -auto >> > Using test set: GL_ARB_texture_rg >> > Testing GL_R8 >> > PIGLIT:subtest {'GL_R8' : 'pass'} >> > Testing GL_R16 >> > PIGLIT:subtest {'GL_R16' : 'pass'} >> > Testing GL_RG >> > PIGLIT:subtest {'GL_RG' : 'pass'} >> > Testing GL_RG8 >> > PIGLIT:subtest {'GL_RG8' : 'pass'} >> > Testing GL_RG16 >> > Segmentation fault (core dumped) >> > >> > Let's run piglit-run.py with "-t >> > ARB_texture_rg/fbo-generatemipmap-formats$ -v": >> > >> > running :: spec/ARB_texture_rg/fbo-generatemipmap-formats >> > crash :: spec/ARB_texture_rg/fbo-generatemipmap-formats >> > [1/1] crash: 1 >> > >> > Looks good, right? But there is no crash in the HTML summary: >> > >> > http://people.freedesktop.org/~mareko/summary/ >> > >> > Do you have any idea how to fix this? >> > >> > Marek >> > _______________________________________________ >> > Piglit mailing list >> > [email protected] >> > http://lists.freedesktop.org/mailman/listinfo/piglit >> >> _______________________________________________ >> Piglit mailing list >> [email protected] >> http://lists.freedesktop.org/mailman/listinfo/piglit _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
