On 01/29/2014 04:19 PM, Robert Collins wrote: > On 26 January 2014 02:25, Sean Dague <[email protected]> wrote: >> Something that's currently just far to manual to figure out is how testr >> is functioning in the parallel case, which tests are running on which >> process, and when does one of the pipelines run out of tests. The testr >> assumption that you don't want output unless there is a failure is a >> really bad assumption, because it's important to know that things are >> progressing, especially in complicated systems. > > IMO There are three distinct use cases - we need to be able to tell > what was concurrent (the saved stream should do that post-hoc). > Secondly we need to identify hangs and so on - human scale. Lastly we > need automation around identifying broken things...
Sure.
>> In tempest we actually have a wrapper that gives us a stream. But it's
>> got a couple of fundamental flaws.
>>
>> First - it is very clear that the expected behavior of start / end of
>> tests doesn't work:
>>
>> 2014-01-24 00:49:59.005 |
>> tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate]
>> 2014-01-24 00:49:59.005 |
>> tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate]
>> ... ok
>
> subunit2pyunit just routes everything through a regular python
> 'unittest.TextTestResult' which doesn't know about timing data or
> concurrency. So we need to replace that.
Ok, sounds fair.
>> Question #1: how do we either make start message be when things actually
>> start? or turn it off entirely?
>>
>>
>> Second - it's not entirely obvious how unbalanced the pipelines are, and
>> that would be extremely useful to see in the output. A starting point
>> for a UI would be to add...
>>
>> " " x (5 * pipelinenumber)
>>
>> To the test results. Then you'd have a visual waterfall to the front of
>> the tests, and could clearly see pipeline differences.
>
> would (worker-N) be sufficient? Or do you want *'s/x's specifically?
So if we had worker-N we could play with a filter above that layer to do
UI formatting.
Realistically I had a stream with enough metadata, and preferably a
solid parsing lib for that stream (so I don't screw that part up), then
we could handle the UI in a filter.
Honestly, UX here is going to require the ability to iterate and
experiment. In much the same way that we've got a completely
disconnected visualization system for zuul from zuul itself, and only
rarely do we need to touch zuul proper to improve status.openstack.org.
>> Question #2: where would this be injectable in the stream of testtools /
>> subunit / testr / unittest?
>
> Long term I defintely want a richer UI in testr, but right now the
> most appropriate thing to do is to prototype the UI you need locally.
Sounds great. Again if we had some of the piece parts, I could do the
rest of the UI locally, just diving down through the layers made me
realize I really needed to figure out where to hook into, as it wasn't
entirely clear (at least to me).
-Sean
--
Sean Dague
Samsung Research America
[email protected] / [email protected]
http://dague.net
signature.asc
Description: OpenPGP digital signature
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
