With Robert's help, I've been working on a new subunit real time filter which has a few features beyond what is currently in tempest:
* displays which testr worker the test is on (to make it easier to
figure out what other tests might be running when your test is running)
* displays the time for each tests (caveat, setupclass time is still
not accounted for)
* displays stdout/stderr inline at the end of a test result (all status)
* displays stacktrace and pythonlogging inline at the end of a *failed*
test result
This merged into Tempest yesterday. Output on success will look
something like this:
2014-05-07 05:43:41.488 | {3}
tempest.api.compute.keypairs.test_keypairs_negative.KeyPairsNegativeTestXML.test_keypair_create_with_invalid_pub_key
[0.301279s] ... ok
2014-05-07 05:43:41.599 | {1}
tempest.api.compute.servers.test_delete_server.DeleteServersTestXML.test_delete_active_server
[28.083070s] ... ok
2014-05-07 05:43:41.610 | {3}
tempest.api.compute.keypairs.test_keypairs_negative.KeyPairsNegativeTestXML.test_keypair_delete_nonexistent_key
[0.121472s] ... ok
2014-05-07 05:43:42.480 | {0}
tempest.api.compute.servers.test_delete_server.DeleteServersAdminTestXML.test_delete_server_while_in_error_state
[24.822415s] ... ok
2014-05-07 05:43:43.384 | {3}
tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestXML.test_security_group_create_get_delete
[0.740040s] ... ok
2014-05-07 05:43:44.383 | {3}
tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestXML.test_security_groups_create_list_delete
[0.998106s] ... ok
2014-05-07 05:43:58.633 | {2}
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_flavor
[0.109771s] ... ok
2014-05-07 05:43:58.634 | {2}
tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_image
... SKIPPED: Only one image found
{###} is testr worker number
Failed output will look like:
http://logs.openstack.org/07/83207/61/check/check-tempest-dsvm-full/88816fd/console.html#_2014-05-07_05_55_32_429
The decision to bring this inline vs. at the end of the test run is that
when watching long tests runs you'll get this info the moment the test
fails, which means you can be following jenkins and start digging into
failures early.
We did expose a testr bug in the process that worker # is not always
getting allocated correctly. That's been filed upstream. It will mean
that some runs will report the wrong worker for a class of tests, so if
it looks like you are missing a worker, that's why.
This also means you can iterate on tempest tests using inline 'print'
calls, which many of us find very handy.
There are some worker summary patches coming through in the queue as
well to give a break down of tests and runtime per worker, to help us
understand how unbalanced a run ends up.
-Sean
--
Sean Dague
http://dague.net
signature.asc
Description: OpenPGP digital signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
