I find the way that CI displays test results to be quite challenging to read/decipher, for the following reasons:
1. Sanity results are displayed by using lynx to dump an html file into job console 2. Tempest results are displayed by using cat to dump out XML file 3. In both cases, pipelines of grep, see, cut, etc are used to determine failure counts Since the test results are available in XML files, I'm inclined to add code to parse those XML files and display the test results in a more brief and human-readable format. What I'm proposing: A. If there are no failures, then display only a summary, perhaps something like this: ================ Sanity Test results: Tests: 28 Failed: 0 Time: 3137.304 sec URL: http://10.204.216.50/Docs/logs/3.0-9736_2015-05-09_17:46:06/junit-noframes.html ================ Tempest Test summary results: Tests: 165 Failed: 0 Time: 745.870 sec URL: http://10.204.216.50/Docs/logs/ci-tempest/result-10677,1-215201519543.xml ================ [ The URL's provide full results in human-readable format ] [ URL's likely on jenkins.opencontrail.org instead of our internal server ] B. Display the failed tests, if any. Like this: Sanity: FAIL: TestBasicPolicyConfig.test_policy_to_deny (94.486 sec) FAIL: TestBasicPolicyModify.test_policy_modify_vn_policy (124.770 sec) Tempest: FAIL: test_add_remove_fixed_ip (39.939 sec) FAIL: test_create_list_show_delete_interfaces (37.679 sec) C. Display the passed tests, if any. Sanity: PASS: TestBasicVMVN4.test_vm_add_delete (72.356 sec) PASS: TestBasicVMVN4.test_vm_file_trf_scp_tests (125.162 sec) Tempest: PASS: test_create_show_delete_security_group_rule (0.883 sec) PASS: test_list_security_groups (0.054 sec) The PASS tests as per (C) are optional IMO because they aren't really of interest, and are easily available via URL in summary (A). However, perhaps the per-test time info is useful to have in the console log. Concerning (B), an additional option: for tempest tests the XML typically contains error message for each test, typically a Python traceback. This could be added to the output in (B). An example: FAIL: test_create_list_show_update_delete_floating_ip (0.028 sec) testcase error log: Traceback (most recent call last): testtools.testresult.real._StringException: Empty attachments: stderr stdout pythonlogging:'': {{{2015-05-21 23:13:07,611 Request (FloatingIPTestJSON:test_create_list_show_update_delete_floating_ip): 400 POST http://192.168.0.66:9696/v2.0/floatingips 0.017s}}} Traceback (most recent call last): File "/root/tempest/tempest/api/network/test_floating_ips.py", line 72, in test_create_list_show_update_delete_floating_ip floating_network_id=self.ext_net_id, port_id=self.ports[0]['id']) File "/root/tempest/tempest/services/network/network_client_base.py", line 144, in _create resp, body = self.post(uri, post_data) File "/root/tempest/tempest/services/network/network_client_base.py", line 71, in post return self.rest_client.post(uri, body, headers) File "/root/tempest/tempest/common/rest_client.py", line 209, in post return self.request('POST', url, extra_headers, headers, body) File "/root/tempest/tempest/common/rest_client.py", line 419, in request resp, resp_body) File "/root/tempest/tempest/common/rest_client.py", line 468, in _error_checker raise exceptions.BadRequest(resp_body) tempest.exceptions.BadRequest: Bad request Details: {u'NeutronError': u"Invalid input for floating_network_id. Reason: '' is not a valid UUID."} Please let me know your opinion as to whether this would be an improvement to your leisurely console-reading time. -- Klash _______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
