Hi,
Day before yesterday, I had pushed code changes for feature "vrouter-vifdump-mirroring-for-bond-slaves" in Gerrit or review. Following is the change details:- https://review.opencontrail.org/#/c/30895/ The Jenkins CI process triggered following Unittest and Systest jobs. All of them passed successfully except one systest for Ubuntu14/Mitaka. -------------------------- Build failed. For information on how to proceed, see https://wiki.opencontrail.org/wiki/GerritJenkinsGit#Test_Failures - ci-contrail-vrouter-unittest-ubuntu14 https://jenkins.opencontrail.org/job/ci-contrail-vrouter-unittest-ubuntu14/2086/ : SUCCESS in 29s - ci-contrail-vrouter-systest-ubuntu14-mitaka https://jenkins.opencontrail.org/job/ci-contrail-vrouter-systest-ubuntu14-mitaka/476/ : FAILURE in 6h 13m 58s - ci-contrail-vrouter-systest-ubuntu14-vcenter https://jenkins.opencontrail.org/job/ci-contrail-vrouter-systest-ubuntu14-vcenter/1800/ : SUCCESS in 1m 41s - ci-contrail-vrouter-systest-ubuntu16-newton https://jenkins.opencontrail.org/job/ci-contrail-vrouter-systest-ubuntu16-newton/90/ : SUCCESS in 2m 52s - ci-contrail-vrouter-systest-centos71-kilo https://jenkins.opencontrail.org/job/ci-contrail-vrouter-systest-centos71-kilo/932/ : SUCCESS in 2m 00s - ci-contrail-vrouter-systest-redhat70-kilo https://jenkins.opencontrail.org/job/ci-contrail-vrouter-systest-redhat70-kilo/1093/ : SUCCESS in 2m 04s -------------------------- >From the logs (Job# 474), it didn't seem like a problem related to the >committed code-changes. Yesterday, following comment was posted twice which >triggered the Jenkins jobs (#475 and #476) automatically for my change. But >both of them failed. However the console log output shows failure at some >different place than original failure logs. I have following queries here. Please help to resolve these:- -> Is it some automated process which keeps on retrying the CI for the change-id? Also as there is difference in failure logs of build #474 and #476, should I conclude that Ubuntu14/Mitaka systest Job failure is a some CI infrastructure related issue (as other systest jobs get passed successfully)? And some correction was done there which triggered the CI process again twice? How should I proceed further with my change? -> How can I debug the root-cause of this issue, as I can't reproduce the systest environment locally so I can't verify the changes against the systests. I did verify the code-changes against the UT suite which also got passed in CI UT job. ------------------------- Patch Set 1: recheck no bug ------------------------- Build #474 console logs https://jenkins.opencontrail.org/job/ci-contrail-vrouter-systest-ubuntu14-mitaka/474/consoleFull --select "status" --json| grep status: 2017-05-01 16:25:49 util.rb:520:up?() ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime: 2017-05-01 16:25:50 contrail-systest-job.rb:385:smlite_install_provision() ssh ci-oc-subslave-ubuntu-10-84-34-113-1 server-manager-client display server --select "status" --json| grep status: 2017-05-01 16:26:52 util.rb:520:up?() ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime: 2017-05-01 16:26:53 contrail-systest-job.rb:385:smlite_install_provision() ssh ci-oc-subslave-ubuntu-10-84-34-113-1 server-manager-client display server --select "status" --json| grep status: 2017-05-01 16:27:55 util.rb:520:up?() ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime: 2017-05-01 16:27:55 IGNORED: Command ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime failed with exit code 255 2017-05-01 16:28:15 util.rb:520:up?() ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime: 2017-05-01 16:28:15 IGNORED: Command ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime failed with exit code 255 2017-05-01 16:28:35 util.rb:520:up?() ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime: 2017-05-01 16:30:43 IGNORED: Command ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime failed with exit code 255 2017-05-01 16:31:03 util.rb:520:up?() ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime: 2017-05-01 16:33:10 IGNORED: Command ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime failed with exit code 255 2017-05-01 16:33:30 util.rb:520:up?() ssh root@ci-oc-subslave-ubuntu-10-84-34-113-1 uptime: 2017-05-01 19:14:07 IGNORED: Command ssh ci-oc-subslave-ubuntu-10-84-34-113-1 server-manager-client display server --select "status" --json| grep status failed with exit code 1 2017-05-01 19:14:07 ERROR: Not able to get the provision status 2017-05-01 19:14:07 Exit unless /root/ci_job_wait is present 2017-05-01 19:14:07util.rb:877:get_console_log() ssh ci-admin@10.84.26.50 mkdir -p /ci-admin/failed_systest_logs/ci-contrail-vrouter-systest-ubuntu14-mitaka/474 2017-05-01 19:14:08util.rb:878:get_console_log() ssh 10.84.26.23 CI_ADMIN/ci-openstack.sh sshpass -p c0ntrail123 rsync -ac b6s8:/var/lib/nova/instances/2091db80-47f7-482d-80ea-35010e175269/console.log /tmp/ci-oc-subslave-ubuntu-10-84-34-113-1_console.log 2017-05-01 19:14:10rsync: change_dir "/var/lib/nova/instances/2091db80-47f7-482d-80ea-35010e175269" failed: No such file or directory (2) 2017-05-01 19:14:10 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.0] 2017-05-01 19:14:10 rsync: [Receiver] write error: Broken pipe (32) 2017-05-01 19:14:172017-05-01 19:14:17:015403: [root@192.168.2.221] out: Connection to http://127.0.0.1:8181/analytics/tables failed HTTPConnectionPool(host='127.0.0.1', port=8181): Max retries exceeded with url: /analytics/tables (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-05-01 19:14:17 2017-05-01 19:14:17:257216: [root@192.168.2.221] out: Connection to http://127.0.0.1:8181/analytics/query failed HTTPConnectionPool(host='127.0.0.1', port=8181): Max retries exceeded with url: /analytics/query (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-05-01 19:14:36 No server with a name or ID of '2091db80-47f7-482d-80ea-35010e175269' exists. 2017-05-01 19:14:36ERROR (CommandError): Unable to delete the specified server(s). Build #476 console logs https://jenkins.opencontrail.org/job/ci-contrail-vrouter-systest-ubuntu14-mitaka/476/consoleFull --select "status" --json| grep status 2017-05-02 13:53:38 ERROR:Sm-lite provision timedout out. It took more than 3600 seconds 2017-05-02 13:53:38 Exit unless /root/ci_job_wait is present Br, Deepak "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."
_______________________________________________ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
