Public bug reported: Failure example:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c9d/819377/2/check/ovn- octavia-provider-tempest-release/c9db1e6/job-output.txt 2021-11-26 07:20:41.865039 | controller | + ./stack.sh:exit_trap:507 : local r=1 2021-11-26 07:20:41.868331 | controller | ++ ./stack.sh:exit_trap:508 : jobs -p 2021-11-26 07:20:41.872000 | controller | + ./stack.sh:exit_trap:508 : jobs=84673 2021-11-26 07:20:41.875064 | controller | + ./stack.sh:exit_trap:511 : [[ -n 84673 ]] 2021-11-26 07:20:41.877646 | controller | + ./stack.sh:exit_trap:511 : [[ -n /opt/stack/logs/devstacklog.txt.2021-11-26-070811 ]] 2021-11-26 07:20:41.880814 | controller | + ./stack.sh:exit_trap:511 : [[ True == \T\r\u\e ]] 2021-11-26 07:20:41.883918 | controller | + ./stack.sh:exit_trap:512 : echo 'exit_trap: cleaning up child processes' 2021-11-26 07:20:41.883992 | controller | exit_trap: cleaning up child processes 2021-11-26 07:20:41.887286 | controller | + ./stack.sh:exit_trap:513 : kill 84673 2021-11-26 07:20:41.890505 | controller | + ./stack.sh:exit_trap:517 : '[' -f /tmp/tmp.kOZU5nmHMT ']' 2021-11-26 07:20:41.893624 | controller | + ./stack.sh:exit_trap:518 : rm /tmp/tmp.kOZU5nmHMT 2021-11-26 07:20:41.897844 | controller | + ./stack.sh:exit_trap:522 : kill_spinner 2021-11-26 07:20:41.901167 | controller | + ./stack.sh:kill_spinner:417 : '[' '!' -z '' ']' 2021-11-26 07:20:41.904165 | controller | + ./stack.sh:exit_trap:524 : [[ 1 -ne 0 ]] 2021-11-26 07:20:41.906863 | controller | + ./stack.sh:exit_trap:525 : echo 'Error on exit' 2021-11-26 07:20:41.906923 | controller | Error on exit 2021-11-26 07:20:41.909908 | controller | + ./stack.sh:exit_trap:527 : type -p generate-subunit 2021-11-26 07:20:41.912989 | controller | + ./stack.sh:exit_trap:528 : generate-subunit 1637910489 752 fail 2021-11-26 07:20:42.221225 | controller | + ./stack.sh:exit_trap:530 : [[ -z /opt/stack/logs ]] 2021-11-26 07:20:42.224703 | controller | + ./stack.sh:exit_trap:533 : /usr/bin/python3.8 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs 2021-11-26 07:20:42.690784 | controller | + ./stack.sh:exit_trap:542 : exit 1 2021-11-26 07:20:42.690829 | controller | *** FINISHED *** 2021-11-26 07:20:53.478505 | controller | ERROR 2021-11-26 07:20:53.478789 | controller | { In the ovn-northd logs there is error like: Nov 26 07:19:11.721742 ubuntu-focal-inmotion-iad3-0027510773 bash[111405]: * Creating empty database /opt/stack/data/ovn/ovnsb_db.db Nov 26 07:19:11.722667 ubuntu-focal-inmotion-iad3-0027510773 bash[111446]: chown: changing ownership of '/opt/stack/data/ovn': Operation not permitted Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.pid': Operation not permitted Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.110711.ctl': Operation not permitted Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.mgmt': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.110207.ctl': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/db.sock': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.snoop': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.pid': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server.log': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/dstat-csv.log': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovs-vswitchd.log': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.summary': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/archive': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server-nb.log': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811.summary.2021-11-26-070811': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs': Operation not permitted Nov 26 07:19:11.725814 ubuntu-focal-inmotion-iad3-0027510773 bash[111449]: chown: cannot access '/usr/local/etc/ovn': No such file or directory Nov 26 07:19:11.727636 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111450]: ovs|00001|vlog|INFO|opened log file /opt/stack/logs/ovsdb-server-sb.log Nov 26 07:19:11.728571 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00002|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to open lock file: Permission denied Nov 26 07:19:11.728596 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00003|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to lock file: Resource temporarily unavailable Nov 26 07:19:11.728973 ubuntu-focal-inmotion-iad3-0027510773 bash[111452]: ovsdb-server: I/O error: /opt/stack/data/ovn/ovnsb_db.db: failed to lock lockfile (Resource temporarily unavailable) Nov 26 07:19:11.734924 ubuntu-focal-inmotion-iad3-0027510773 ovn-sbctl[111456]: ovs|00001|sbctl|INFO|Called as ovn-sbctl --no-leader-only init Nov 26 07:20:41.854729 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: [email protected]: start operation timed out. Terminating. Nov 26 07:20:41.855570 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: [email protected]: Killing process 111456 (ovn-sbctl) with signal SIGKILL. Nov 26 07:20:41.855656 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: [email protected]: Failed with result 'timeout'. Nov 26 07:20:41.855824 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: Failed to start Devstack [email protected]. ** Affects: neutron Importance: Critical Assignee: Slawek Kaplonski (slaweq) Status: Confirmed ** Tags: gate-failure ovn-octavia-provider tempest -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1952395 Title: Tempes jobs in the ovn-octavia-provider are broken Status in neutron: Confirmed Bug description: Failure example: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c9d/819377/2/check/ovn- octavia-provider-tempest-release/c9db1e6/job-output.txt 2021-11-26 07:20:41.865039 | controller | + ./stack.sh:exit_trap:507 : local r=1 2021-11-26 07:20:41.868331 | controller | ++ ./stack.sh:exit_trap:508 : jobs -p 2021-11-26 07:20:41.872000 | controller | + ./stack.sh:exit_trap:508 : jobs=84673 2021-11-26 07:20:41.875064 | controller | + ./stack.sh:exit_trap:511 : [[ -n 84673 ]] 2021-11-26 07:20:41.877646 | controller | + ./stack.sh:exit_trap:511 : [[ -n /opt/stack/logs/devstacklog.txt.2021-11-26-070811 ]] 2021-11-26 07:20:41.880814 | controller | + ./stack.sh:exit_trap:511 : [[ True == \T\r\u\e ]] 2021-11-26 07:20:41.883918 | controller | + ./stack.sh:exit_trap:512 : echo 'exit_trap: cleaning up child processes' 2021-11-26 07:20:41.883992 | controller | exit_trap: cleaning up child processes 2021-11-26 07:20:41.887286 | controller | + ./stack.sh:exit_trap:513 : kill 84673 2021-11-26 07:20:41.890505 | controller | + ./stack.sh:exit_trap:517 : '[' -f /tmp/tmp.kOZU5nmHMT ']' 2021-11-26 07:20:41.893624 | controller | + ./stack.sh:exit_trap:518 : rm /tmp/tmp.kOZU5nmHMT 2021-11-26 07:20:41.897844 | controller | + ./stack.sh:exit_trap:522 : kill_spinner 2021-11-26 07:20:41.901167 | controller | + ./stack.sh:kill_spinner:417 : '[' '!' -z '' ']' 2021-11-26 07:20:41.904165 | controller | + ./stack.sh:exit_trap:524 : [[ 1 -ne 0 ]] 2021-11-26 07:20:41.906863 | controller | + ./stack.sh:exit_trap:525 : echo 'Error on exit' 2021-11-26 07:20:41.906923 | controller | Error on exit 2021-11-26 07:20:41.909908 | controller | + ./stack.sh:exit_trap:527 : type -p generate-subunit 2021-11-26 07:20:41.912989 | controller | + ./stack.sh:exit_trap:528 : generate-subunit 1637910489 752 fail 2021-11-26 07:20:42.221225 | controller | + ./stack.sh:exit_trap:530 : [[ -z /opt/stack/logs ]] 2021-11-26 07:20:42.224703 | controller | + ./stack.sh:exit_trap:533 : /usr/bin/python3.8 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs 2021-11-26 07:20:42.690784 | controller | + ./stack.sh:exit_trap:542 : exit 1 2021-11-26 07:20:42.690829 | controller | *** FINISHED *** 2021-11-26 07:20:53.478505 | controller | ERROR 2021-11-26 07:20:53.478789 | controller | { In the ovn-northd logs there is error like: Nov 26 07:19:11.721742 ubuntu-focal-inmotion-iad3-0027510773 bash[111405]: * Creating empty database /opt/stack/data/ovn/ovnsb_db.db Nov 26 07:19:11.722667 ubuntu-focal-inmotion-iad3-0027510773 bash[111446]: chown: changing ownership of '/opt/stack/data/ovn': Operation not permitted Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.pid': Operation not permitted Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.110711.ctl': Operation not permitted Nov 26 07:19:11.723666 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.mgmt': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.110207.ctl': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/db.sock': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.snoop': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.pid': Operation not permitted Nov 26 07:19:11.723927 ubuntu-focal-inmotion-iad3-0027510773 bash[111447]: chown: changing ownership of '/usr/local/var/run/openvswitch': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server.log': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/dstat-csv.log': Operation not permitted Nov 26 07:19:11.724747 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovs-vswitchd.log': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.summary': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/archive': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/ovsdb-server-nb.log': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs/devstacklog.txt.2021-11-26-070811.summary.2021-11-26-070811': Operation not permitted Nov 26 07:19:11.725149 ubuntu-focal-inmotion-iad3-0027510773 bash[111448]: chown: changing ownership of '/opt/stack/logs': Operation not permitted Nov 26 07:19:11.725814 ubuntu-focal-inmotion-iad3-0027510773 bash[111449]: chown: cannot access '/usr/local/etc/ovn': No such file or directory Nov 26 07:19:11.727636 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111450]: ovs|00001|vlog|INFO|opened log file /opt/stack/logs/ovsdb-server-sb.log Nov 26 07:19:11.728571 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00002|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to open lock file: Permission denied Nov 26 07:19:11.728596 ubuntu-focal-inmotion-iad3-0027510773 ovsdb-server[111452]: ovs|00003|lockfile|WARN|/opt/stack/data/ovn/.ovnsb_db.db.~lock~: failed to lock file: Resource temporarily unavailable Nov 26 07:19:11.728973 ubuntu-focal-inmotion-iad3-0027510773 bash[111452]: ovsdb-server: I/O error: /opt/stack/data/ovn/ovnsb_db.db: failed to lock lockfile (Resource temporarily unavailable) Nov 26 07:19:11.734924 ubuntu-focal-inmotion-iad3-0027510773 ovn-sbctl[111456]: ovs|00001|sbctl|INFO|Called as ovn-sbctl --no-leader-only init Nov 26 07:20:41.854729 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: [email protected]: start operation timed out. Terminating. Nov 26 07:20:41.855570 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: [email protected]: Killing process 111456 (ovn-sbctl) with signal SIGKILL. Nov 26 07:20:41.855656 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: [email protected]: Failed with result 'timeout'. Nov 26 07:20:41.855824 ubuntu-focal-inmotion-iad3-0027510773 systemd[1]: Failed to start Devstack [email protected]. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1952395/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
