Public bug reported: It happens pretty often recently that during our scenario tests we are running out of memory and oom-killer is killing mysql process as it is number 1 in memory consumption. That is causing job's failures.
It seems for me that it happens when there are running tests which uses vms with advanced image (ubuntu). Maybe we should extract those tests and run them as second stage with "--concurency 1"? Examples of failures: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85a/803462/2/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/85afc13/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/803936/1/gate/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/9445e5f/testr_results.html https://09e003a6a650f320c43d-e30f275ad83ed88289c7399adb6c5ee6.ssl.cf1.rackcdn.com/804236/5/check/neutron- tempest-plugin-scenario-linuxbridge/770722a/testr_results.html https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/797221/1/check/neutron- tempest-plugin-scenario-openvswitch/2a7ab79/testr_results.html https://27020bbcd4882754b192-88656c065c39ed46f44b21a92a1cea67.ssl.cf5.rackcdn.com/800445/7/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/5e597ae/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/800446/9/check/neutron- tempest-plugin-scenario-openvswitch/e7d72c9/testr_results.html https://637b02491f0435a9a86b-ccec73fd7dde7a9826f6a9aeb49ab878.ssl.cf5.rackcdn.com/804397/1/gate/neutron- tempest-plugin-scenario-linuxbridge/64bae23/testr_results.html https://d1b1a7bc5606074c0db2-9f552c22a38891cd59267376a7a41496.ssl.cf5.rackcdn.com/802596/12/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/de03f1f/testr_results.html https://b395fe859a68f8d08e03-e48e76b6f53fcff59de7a7c1c3da6c62.ssl.cf1.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/98bfff3/testr_results.html https://6ee071d84f4801a650d3-2635c9269ad2bde2592553cd282ad960.ssl.cf2.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-linuxbridge/a9282d0/testr_results.html Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20Unexpected%20API%20Error.%20Please%20report%20this%20at%20http%3A%2F%2Fbugs.launchpad.net%2Fnova%2F%20and%20attach%20the%20Nova%20API%20log%20if%20possible.%5C%22 ** Affects: neutron Importance: Critical Status: Confirmed ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1940243 Title: Neutron-tempest-plugin scenario tests - oom-killer is killing mysql process Status in neutron: Confirmed Bug description: It happens pretty often recently that during our scenario tests we are running out of memory and oom-killer is killing mysql process as it is number 1 in memory consumption. That is causing job's failures. It seems for me that it happens when there are running tests which uses vms with advanced image (ubuntu). Maybe we should extract those tests and run them as second stage with "--concurency 1"? Examples of failures: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85a/803462/2/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/85afc13/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/803936/1/gate/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/9445e5f/testr_results.html https://09e003a6a650f320c43d-e30f275ad83ed88289c7399adb6c5ee6.ssl.cf1.rackcdn.com/804236/5/check/neutron- tempest-plugin-scenario-linuxbridge/770722a/testr_results.html https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/797221/1/check/neutron- tempest-plugin-scenario-openvswitch/2a7ab79/testr_results.html https://27020bbcd4882754b192-88656c065c39ed46f44b21a92a1cea67.ssl.cf5.rackcdn.com/800445/7/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/5e597ae/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/800446/9/check/neutron- tempest-plugin-scenario-openvswitch/e7d72c9/testr_results.html https://637b02491f0435a9a86b-ccec73fd7dde7a9826f6a9aeb49ab878.ssl.cf5.rackcdn.com/804397/1/gate/neutron- tempest-plugin-scenario-linuxbridge/64bae23/testr_results.html https://d1b1a7bc5606074c0db2-9f552c22a38891cd59267376a7a41496.ssl.cf5.rackcdn.com/802596/12/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/de03f1f/testr_results.html https://b395fe859a68f8d08e03-e48e76b6f53fcff59de7a7c1c3da6c62.ssl.cf1.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/98bfff3/testr_results.html https://6ee071d84f4801a650d3-2635c9269ad2bde2592553cd282ad960.ssl.cf2.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-linuxbridge/a9282d0/testr_results.html Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20Unexpected%20API%20Error.%20Please%20report%20this%20at%20http%3A%2F%2Fbugs.launchpad.net%2Fnova%2F%20and%20attach%20the%20Nova%20API%20log%20if%20possible.%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1940243/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
