Reopening the bz as seeing few failures(5 failures in last 15 days as per https://opensearch.logs.openstack.org) in linuxbridge and openvswitch scenario jobs:- https://1b33868f301e2201a22c-a64bb815b8796eabf8a53948331bd878.ssl.cf5.rackcdn.com/845366/2/check/neutron-tempest-plugin-openvswitch/aa3106c/controller/logs/syslog.txt https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_82d/843649/2/gate/neutron-tempest-plugin-openvswitch-iptables_hybrid/82d4dbe/controller/logs/syslog.txt https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_052/842135/1/check/neutron-tempest-plugin-linuxbridge/0528022/controller/logs/syslog.txt https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f11/844680/1/check/neutron-tempest-plugin-linuxbridge/f111e3f/controller/logs/syslog.txt https://bd6c3da3b1723c5f3502-97cb3b32849366f5bed744685e46b266.ssl.cf1.rackcdn.com/844860/3/check/neutron-tempest-plugin-linuxbridge/35e5065/controller/logs/syslog.txt
In all the above failures, SwapFree: 0 kB and many vms were active(using 256 mb/128 mb each) and mysql utilizing approx 10% of total memory. Also multiple(around 10) neutron-keepalived-state-change processes utilizing approx 12-14% of memory. Considering only seen in master jobs and with timings seems this triggered after https://review.opendev.org/c/openstack/neutron-tempest- plugin/+/836912 as more api tests running along with scenario tests now. The swap used in these jobs is 1 gb, so it can be increased to some extent like 3-4 gb to avoid these issues due to memory shortage. ** Changed in: neutron Status: Fix Released => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1940243 Title: Neutron-tempest-plugin scenario tests - oom-killer is killing mysql process Status in neutron: Triaged Bug description: It happens pretty often recently that during our scenario tests we are running out of memory and oom-killer is killing mysql process as it is number 1 in memory consumption. That is causing job's failures. It seems for me that it happens when there are running tests which uses vms with advanced image (ubuntu). Maybe we should extract those tests and run them as second stage with "--concurency 1"? Examples of failures: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85a/803462/2/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/85afc13/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/803936/1/gate/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/9445e5f/testr_results.html https://09e003a6a650f320c43d-e30f275ad83ed88289c7399adb6c5ee6.ssl.cf1.rackcdn.com/804236/5/check/neutron- tempest-plugin-scenario-linuxbridge/770722a/testr_results.html https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/797221/1/check/neutron- tempest-plugin-scenario-openvswitch/2a7ab79/testr_results.html https://27020bbcd4882754b192-88656c065c39ed46f44b21a92a1cea67.ssl.cf5.rackcdn.com/800445/7/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/5e597ae/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/800446/9/check/neutron- tempest-plugin-scenario-openvswitch/e7d72c9/testr_results.html https://637b02491f0435a9a86b-ccec73fd7dde7a9826f6a9aeb49ab878.ssl.cf5.rackcdn.com/804397/1/gate/neutron- tempest-plugin-scenario-linuxbridge/64bae23/testr_results.html https://d1b1a7bc5606074c0db2-9f552c22a38891cd59267376a7a41496.ssl.cf5.rackcdn.com/802596/12/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/de03f1f/testr_results.html https://b395fe859a68f8d08e03-e48e76b6f53fcff59de7a7c1c3da6c62.ssl.cf1.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/98bfff3/testr_results.html https://6ee071d84f4801a650d3-2635c9269ad2bde2592553cd282ad960.ssl.cf2.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-linuxbridge/a9282d0/testr_results.html Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20Unexpected%20API%20Error.%20Please%20report%20this%20at%20http%3A%2F%2Fbugs.launchpad.net%2Fnova%2F%20and%20attach%20the%20Nova%20API%20log%20if%20possible.%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1940243/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp