Reviewed: https://review.openstack.org/446991 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9049c32c7c7314e05250551807b4164082da4695 Submitter: Jenkins Branch: master
commit 9049c32c7c7314e05250551807b4164082da4695 Author: Attila Czira <[email protected]> Date: Fri Mar 17 11:43:11 2017 +0100 Stabilizing process monitor function test case The problem seems to be that the TC is starting new processes parallelly but it does not wait for them to start properly instead it tries to kill one of them immediately but in some cases if the startup phase is delayed the process that is about to be killed is not started yet. The fix is to introduce a waiting phase until all the child processes are up and running right after the initial spawning phase. As the problem is not functional the fix is difficult to test. I executed this TC in a loop with this fix for a 5000 times and the problem has not appeared. Without the fix I had 3.8% failure rate. Change-Id: I19a8a4c35efc35a002478204fd24910d73c9c8c2 closes-bug: #1513771 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1513771 Title: functional tests random failure on TestProcessMonitor.test_respawn_handler trying to kill a process Status in neutron: Fix Released Bug description: 2015-11-06 08:16:42.407 | 2015-11-06 08:16:42.364 | ============================== 2015-11-06 08:16:42.408 | 2015-11-06 08:16:42.366 | Failed 1 tests - output below: 2015-11-06 08:16:42.408 | 2015-11-06 08:16:42.367 | ============================== 2015-11-06 08:16:42.408 | 2015-11-06 08:16:42.369 | 2015-11-06 08:16:42.408 | 2015-11-06 08:16:42.370 | neutron.tests.functional.agent.linux.test_process_monitor.TestProcessMonitor.test_respawn_handler 2015-11-06 08:16:42.408 | 2015-11-06 08:16:42.372 | ------------------------------------------------------------------------------------------------- 2015-11-06 08:16:42.409 | 2015-11-06 08:16:42.373 | 2015-11-06 08:16:42.409 | 2015-11-06 08:16:42.374 | Captured traceback: 2015-11-06 08:16:42.409 | 2015-11-06 08:16:42.376 | ~~~~~~~~~~~~~~~~~~~ 2015-11-06 08:16:42.409 | 2015-11-06 08:16:42.377 | Traceback (most recent call last): 2015-11-06 08:16:42.409 | 2015-11-06 08:16:42.378 | File "neutron/tests/functional/agent/linux/test_process_monitor.py", line 108, in test_respawn_handler 2015-11-06 08:16:42.409 | 2015-11-06 08:16:42.380 | self._kill_last_child() 2015-11-06 08:16:42.410 | 2015-11-06 08:16:42.381 | File "neutron/tests/functional/agent/linux/test_process_monitor.py", line 77, in _kill_last_child 2015-11-06 08:16:42.410 | 2015-11-06 08:16:42.383 | self._child_processes[-1].disable() 2015-11-06 08:16:42.410 | 2015-11-06 08:16:42.385 | File "neutron/agent/linux/external_process.py", line 109, in disable 2015-11-06 08:16:42.410 | 2015-11-06 08:16:42.386 | utils.execute(cmd, run_as_root=True) 2015-11-06 08:16:42.410 | 2015-11-06 08:16:42.388 | File "neutron/agent/linux/utils.py", line 159, in execute 2015-11-06 08:16:42.411 | 2015-11-06 08:16:42.389 | raise RuntimeError(m) 2015-11-06 08:16:42.411 | 2015-11-06 08:16:42.391 | RuntimeError: 2015-11-06 08:16:42.411 | 2015-11-06 08:16:42.393 | Command: ['sudo', 'kill', '-9', 'None'] 2015-11-06 08:16:42.411 | 2015-11-06 08:16:42.394 | Exit code: 1 2015-11-06 08:16:42.413 | 2015-11-06 08:16:42.396 | Stdin: 2015-11-06 08:16:42.415 | 2015-11-06 08:16:42.397 | Stdout: 2015-11-06 08:16:42.416 | 2015-11-06 08:16:42.399 | Stderr: kill: failed to parse argument: 'None' 2015-11-06 08:16:42.418 | 2015-11-06 08:16:42.400 | 2015-11-06 08:16:42.419 | 2015-11-06 08:16:42.402 | 2015-11-06 08:16:42.440 | 2015-11-06 08:16:42.403 | 2015-11-06 08:16:42.441 | 2015-11-06 08:16:42.405 | Captured pythonlogging: 2015-11-06 08:16:42.441 | 2015-11-06 08:16:42.407 | ~~~~~~~~~~~~~~~~~~~~~~~ 2015-11-06 08:16:42.441 | 2015-11-06 08:16:42.408 | 2015-11-06 08:14:13,232 ERROR [neutron.agent.linux.utils] Unable to convert value in /tmp/tmpPzSZY1/tmpLtA4wt/external/pids/test-uuid-1.pid 2015-11-06 08:16:42.441 | 2015-11-06 08:16:42.410 | 2015-11-06 08:14:13,243 ERROR [neutron.agent.linux.utils] 2015-11-06 08:16:42.441 | 2015-11-06 08:16:42.411 | Command: ['sudo', 'kill', '-9', 'None'] 2015-11-06 08:16:42.441 | 2015-11-06 08:16:42.413 | Exit code: 1 2015-11-06 08:16:42.442 | 2015-11-06 08:16:42.414 | Stdin: 2015-11-06 08:16:42.442 | 2015-11-06 08:16:42.416 | Stdout: 2015-11-06 08:16:42.442 | 2015-11-06 08:16:42.418 | Stderr: kill: failed to parse argument: 'None' 2015-11-06 08:16:42.442 | 2015-11-06 08:16:42.420 | 2015-11-06 08:16:42.442 | 2015-11-06 08:16:42.421 | 2015-11-06 08:16:42.442 | 2015-11-06 08:16:42.423 | 2015-11-06 08:16:42.443 | 2015-11-06 08:16:42.425 | To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1513771/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
