Public bug reported: Ran into this while hacking on something locally and running the server groups functional tests:
============================== Failed 1 tests - output below: ============================== nova.tests.functional.test_server_group.ServerGroupTestV215.test_rebuild_with_anti_affinity ------------------------------------------------------------------------------------------- Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 19:45:29,525 ERROR [nova.scheduler.utils] Error from last host: host2 (node host2): ['Traceback (most recent call last):\n', ' File "nova/compute/manager.py", line 1831, in _do_build_and_run_instance\n filter_properties)\n', ' File "nova/compute/manager.py", line 2061, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', 'RescheduledException: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Anti-affinity instance group policy was violated.\n'] 19:45:29,526 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: 'NoneType' object is not iterable 19:45:29,527 WARNING [nova.scheduler.utils] Setting instance to ERROR state. Two instances are being booted simultaneously and both land on the same host, so the second one will fail the late affinity check and raise a RescheduledException to be rescheduled to another host. But conductor fails to do that because the 'group_members' key doesn't exist in filter_properties and an attempt to make a list out of it fails [1]. In the past, code [2] was added 'group_members' to filter_properties to handle affinity and a more recent change removed most of it but missed 'group_members' [3]. So nothing is ever setting filter_properties['group_members'] but RequestSpec.from_primitives() expects it to be there and blows up trying to make a list from None. [1] https://github.com/openstack/nova/blob/ad6d339/nova/objects/request_spec.py#L205 [2] https://review.openstack.org/#/c/148277 [3] https://review.openstack.org/#/c/469037 ** Affects: nova Importance: High Status: Confirmed ** Affects: nova/pike Importance: High Status: Confirmed ** Tags: affinity requestspec reschedule server-groups -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1719730 Title: Reschedule after the late affinity check fails with "'NoneType' object is not iterable" Status in OpenStack Compute (nova): Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Bug description: Ran into this while hacking on something locally and running the server groups functional tests: ============================== Failed 1 tests - output below: ============================== nova.tests.functional.test_server_group.ServerGroupTestV215.test_rebuild_with_anti_affinity ------------------------------------------------------------------------------------------- Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 19:45:29,525 ERROR [nova.scheduler.utils] Error from last host: host2 (node host2): ['Traceback (most recent call last):\n', ' File "nova/compute/manager.py", line 1831, in _do_build_and_run_instance\n filter_properties)\n', ' File "nova/compute/manager.py", line 2061, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', 'RescheduledException: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Build of instance c249e39f-0d38-40ce-860d-6c72cdeba436 was re-scheduled: Anti-affinity instance group policy was violated.\n'] 19:45:29,526 WARNING [nova.scheduler.utils] Failed to compute_task_build_instances: 'NoneType' object is not iterable 19:45:29,527 WARNING [nova.scheduler.utils] Setting instance to ERROR state. Two instances are being booted simultaneously and both land on the same host, so the second one will fail the late affinity check and raise a RescheduledException to be rescheduled to another host. But conductor fails to do that because the 'group_members' key doesn't exist in filter_properties and an attempt to make a list out of it fails [1]. In the past, code [2] was added 'group_members' to filter_properties to handle affinity and a more recent change removed most of it but missed 'group_members' [3]. So nothing is ever setting filter_properties['group_members'] but RequestSpec.from_primitives() expects it to be there and blows up trying to make a list from None. [1] https://github.com/openstack/nova/blob/ad6d339/nova/objects/request_spec.py#L205 [2] https://review.openstack.org/#/c/148277 [3] https://review.openstack.org/#/c/469037 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1719730/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
