Public bug reported: We have a downstream failure of neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest.test_duplicate_policy_error probably because http timeout is different on RHEL based boxes, set 120 seconds. The reason why it started failing in Stein is the recent bump in the time requiring to complete the retry DB mechanism: https://review.opendev.org/#/c/583527/5 The patch increases the required time for a reply about resource already existing to more than 160 seconds.
That is because Neutron server retries for every exception about existing entry from the DB layer: https://opendev.org/openstack/neutron-lib/src/branch/master/neutron_lib/db/api.py#L119 Steps to reproduce: net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }') openstack network rbac create --type network --action access_as_shared --target-project admin $net_id openstack network rbac create --type network --action access_as_shared --target-project admin $net_id ( yes, it's the same command twice) I don't understand which race scenario the retry mechanism for resource create it tries to solve. However, I can think of a race scenario it introduces: $ openstack network rbac delete 8a00a24e-182a-4e5e-8694-35e66635b581 $ openstack network delete $net_id $ net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }') $ rbac_id=$(openstack network rbac create --type network --action access_as_shared --target-project admin $net_id | awk '/ id /{ print $4 }') $ openstack network rbac create --type network --action access_as_shared --target-project admin $net_id & [1] 31383 $ sleep 10 $ openstack network rbac delete $rbac_id $ fg openstack network rbac create --type network --action access_as_shared --target-project admin $net_id +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | access_as_shared | | id | 13c5b655-e1b5-4c72-a8af-1ed2f9ddcf89 | | location | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': 'cdf84b19b71249ffaffde6627d06da12', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) | | name | None | | object_id | 618108b7-d191-4459-b18a-30b7a65be005 | | object_type | network | | project_id | cdf84b19b71249ffaffde6627d06da12 | | target_project_id | cdf84b19b71249ffaffde6627d06da12 | +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ The result should be that second creation of existing resource should fail and there should not exist any rbac policy. However, the second creation succeeded and there does exist the policy, that should have been deleted. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1831647 Title: Creation of existing resource takes too much time or fails Status in neutron: New Bug description: We have a downstream failure of neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest.test_duplicate_policy_error probably because http timeout is different on RHEL based boxes, set 120 seconds. The reason why it started failing in Stein is the recent bump in the time requiring to complete the retry DB mechanism: https://review.opendev.org/#/c/583527/5 The patch increases the required time for a reply about resource already existing to more than 160 seconds. That is because Neutron server retries for every exception about existing entry from the DB layer: https://opendev.org/openstack/neutron-lib/src/branch/master/neutron_lib/db/api.py#L119 Steps to reproduce: net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }') openstack network rbac create --type network --action access_as_shared --target-project admin $net_id openstack network rbac create --type network --action access_as_shared --target-project admin $net_id ( yes, it's the same command twice) I don't understand which race scenario the retry mechanism for resource create it tries to solve. However, I can think of a race scenario it introduces: $ openstack network rbac delete 8a00a24e-182a-4e5e-8694-35e66635b581 $ openstack network delete $net_id $ net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }') $ rbac_id=$(openstack network rbac create --type network --action access_as_shared --target-project admin $net_id | awk '/ id /{ print $4 }') $ openstack network rbac create --type network --action access_as_shared --target-project admin $net_id & [1] 31383 $ sleep 10 $ openstack network rbac delete $rbac_id $ fg openstack network rbac create --type network --action access_as_shared --target-project admin $net_id +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | access_as_shared | | id | 13c5b655-e1b5-4c72-a8af-1ed2f9ddcf89 | | location | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': 'cdf84b19b71249ffaffde6627d06da12', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) | | name | None | | object_id | 618108b7-d191-4459-b18a-30b7a65be005 | | object_type | network | | project_id | cdf84b19b71249ffaffde6627d06da12 | | target_project_id | cdf84b19b71249ffaffde6627d06da12 | +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ The result should be that second creation of existing resource should fail and there should not exist any rbac policy. However, the second creation succeeded and there does exist the policy, that should have been deleted. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1831647/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
