Public bug reported: Neutron quota request implementation can end in a DB lock status. The quota is assigned per resource (port, network, security group, etc.) and per project. When a request is done, a DB lock is set for this (resource, project) tuple. This lock in the DB engine to lock this tuple in all workers of all API servers.
That implies there is a a bottleneck when a high number of requests arrive to the API at the same time. If the number of requests exceeds the number of resources processes, the DB locked transactions will increase indefinitely. This can be seen in the DB executing: $ mysql -e "show processlist;" | egrep "reservations|quotausages" The query used by Neutron to lock this (resource, project) tuple is: UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \ AND quotausages.resource = <resource_type> An improved quota system should be implemented that allow parallel resource request and avoids this DB lock status. NOTE: please check [2][3]. "Neutron does not enforce quotas in such a way that a quota violation like this could never occur". That means even with this restrictive DB locking method, resource overcommit is possible. [1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150 [2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2 [3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661 ** Affects: neutron Importance: Wishlist Status: New ** Changed in: neutron Importance: Undecided => Wishlist ** Description changed: Neutron quota request implementation can end in a DB lock status. The quota is assigned per resource (port, network, security group, etc.) and per project. When a request is done, a DB lock is set for this (resource, project) tuple. This lock in the DB engine to lock this tuple in all workers of all API servers. That implies there is a a bottleneck when a high number of requests arrive to the API at the same time. If the number of requests exceeds the number of resources processes, the DB locked transactions will increase indefinitely. This can be seen in the DB executing: - $ mysql -e "show processlist;" | egrep "reservations|quotausages" - + $ mysql -e "show processlist;" | egrep "reservations|quotausages" The query used by Neutron to lock this (resource, project) tuple is: - UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \ - AND quotausages.resource = <resource_type> + UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \ + AND quotausages.resource = <resource_type> - - An improved quota system should be implemented that allow parallel resource request and avoids this DB lock status. + An improved quota system should be implemented that allow parallel + resource request and avoids this DB lock status. NOTE: please check [2][3]. "Neutron does not enforce quotas in such a way that a quota violation like this could never occur". That means even with this restrictive DB locking method, resource overcommit is possible. - [1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150 [2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2 [3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5 + + Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1926787 Title: [DB] Neutron quota request implementation can end in a lock status Status in neutron: New Bug description: Neutron quota request implementation can end in a DB lock status. The quota is assigned per resource (port, network, security group, etc.) and per project. When a request is done, a DB lock is set for this (resource, project) tuple. This lock in the DB engine to lock this tuple in all workers of all API servers. That implies there is a a bottleneck when a high number of requests arrive to the API at the same time. If the number of requests exceeds the number of resources processes, the DB locked transactions will increase indefinitely. This can be seen in the DB executing: $ mysql -e "show processlist;" | egrep "reservations|quotausages" The query used by Neutron to lock this (resource, project) tuple is: UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \ AND quotausages.resource = <resource_type> An improved quota system should be implemented that allow parallel resource request and avoids this DB lock status. NOTE: please check [2][3]. "Neutron does not enforce quotas in such a way that a quota violation like this could never occur". That means even with this restrictive DB locking method, resource overcommit is possible. [1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150 [2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2 [3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1926787/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
