Public bug reported: Input: - OpenStack cluster of 2024.1 release - Total number of VMs = 9k - Total number of security groups = 6.4k - Total number of security groups rules = 122k
Problem description: Nova servers list API exceeded 60s timeout in processing request of retrieving detailed information of 1k servers(default limit). OpenStack SDK equivalent call `conn.compute.servers(all_projects=True, paginated=False)`. The debugging showed that it takes <5s the retrieve all info from Nova's db and all remaining time is wasted by calling Neutron to retrieve information about security groups. Nova's logic to retrieve security group info - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L532 : - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete. - retrieving discovered security groups. Same here, separate call for each 150 items - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L500-L529 . Nova passes fields=["id", "name"] filter to neutron API - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L547 to avoid neutron fetching security group rules which can be a heavy operation. Each such call takes ~9s. https://review.opendev.org/c/openstack/neutron/+/929967 is applied to neutron server's. It improved the case, but has not resolved it. Additional info: Nova uses python-neutronclient library, which in my experiments behaves quicker than openstacksdk. ** Affects: neutron Importance: Undecided Status: New ** Description changed: Input: - OpenStack cluster of 2024.1 release - Total number of VMs = 9k - Total number of security groups = 6.4k - Total number of security groups rules = 122k Problem description: Nova servers list API exceeded 60s timeout in processing request of retrieving detailed information of 1k servers(default limit). OpenStack SDK equivalent call `conn.compute.servers(all_projects=True, paginated=False)`. The debugging showed that it takes <5s the retrieve all info from Nova's db and all remaining time is wasted by calling Neutron to retrieve information about security groups. Nova's logic to retrieve security group info - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L532 : - - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete. + - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete. - retrieving discovered security groups. Same here, separate call for each 150 items - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L500-L529 . Nova passes fields=["id", "name"] filter to neutron API - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L547 to avoid neutron fetching security group rules which can be a heavy operation. Each such call takes ~9s. https://review.opendev.org/c/openstack/neutron/+/929967 is applied to neutron server's. It improved the case, but has not resolved it. + + Additional info: Nova uses python-neutronclient library, which in my + experiments behaves quicker than openstacksdk. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2083682 Title: Slowness of security groups list API Status in neutron: New Bug description: Input: - OpenStack cluster of 2024.1 release - Total number of VMs = 9k - Total number of security groups = 6.4k - Total number of security groups rules = 122k Problem description: Nova servers list API exceeded 60s timeout in processing request of retrieving detailed information of 1k servers(default limit). OpenStack SDK equivalent call `conn.compute.servers(all_projects=True, paginated=False)`. The debugging showed that it takes <5s the retrieve all info from Nova's db and all remaining time is wasted by calling Neutron to retrieve information about security groups. Nova's logic to retrieve security group info - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L532 : - retrieving all ports for servers. Nova does a separate call to neutron for each 150 items to not exceed URL size limit - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L471-L497. Each such call takes less than 0.5s to complete. - retrieving discovered security groups. Same here, separate call for each 150 items - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L500-L529 . Nova passes fields=["id", "name"] filter to neutron API - https://github.com/openstack/nova/blob/stable/2024.1/nova/network/security_group_api.py#L547 to avoid neutron fetching security group rules which can be a heavy operation. Each such call takes ~9s. https://review.opendev.org/c/openstack/neutron/+/929967 is applied to neutron server's. It improved the case, but has not resolved it. Additional info: Nova uses python-neutronclient library, which in my experiments behaves quicker than openstacksdk. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2083682/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
