Reviewed: https://review.openstack.org/298023 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=990eafe4c40e736744494624ca06d137ff6d49ea Submitter: Jenkins Branch: master
commit 990eafe4c40e736744494624ca06d137ff6d49ea Author: Hans Lindgren <[email protected]> Date: Sat Mar 26 19:58:49 2016 +0100 Reduce number of db calls during image cache manager periodic task Make a single db call to get bdms for all instances instead of one call per instance. Change-Id: I74864b398f2d17a24b9ed676945183401e9872a0 Closes-Bug: #1484847 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1484847 Title: image_cache_manager message storm Status in OpenStack Compute (nova): Fix Released Bug description: The image_cache_manager periodic task running on behalf of the n-cpu. image_cache_manager queries all instances which uses the same file-system as him. (The message may contain all compute nodes in the region, if they are using the same shared pNFS) https://github.com/openstack/nova/blob/b91f3f60997dddb2f7c2fc007fe02b7dff1e0224/nova/compute/manager.py#L6333 After all instance received it does looped query via rpc (typically one line response selects). https://github.com/openstack/nova/blob/b91f3f60997dddb2f7c2fc007fe02b7dff1e0224/nova/virt/imagecache.py#L105 At the end it will just needs to know which image is used. If we consider a default settings on 1024 compute node with shared filesystem where each hosts 16 vm we will have nr_vm_per_node = 16 nr_vm = nr_cpu_node * nr_vm_per_node nr_cpu_node * nr_vm / interval_sec 1024*16384/2400 = 6990.50 message/sec. It will take down the nova conductor queue. https://github.com/openstack/nova/blob/b91f3f60997dddb2f7c2fc007fe02b7dff1e0224/nova/compute/manager.py#L6329 Mentions some future re-factoring, but that TODO note is ~3 years old. The looped BlockDeviceMappingList messages MUST be eliminated! One option is to remote the whole statistic calculation to the service which has direct DB connection and able to select multiple related BlockDeviceMapping. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1484847/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
