On 5/19/2017 1:40 PM, Chris Friesen wrote:
Recently we noticed failures in Newton when we attempted to live-migrate
an instance with 16 vifs. We tracked it down to an RPC timeout in nova
which timed out waiting for the 'refresh_cache-%s' lock in
get_instance_nw_info(). This led to a few other discoveries.
First, we have no fair locking in OpenStack. The live migration code
path was waiting for the lock, but the code processing the incoming
"network-changed" events kept getting the lock instead even though they
arrived while the live migration code was already blocked waiting for
the lock.
I'm told that etcd gives us a DLM which is unicorns and rainbows, would
that help us here?
Second, it turns out the cost of processing the "network-changed" events
is astronomical.
1) In Newton nova commit 5de902a was merged to fix evacuate bugs, but it
meant both source and dest compute nodes got the "network-changed"
events. This doubled the number of neutron API calls during a live
migration.
As you noted below, that change was made specifically for evacuate. With
the migration object we know the type of migration and could scope this
behavior to just evacuate. However, I'm sort of confused by that change
- why are we sending external events to the source compute during an
evacuation? Isn't the source compute down and thus can't receive and
process the event?
2) A "network-changed" event is sent from neutron each time something
changes. There are multiple of these events for each vif during a
live-migration. In the current upstream code the only information
passed with the event is the instance id, so nova will loop over all the
ports in the instance and build up all the information about
subnets/floatingIP/fixedIP/etc. for that instance. This results in
O(N^2) neutron API calls where N is the number of vifs in the instance.
While working on the patches you reference in #3 I was also working on
seeing if we can do some bulk queries to Neutron:
https://review.openstack.org/#/c/465792/
It looks like that's not working though. Kevin Benton seemed to think at
the time (it was late the other night) that passing a list of filter
parameters would get turned into an OR in the database query, but I'm
not sure that's happening (see that Tempest failed on that patch). I
don't have a devstack handy but it seems we could prove this via simple
curl requests.
3) mriedem has proposed a patch series
(https://review.openstack.org/#/c/465783 and
https://review.openstack.org/#/c/465787) that would change neutron to
include the port ID, and allow nova to update just that port. This
reduces the cost to O(N), but it's still significant.
In a hardware lab with 4 compute nodes I created 4 boot-from-volume
instances, each with 16 vifs. I then live-migrated them all in
parallel. (The one on compute-0 was migrated to compute-1, the one on
compute-1 was migrated to compute-2, etc.) The aggregate CPU usage for
a few critical components on the controller node is shown below. Note
in particular the CPU usage for neutron--it's using most of 10 CPUs for
~10 seconds, spiking to 13 CPUs. This seems like an absurd amount of
work to do just to update the cache in nova.
Labels:
L0: neutron-server
L1: nova-conductor
L2: beam.smp
L3: postgres
- - - L0 L1 L2 L3
date time dt occ occ occ occ
yyyy/mm/dd hh:mm:ss.dec (s) (%) (%) (%) (%)
2017-05-19 17:51:38.710 2.173 19.75 1.28 2.85 1.96
2017-05-19 17:51:40.012 1.302 1.02 1.75 3.80 5.07
2017-05-19 17:51:41.334 1.322 2.34 2.66 5.25 1.76
2017-05-19 17:51:42.681 1.347 91.79 3.31 5.27 5.64
2017-05-19 17:51:44.035 1.354 40.78 7.27 3.48 7.34
2017-05-19 17:51:45.406 1.371 7.12 21.35 8.66 19.58
2017-05-19 17:51:46.784 1.378 16.71 196.29 6.87 15.93
2017-05-19 17:51:48.133 1.349 18.51 362.46 8.57 25.70
2017-05-19 17:51:49.508 1.375 284.16 199.30 4.58 18.49
2017-05-19 17:51:50.919 1.411 512.88 17.61 7.47 42.88
2017-05-19 17:51:52.322 1.403 412.34 8.90 9.15 19.24
2017-05-19 17:51:53.734 1.411 320.24 5.20 10.59 9.08
2017-05-19 17:51:55.129 1.396 304.92 2.27 10.65 10.29
2017-05-19 17:51:56.551 1.422 556.09 14.56 10.74 18.85
2017-05-19 17:51:57.977 1.426 979.63 43.41 14.17 21.32
2017-05-19 17:51:59.382 1.405 902.56 48.31 13.69 18.59
2017-05-19 17:52:00.808 1.425 1140.99 74.28 15.12 17.18
2017-05-19 17:52:02.238 1.430 1013.91 69.77 16.46 21.19
2017-05-19 17:52:03.647 1.409 964.94 175.09 15.81 27.23
2017-05-19 17:52:05.077 1.430 838.15 109.13 15.70 34.12
2017-05-19 17:52:06.502 1.425 525.88 79.09 14.42 11.09
2017-05-19 17:52:07.954 1.452 614.58 38.38 12.20 17.89
2017-05-19 17:52:09.380 1.426 763.25 68.40 12.36 16.08
2017-05-19 17:52:10.825 1.445 901.57 73.59 15.90 41.12
2017-05-19 17:52:12.252 1.427 966.15 42.97 16.76 23.07
2017-05-19 17:52:13.702 1.450 902.40 70.98 19.66 17.50
2017-05-19 17:52:15.173 1.471 1023.33 59.71 19.78 18.91
2017-05-19 17:52:16.605 1.432 1127.04 64.19 16.41 26.80
2017-05-19 17:52:18.046 1.442 1300.56 68.22 16.29 24.39
2017-05-19 17:52:19.517 1.471 1055.60 71.74 14.39 17.09
2017-05-19 17:52:20.983 1.465 845.30 61.48 15.24 22.86
2017-05-19 17:52:22.447 1.464 1027.33 65.53 15.94 26.85
2017-05-19 17:52:23.919 1.472 1003.08 56.97 14.39 28.93
2017-05-19 17:52:25.367 1.448 702.50 45.42 11.78 20.53
2017-05-19 17:52:26.814 1.448 558.63 66.48 13.22 29.64
2017-05-19 17:52:28.276 1.462 620.34 206.63 14.58 17.17
2017-05-19 17:52:29.749 1.473 555.62 110.37 10.95 13.27
2017-05-19 17:52:31.228 1.479 436.66 33.65 9.00 21.55
2017-05-19 17:52:32.685 1.456 417.12 87.44 13.44 12.27
2017-05-19 17:52:34.128 1.443 368.31 87.08 11.95 14.70
2017-05-19 17:52:35.558 1.430 171.66 11.67 9.28 13.36
2017-05-19 17:52:36.976 1.417 231.82 10.57 7.03 14.08
2017-05-19 17:52:38.413 1.438 241.14 77.78 6.86 15.34
2017-05-19 17:52:39.827 1.413 85.01 63.72 5.85 14.01
2017-05-19 17:52:41.200 1.373 3.31 3.43 7.18 1.78
2017-05-19 17:52:42.556 1.357 60.68 2.94 6.51 6.16
2017-05-19 17:52:44.019 1.463 24.23 5.94 3.45 3.15
2017-05-19 17:52:45.376 1.356 0.93 3.91 5.13 0.83
2017-05-19 17:52:46.699 1.323 7.68 4.12 5.43 0.45
2017-05-19 17:52:48.033 1.334 5.85 1.70 6.00 1.91
2017-05-19 17:52:49.373 1.341 66.28 2.37 4.49 16.40
2017-05-19 17:52:50.715 1.342 31.67 3.03 3.66 6.91
2017-05-19 17:52:52.023 1.308 2.80 2.35 3.30 10.76
2017-05-19 17:52:53.330 1.307 6.94 5.78 3.25 2.30
2017-05-19 17:52:54.699 1.368 3.11 2.67 8.34 1.01
2017-05-19 17:52:56.049 1.351 23.14 2.28 2.83 5.30
2017-05-19 17:52:57.434 1.384 46.86 5.02 6.27 11.93
2017-05-19 17:52:58.803 1.370 3.78 10.26 3.08 2.08
2017-05-19 17:53:00.206 1.403 66.09 8.20 4.07 1.27
2017-05-19 17:53:01.542 1.336 63.71 9.70 3.17 4.89
2017-05-19 17:53:02.855 1.312 21.53 3.99 4.33 5.03
It seems like it should be possible to reduce the amount of work
involved here. One possibility would be to partially revert nova commit
5de902a for the live-migration case, since it seemed to work fine in
Mitaka and earlier. Another possibility would be to include additional
information about what changed in the "network-changed" event, which
would reduce the amount of queries that nova would need to do.
As noted above, I'm curious how sending events to the source compute
during an evacuate, when the source compute is supposed to be down,
helps anything? Maybe artom can help clarify.
As for the live migration case, looking at related changes around that
time it looks like what was really needed was the
network-vif-(un)plugged events going to both hosts, but as a result
we're sending all events, including network-changed (which happens when
a port is updated, which could happen when it's bound/unbound to/from a
host during migration). So we could also consider scoping which types of
events go to both hosts, but that gets a bit hairy in the API making
decisions about what can or should be ignored - seems like that would
open us up to other bugs later.
In a larger cloud this is going to cause major issues for NFV-type
workloads, where instances having many vifs is going to be relatively
common.
Chris
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
