Hi Mike, I'm no networking or HA expert, but I've added some comments
inline and cc'd Florian Haas (who *is* an HA expert!) to see if he can
help you out...
On 10/29/2014 12:34 AM, Mike Smith wrote:
I’ve been running nova-network in VLAN mode as an active/passive
cluster resource (corosync + rgmanager) on my OpenStack Havana and
Folsom controller pairs for a good long while. This week I found an
oddity that I hadn’t noticed before, and I’d like to ask the
community about it.
When nova-network starts up, it of course launches a dnsmasq process
for each network, which listens on the .1 address of the assigned
network and acts as the gateway for that network. When the
nova-network service is moved to the passive node, nova-network
starts up dnsmasq processes on that node as well, again listening on
the .1 addresses. However, since now both nodes have the .1
addresses configured, they basically take turns ARPing for the
addresses and stealing the traffic from each other. VMs will route
through the “active” node for a minute or so and then suddenly start
routing through the “passive” node. Then the cycle repeats. Among
other things, this results in only one controller at a time being
able to reach the VMs and adds latency to VM traffic when the shift
happens.
It sounds like your failover is not actually failing over. In other
words, it sounds like your previously active node is not being marked as
fully down in order to facilitate the transition to the backup/passive
node. I would expect some minimal disruption during the failover while
the ARP table entries are repopulated when network connectivity to the
old active node is not possible, but it's the "Then the cycle repeats."
part that has me questioning things...
To stop this, I had to manually remove the VLAN interfaces from the
bridges, bring down the bridges, then delete the bridges from the
now-passive node. Things then returned to normal, with all traffic
flowing through the “active” controller and both controllers being
able to reach the VMs.
I have not seen anything in the HA guides about how people are
preventing this situation from occuring - nothing about killing off
dnsmasq or tearing down these network interfaces to prevent the ARP
wars. Anybody else out there experienced this? How are people
handling the situation?
I am considering bringing up arptables to block ARP for the gateway
addresses when cluster failover happens, or alternatively automating
the tear-down of these gateway addresses. Am I missing something
here?
I'll let Florian talk about what is expected of the networking layer
during failover, but I'll just say that we used multi-host nova-network
node in our Folsom deployments to great effect. It was incredibly
reliable, and the nice thing about it was that if nova-network went down
on a compute node, it only affected the VMs running on that particular
compute node. A simple (re)start of nova-network daemon was enough to
bring up tenant networking on the compute node, and there was no
disruption in service to any other VMs on other compute nodes. The
downside was each compute node would use an extra public IP address...
Anyway, just something to think about. The DVR functionality in Neutron
is attempting to achieve some parity with the nova-network multi-host
functionality, so if you're interested in this area, it's something to
keep an eye on.
All the best,
-jay
Thanks,
Mike Smith Principal Engineer, Website Systems Overstock.com
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
