Public bug reported:
In a DVR environment, when lowering the MTU of a network, traffic going
to an instance through a floating IP is broken.
Description:
* Ping traffic to a VM through its FIP works.
* Change the MTU of its network through "neutron net-update <network> --mtu
1440".
* Ping to the same FIP doesn't work.
After a long debugging session with Anil Venkata, we've found that
packets reach br-ex and then they hit this OF rule with normal action:
cookie=0x1f847e4bf0de0aea, duration=70306.532s, table=3,
n_packets=1579251, n_bytes=796614220, idle_age=0, hard_age=65534,
priority=1 actions=NORMAL
We would expect this rule to switch the packet to br-int so that it can be
forwarded to the fip namespace (ie. with dst MAC address set to the floating ip
gw (owner=network:floatingip_agent_gateway):
$ sudo ovs-vsctl list interface
_uuid : 1f2b6e86-d303-42f4-9467-5dab78fc7199
admin_state : down
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
error : []
external_ids : {attached-mac="fa:16:3e:9d:0c:4f",
iface-id="8ec34826-b1a6-48ce-9c39-2fd3e8167eb4", iface-status=active}
name : "fg-8ec34826-b1"
[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl fdb/show br-ex
port VLAN MAC Age
[...]
7 10 fa:16:3e:9d:0c:4f 0
$ sudo ovs-ofctl show br-ex | grep "7("
7(phy-br-ex): addr:36:63:93:fc:af:e2
And from there, to the fip namespace which would route the packet to the
qrouter namespace, etc.
However, when we change the MTU through the following command:
"neutron net-update <network> --mtu 1440"
We see that, after a few seconds, the MAC address of the FIP changes so
when traffic arrives br-ex and NORMAL action is performed, it will not
be output to br-int through the patch-port but instead, through eth1 and
traffic won't work anymore.
[heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
10.0.0.113 ether fa:16:3e:9d:0c:4f C
vlan10
neutron port-set xxxxx --mtu 1440
$ arp -n | grep ".113"
10.0.0.113 ether fa:16:3e:20:f9:85 C
vlan10
When setting the MAC address manually, ping starts working again:
$ arp -s 10.0.0.113 fa:16:3e:9d:0c:4f
$ ping 10.0.0.113
PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.
64 bytes from 10.0.0.113: icmp_seq=1 ttl=62 time=1.17 ms
64 bytes from 10.0.0.113: icmp_seq=2 ttl=62 time=0.561 ms
Additional notes:
When we set the MAC address manually and traffic gets working back
again, lowering the MTU doesn't change the MAC address (we can't see any
gARP's coming through).
When we delete the ARP entry for the FIP and try to ping the FIP, the
wrong MAC address is set.
[heat-admin@overcloud-novacompute-0 ~]$ sudo arp -d 10.0.0.113
[heat-admin@overcloud-novacompute-0 ~]$ ping 10.0.0.113 -c 2
PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.
--- 10.0.0.113 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
[heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
10.0.0.113 ether fa:16:3e:20:f9:85 C
vlan10
** Affects: neutron
Importance: Undecided
Assignee: Daniel Alvarez (dalvarezs)
Status: New
** Tags: l3-dvr-backlog
** Changed in: neutron
Assignee: (unassigned) => Daniel Alvarez (dalvarezs)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1723472
Title:
[DVR] Lowering the MTU breaks FIP traffic
Status in neutron:
New
Bug description:
In a DVR environment, when lowering the MTU of a network, traffic
going to an instance through a floating IP is broken.
Description:
* Ping traffic to a VM through its FIP works.
* Change the MTU of its network through "neutron net-update <network> --mtu
1440".
* Ping to the same FIP doesn't work.
After a long debugging session with Anil Venkata, we've found that
packets reach br-ex and then they hit this OF rule with normal action:
cookie=0x1f847e4bf0de0aea, duration=70306.532s, table=3,
n_packets=1579251, n_bytes=796614220, idle_age=0, hard_age=65534,
priority=1 actions=NORMAL
We would expect this rule to switch the packet to br-int so that it can be
forwarded to the fip namespace (ie. with dst MAC address set to the floating ip
gw (owner=network:floatingip_agent_gateway):
$ sudo ovs-vsctl list interface
_uuid : 1f2b6e86-d303-42f4-9467-5dab78fc7199
admin_state : down
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
error : []
external_ids : {attached-mac="fa:16:3e:9d:0c:4f",
iface-id="8ec34826-b1a6-48ce-9c39-2fd3e8167eb4", iface-status=active}
name : "fg-8ec34826-b1"
[heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl fdb/show br-ex
port VLAN MAC Age
[...]
7 10 fa:16:3e:9d:0c:4f 0
$ sudo ovs-ofctl show br-ex | grep "7("
7(phy-br-ex): addr:36:63:93:fc:af:e2
And from there, to the fip namespace which would route the packet to the
qrouter namespace, etc.
However, when we change the MTU through the following command:
"neutron net-update <network> --mtu 1440"
We see that, after a few seconds, the MAC address of the FIP changes
so when traffic arrives br-ex and NORMAL action is performed, it will
not be output to br-int through the patch-port but instead, through
eth1 and traffic won't work anymore.
[heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
10.0.0.113 ether fa:16:3e:9d:0c:4f C
vlan10
neutron port-set xxxxx --mtu 1440
$ arp -n | grep ".113"
10.0.0.113 ether fa:16:3e:20:f9:85 C
vlan10
When setting the MAC address manually, ping starts working again:
$ arp -s 10.0.0.113 fa:16:3e:9d:0c:4f
$ ping 10.0.0.113
PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.
64 bytes from 10.0.0.113: icmp_seq=1 ttl=62 time=1.17 ms
64 bytes from 10.0.0.113: icmp_seq=2 ttl=62 time=0.561 ms
Additional notes:
When we set the MAC address manually and traffic gets working back
again, lowering the MTU doesn't change the MAC address (we can't see
any gARP's coming through).
When we delete the ARP entry for the FIP and try to ping the FIP, the
wrong MAC address is set.
[heat-admin@overcloud-novacompute-0 ~]$ sudo arp -d 10.0.0.113
[heat-admin@overcloud-novacompute-0 ~]$ ping 10.0.0.113 -c 2
PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data.
--- 10.0.0.113 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
[heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113"
10.0.0.113 ether fa:16:3e:20:f9:85 C
vlan10
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1723472/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
