On 12/13/2016 02:45 PM, Gustavo Randich wrote:
Hi Openstackers,
We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone can
help ;)
When our hosts boots up, the ARP cache population loop of L3 Agent is delaying
the start of neutron-ns-metadata-proxy for around a minute -- see logs below;
then, when nova-compute launches VMs, all of cloud-init runs fail with timeout
when reading metadata
Hi Gustavo,
We had seen a similar slowdown with DVR and the ARP cache, see:
https://bugs.launchpad.net/neutron/+bug/1511134
https://review.openstack.org/#/c/239543/
We decided against that approach in favor of using privsep and the pyroute2
library. That adoption has not been as fast as we hoped, so it is probably time
to re-visit this decision and possibly resurrect that change.
-Brian
To workaround this, we've made a systemd unit on which nova-compute is
dependent; this unit waits for ns-metadata-proxy process to appear, and only
then nova-compute starts
Curiously, in dvr_local_router.py, in _update_arp_entry function, there is a
comment saying "# TODO(mrsmith): optimize the calls below for bulk calls"...
By now we have a single virtual router with 170 VMs, but the number of VMs will
grow, so my questions are
Should this be issue of concern?
Is there a better / faster / bulk way to execute those "ip neigh" commands?
Or simply, metadata proxy should launch before ARP cache population?
PD: I've also seen (obviously) this ARP cache population in the L3 agent of
Neutron Nodes, and I hope it does not affect / delay the HA failover
mechanism... (didn't test yet)
# journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf,,g' | head -25
Dec 13 13:33:43 e71-host15 sudo[20157]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid
--metadata_proxy_socket=/var/
Dec 13 13:33:55 e71-host15 sudo[20309]: ip -o netns list
Dec 13 13:33:55 e71-host15 sudo[20315]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv4.ip_forward=1
Dec 13 13:33:55 e71-host15 sudo[20322]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv6.conf.all.forwarding=1
Dec 13 13:33:56 e71-host15 sudo[20331]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show rfp-6149559f-f
Dec 13 13:33:56 e71-host15 sudo[20336]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:56 e71-host15 sudo[20342]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:56 e71-host15 sudo[20345]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip addr show qr-24f3070a-d4
permanent
Dec 13 13:33:56 e71-host15 sudo[20348]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route list dev qr-24f3070a-d4
scope link
Dec 13 13:33:56 e71-host15 sudo[20354]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -6 route list dev qr-24f3070a-d4
scope link
Dec 13 13:33:56 e71-host15 sudo[20357]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 arping -A -I qr-24f3070a-d4 -c 3 -w
4.5 10.96.0.1
Dec 13 13:33:57 e71-host15 sudo[20368]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:57 e71-host15 sudo[20372]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.100
lladdr fa:16:3e:1b:d6:cd nud permanent dev qr-24f3070a-d4
Dec 13 13:33:57 e71-host15 sudo[20375]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:57 e71-host15 sudo[20378]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.101
lladdr fa:16:3e:b4:12:28 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20384]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20387]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.102
lladdr fa:16:3e:3f:bb:58 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20390]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20393]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.103
lladdr fa:16:3e:5a:90:67 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20399]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20402]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.104
lladdr fa:16:3e:ba:fc:f3 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20405]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:59 e71-host15 sudo[20411]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.105
lladdr fa:16:3e:0a:16:d1 nud permanent dev qr-24f3070a-d4
...
...
...
# journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf,,g' | tail -25
Dec 13 13:34:51 e71-host15 sudo[21771]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.1.83
lladdr fa:16:3e:8d:d2:7d nud permanent dev qr-24f3070a-d4
Dec 13 13:34:51 e71-host15 sudo[21777]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default via
10.96.0.2 dev qr-24f3070a-d4 table 174063617
Dec 13 13:34:51 e71-host15 sudo[21780]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
Dec 13 13:34:52 e71-host15 sudo[21783]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
Dec 13 13:34:52 e71-host15 sudo[21786]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth 1
-type l -printf %f
Dec 13 13:34:52 e71-host15 sudo[21789]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv4.conf.all.send_redirects=0
Dec 13 13:34:52 e71-host15 sudo[21792]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default via
10.96.0.2 dev qr-24f3070a-d4 table 174063617
Dec 13 13:34:52 e71-host15 sudo[21795]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
Dec 13 13:34:52 e71-host15 sudo[21801]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
Dec 13 13:34:53 e71-host15 sudo[21804]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:34:53 e71-host15 sudo[21807]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.2
lladdr fa:16:3e:d9:34:cb nud permanent dev qr-24f3070a-d4
Dec 13 13:34:53 e71-host15 sudo[21817]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth 1
-type l -printf %f
Dec 13 13:34:53 e71-host15 sudo[21822]: ip netns exec
fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
Dec 13 13:34:53 e71-host15 sudo[21827]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
Dec 13 13:34:53 e71-host15 sudo[21833]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
Dec 13 13:34:54 e71-host15 sudo[21836]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
Dec 13 13:34:54 e71-host15 sudo[21839]: ip netns exec
fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
Dec 13 13:34:54 e71-host15 sudo[21844]: ip netns exec
fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
Dec 13 13:34:54 e71-host15 sudo[21849]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
Dec 13 13:34:54 e71-host15 sudo[21852]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
Dec 13 13:34:54 e71-host15 sudo[21855]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
Dec 13 13:34:54 e71-host15 sudo[21861]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
Dec 13 13:34:55 e71-host15 sudo[21864]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
Dec 13 13:34:55 e71-host15 sudo[21869]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
Dec 13 13:34:55 e71-host15 sudo[21877]: ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--router_id=6149559f-fa54-493c-bf37-7d1827181228 --state_path=/var/lib/neutron
--metadata_port=9697 --metadata_proxy_user=118 --metadata_proxy_group=120
--debug --verbose --use-syslog --syslog-log-facility=LOG_LOCAL4
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
