[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
Cloud-init now prefers dhcpcd over dhclient, which includes RFC 5227 support. Closing. ** Changed in: cloud-init (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in MAAS: Invalid Status in cloud-init package in Ubuntu: Fix Released Status in isc-dhcp package in Ubuntu: Confirmed Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
Marking bug as Invalid for MAAS as per previous comment, incomplete status since, and input from Björn and Jerzy. ** Changed in: maas Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in MAAS: Invalid Status in cloud-init package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: Confirmed Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
** Tags added: se-00140843 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in MAAS: Incomplete Status in cloud-init package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: Confirmed Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
I'm marking this bug as incomplete for MAAS, since it's not clear what actually needs to be fixed in MAAS. It seems like this needs to be fixed at a lower level. ** Changed in: maas Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in MAAS: Incomplete Status in cloud-init package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: Confirmed Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: isc-dhcp (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in MAAS: New Status in cloud-init package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: Confirmed Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
** Also affects: maas Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in MAAS: New Status in cloud-init package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: Confirmed Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
This causes problems for me as well during maas re-imaging wiht maas 2.9.2. see https://discourse.maas.io/t/changing-ips-and-lack-of- gratuitous-arp-and-the-pain-it-causes/4800 Ideally when pxebooting, pxelinux.0 should send a gratuitous arp and in theory it should solve the issue. Perhaps I'm mistaken... -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in MAAS: New Status in cloud-init package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: Confirmed Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1677668] Re: no GARPs during ephemeral boot
First of all thanks Jay for the great in depth extra-insight! As Sam added, Cloud-Init can't do a lot here since it doesn't get a config to this stage. The only thing it could consider to do is unconditionally always trigger a GARP. But that would be: 1. offending the cloud-init design which is that it is supposed to do what it is told (not more not less) 2. add the dependency to a tool like arping for the image so that cloud-init can issue the GARP So one step back - implementation in dhclient as-is as script: 1. this would still have to be unconditionally, but at least be bound to dhcp 2. it would have the same need for arping (or similar) in the image 2.1 an alternative to the dependency would be to re-implement, but redundancy always is worse I wonder if the following would be an option: You already get your IP address, so you get your dhcp reply; Just the environment doesn't realize you moved. IIRC dhcp can transport extra options, so to get rid of the "unconditional" thing. Could dhcp grow a feature to understand a "dhcp-please-garp" option? Handling this option could be done in a dhclient script then. And on the other side the MAAS dhcp server could present this option. That at least would make it 1. conditionally only where requested 2. maas has the control since it is the dhcp 3. not affect environments where things are not providing this option (for SRU-ability) Making the cloud-init a wishlist item, since it seems a feature request more than a bug there and also as outlined above not really fixable there. Instead adding a dhclient task. All of this has to consider, will more of systemd-networkd replace dhclient? If so considerations have to be made for that or it will regress as soon as things are switched. ** Also affects: isc-dhcp (Ubuntu) Importance: Undecided Status: New ** Changed in: cloud-init (Ubuntu) Status: New => Incomplete ** Changed in: cloud-init (Ubuntu) Importance: Undecided => Wishlist -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1677668 Title: no GARPs during ephemeral boot Status in cloud-init package in Ubuntu: Incomplete Status in isc-dhcp package in Ubuntu: New Bug description: Deploys time out with an error on the console that says, "Can not apply stage final, no datasource found! Likely bad things to come!" How to duplicate: MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) 1) Rack Controller and Region Controller in different VLANs 2) Use Cisco ASA as the router with "ARP Inspection" enabled 3) Clear the router ARP cache 4) Deploy 2 maas machines with interfaces set to "Static assign" 5) Observe deploys successfully 6) Release both machines and swap IP's. 7) Redeploy the same 2 machines 8) Observe deploy failure with the machine consoles stuck in the "ubuntu login" screen with "Can not apply stage final, no datasource Found! Likely bad things to come!" The root cause is that during ephemeral PXE booting, no GARPs are sent, which in our environment will cause our router (Cisco ASA) to hold on to ARP table entries until it expires (default= 4 hours). Then combined with ASA feature "ARP Inspection" will drop packets from a MaaS machine using the previously used IP from a different MaaS machine. The ephemeral boot image ephemeral-ubuntu-amd64-ga-16.04-xenial-daily. Running tcpdump on the Rack Controller, showed no GARPs from the deploying MaaS machine. If there were GARPs sent, then the router would refresh its ARP cache thus avoiding the ARP Inspection dropping. [Excerpt from Cisco ASA] http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/basic-arp-mac.pdf When you enable ARP inspection, the ASA compares the MAC address, IP address, and source interface in all ARP packets to static entries in the ARP table, and takes the following actions: • If the IP address, MAC address, and source interface match an ARP entry, the packet is passed through. • If there is a mismatch between the MAC address, the IP address, or the interface, then the ASA drops the packet. • If the ARP packet does not match any entries in the static ARP table, then you can set the ASA to either forward the packet out all interfaces (flood), or to drop the packet. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1677668/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
