[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Launchpad Bug Tracker]

This bug was fixed in the package klibc - 2.0.3-0ubuntu1.14.04.3

---
klibc (2.0.3-0ubuntu1.14.04.3) trusty; urgency=medium

  * debian/patches/dhcp-one-socket-per-interface.patch: Use separate
sockets for DHCP from multiple interfaces.  Thanks to Jay Vosburgh
. (LP: #1652348)

 -- Mathieu Trudel-Lapierre   Tue, 13 Jun 2017
11:11:44 -0400

** Changed in: klibc (Ubuntu Trusty)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Trusty:
  Fix Released
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Released
Status in klibc source package in Zesty:
  Fix Released

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Francis Ginther]

The trusty patch, 2.0.3-0ubuntu1.14.04.3 in -proposed, passed testing.

** Tags added: verification-done-trusty

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Trusty:
  Fix Committed
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Released
Status in klibc source package in Zesty:
  Fix Released

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Launchpad Bug Tracker]

This bug was fixed in the package klibc - 2.0.4-8ubuntu3.1

---
klibc (2.0.4-8ubuntu3.1) yakkety; urgency=medium

  * debian/patches/dhcp-one-socket-per-interface.patch: Use separate
sockets for DHCP from multiple interfaces.  Thanks to Jay Vosburgh
.  Closes LP: #1652348.

 -- Steve Langasek   Tue, 24 Jan 2017
11:38:13 -0800

** Changed in: klibc (Ubuntu Yakkety)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Trusty:
  Fix Committed
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Released
Status in klibc source package in Zesty:
  Fix Released

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Mathieu Trudel-Lapierre]

Verification-done-yakkety:
I've verified that ipconfig can properly get an IP address via DHCP when there 
are multiple interfaces.

I verified klibc 2.0.3-0ubuntu1.14.04.3.

** Tags added: verification-done-yakkety

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Trusty:
  Fix Committed
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Committed
Status in klibc source package in Zesty:
  Fix Released

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Steve Langasek]

Hello Paul, or anyone else affected,

Accepted klibc into trusty-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/klibc/2.0.3-0ubuntu1.14.04.3 in a
few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: klibc (Ubuntu Trusty)
   Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Trusty:
  Fix Committed
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Committed
Status in klibc source package in Zesty:
  Fix Released

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to 

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Mathieu Trudel-Lapierre]

** Also affects: klibc (Ubuntu Trusty)
   Importance: Undecided
   Status: New

** Changed in: klibc (Ubuntu Trusty)
 Assignee: (unassigned) => Mathieu Trudel-Lapierre (cyphermox)

** Changed in: klibc (Ubuntu Trusty)
   Status: New => In Progress

** Changed in: klibc (Ubuntu Trusty)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Trusty:
  In Progress
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Committed
Status in klibc source package in Zesty:
  Fix Released

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Eric Desrochers]

** Also affects: klibc (Ubuntu Zesty)
   Importance: Undecided
   Status: New

** Changed in: klibc (Ubuntu Zesty)
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Committed
Status in klibc source package in Zesty:
  Fix Released

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Paul Graydon]

That would be a different bug, unfortunately.

Mine was specifically down to ipconfig not handling multiple network
interfaces correctly, triaged and successfully fixed by Canonical, and
exhaustively validated in our infrastructure.

Roughly speaking, it would quickly loop through the interfaces sending
out DHCP requests, and then listen on all for the responses, but it was
only able to track the request for one interface, which was whatever the
last interface it sent the request out of.  There's a session id
associated with it, and if that didn't match it would drop the packets.

Booting became dependent on the order of network interfaces returned by
the kernel, something that isn't guaranteed in any way, and explains why
it was working with some kernels and not with others, as it would only
work if the one interface that got DHCP responses was the last one
returned by the kernel.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Committed

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Paul Graydon]

Note: I've also discovered (unsurprisingly, I guess?) that I see the
exact same behaviour on Ubuntu 14.04.  Can this fix be backported?

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Committed

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Launchpad Bug Tracker]

This bug was fixed in the package klibc - 2.0.4-8ubuntu1.16.04.3

---
klibc (2.0.4-8ubuntu1.16.04.3) xenial; urgency=medium

  * debian/patches/dhcp-one-socket-per-interface.patch: Use separate
sockets for DHCP from multiple interfaces.  Thanks to Jay Vosburgh
.  Closes LP: #1652348.

 -- Steve Langasek   Tue, 24 Jan 2017
11:40:58 -0800

** Changed in: klibc (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Fix Released
Status in klibc source package in Yakkety:
  Fix Committed

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Brian Murray]

** Tags added: verification-done-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Fix Committed
Status in klibc source package in Yakkety:
  Fix Committed

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Paul Graydon]

I've tested and confirmed that this solves the issue on 16.04

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Fix Committed
Status in klibc source package in Yakkety:
  Fix Committed

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Brian Murray]

Hello Paul, or anyone else affected,

Accepted klibc into yakkety-proposed. The package will build now and be
available at https://launchpad.net/ubuntu/+source/klibc/2.0.4-8ubuntu3.1
in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: klibc (Ubuntu Yakkety)
   Status: New => Fix Committed

** Tags added: verification-needed

** Changed in: klibc (Ubuntu Xenial)
   Status: Triaged => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Fix Committed
Status in klibc source package in Yakkety:
  Fix Committed

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : 

Re: [Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Steve Langasek]

On Tue, Jan 24, 2017 at 10:51:56PM -, Paul Graydon wrote:
> Given this is a fundamental bug in klibc, is there a plan to try to
> upstream this patch?

It has been upstreamed to the Debian git repository.  Given that we will be
looking at removing klibc from the initramfs altogether in future releases
(in favor of just using isc-dhcp), I don't intend to pursue it any further
than that.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Triaged

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Paul Graydon]

Given this is a fundamental bug in klibc, is there a plan to try to
upstream this patch?

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Triaged

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Launchpad Bug Tracker]

This bug was fixed in the package klibc - 2.0.4-8ubuntu4

---
klibc (2.0.4-8ubuntu4) zesty; urgency=medium

  * debian/patches/dhcp-one-socket-per-interface.patch: Use separate
sockets for DHCP from multiple interfaces.  Thanks to Jay Vosburgh
.  Closes LP: #1652348.

 -- Steve Langasek   Tue, 24 Jan 2017
11:38:13 -0800

** Changed in: klibc (Ubuntu)
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Fix Released
Status in klibc source package in Xenial:
  Triaged

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Steve Langasek]

** Description changed:

- Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been
- (re?)introduced that is breaking dhcp booting in the initrd environment.
- This is stopping instances that use iscsi storage from being able to
- connect.
+ [SRU justification]
+ Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.
+ 
+ [Test case]
+ Detailed reproducer described in 
+ .
+ 
+ [Regression potential]
+ Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.
+ 
+ 
+ Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.
  
  Over serial console it outputs:
  
  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up
  
  with increasing delays until it fails.  At which point a simple ipconfig
  -t dhcp -d "ens2f0"  works.  The console output is slightly garbled but
  should give you an idea:
  
  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi
  
  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig command
  is issued manually, an identical dhcp request and response happens, only
  this time it is accepted.  It doesn't appear to be that the messages are
  being sent and received incorrectly, just silently ignored by ipconfig.
  
  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we dropped
  it, at which point we discovered the original bug was no longer an
  issue.
  
  Putting "ip=dhcp" back on with this kernel no longer fixes the problem.
  
  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.
  
  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts
  
  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  In Progress
Status in klibc source package in Xenial:
  Triaged

Bug description:
  [SRU justification]
  Changes to ordering of kernel enumeration of network interfaces, which may 
happen in any release, can regress network configuration from an initramfs.  
Support for netbooting should not depend on interface order, it should work 
reliably on all systems.

  [Test case]
  Detailed reproducer described in 
  .

  [Regression potential]
  Moderate regression potential, because of a relatively large patch touching a 
not-widely-used but still critical piece of code.  Regression testing should 
include verifying that MAAS-booted cloud images still work as expected in a 
variety of environments.

  
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been (re?)introduced 
that is breaking dhcp booting in the initrd environment.  This is stopping 
instances that use iscsi storage from being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Ubuntu Foundations Team Bug Bot]

The attachment "klibc-fix-1.patch" seems to be a patch.  If it isn't,
please remove the "patch" flag from the attachment, remove the "patch"
tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the
team.

[This is an automated message performed by a Launchpad user owned by
~brian-murray, for any issues please contact him.]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  In Progress
Status in klibc source package in Xenial:
  Triaged

Bug description:
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been
  (re?)introduced that is breaking dhcp booting in the initrd
  environment.  This is stopping instances that use iscsi storage from
  being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Jay Vosburgh]

Patch proposal to modify ipconfig to use one packet socket per interface


** Patch added: "klibc-fix-1.patch"
   
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+attachment/4806861/+files/klibc-fix-1.patch

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  In Progress
Status in klibc source package in Xenial:
  Triaged

Bug description:
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been
  (re?)introduced that is breaking dhcp booting in the initrd
  environment.  This is stopping instances that use iscsi storage from
  being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Jay Vosburgh]

** Changed in: klibc (Ubuntu)
   Status: Confirmed => In Progress

** Tags removed: kernel-bug-exists-upstream kernel-bug-exists-
upstream-4.10-rc1

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  In Progress
Status in klibc source package in Xenial:
  Triaged

Bug description:
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been
  (re?)introduced that is breaking dhcp booting in the initrd
  environment.  This is stopping instances that use iscsi storage from
  being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu kernel bisect offending commit:
  # first bad commit: [fd4b5fa6e3487d15ede746f92601af008b2abbc0] mnt: Add a per 
mount namespace limit on the number of mounts

  Ubuntu kernel bisect offending commit submission:
  https://lkml.org/lkml/2016/10/5/308

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348/+subscriptions

-- 
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

[Touch-packages] [Bug 1652348] Re: initrd dhcp fails / ignores valid response

[Jay Vosburgh]

I have instrumented ipconfig, and determined that the ultimate source of the 
problem
is that, for the case of multiple interfaces, ipconfig has a dependency on the 
kernel's probe order of the network interfaces.

For whatever reason, the -31 kernel probes the network devices in one
order (e.g., ens3 then ens4), and the -57 kernel in the other order
(ens4 first then ens3).

The probe order of network devices (and PCI devices in general) is
explicitly not defined, and so this is not a bug in the kernel itself;
ipconfig is failing due to its dependency on a specific enumeration
order.

The issue in ipconfig is that it is using a single packet socket to
attempt to multiplex packet traffic on multiple interfaces.  Presuming
that ens3 will answer DHCP and ens4 will not, for the case that works,
the order ends up being something like:

send DHCP request on ens3
send DHCP request on ens4
[ system gets DHCP response via ens3 ]
try to receive DHCP reply sent by peer for ens3; this matches, and all is happy

For the case that it fails, the sequence is roughly:

send DHCP request on ens4
send DHCP request on ens3
[ system gets DHCP response via ens3 ]
try to receive DHCP reply sent by peer for ens4; the reply is actually for 
ens3, so ipconfig
throws it away (as the XID, et al, don't match what is expected for the ens4 
DHCP request).

This repeats until ipconfig gives up.

As I said above, the issue is that ipconfig is trying to multiplex
traffic for two interfaces on one packet socket.  This is fine for
sending, but for receiving on an unbound packet socket, there is no way
to receive a packet sent to a specific interface.  Packets are delivered
to recvfrom/recvmsg in the order received.

I note that ipconfig sets sll.sll_ifindex on the msghdr provided to
recvfrom and recvmsg system calls; perhaps the author believed that this
limits received packets to only packets received on that ifindex.  This
is not the case, and the sll_ifindex passed to recvfrom/recvmsg is
ignored.

I'm looking into whether or not there is an simple fix for this that
will let ipconfig function without major rework to utilize one packet
socket per interface.



** Tags removed: kernel-key

** Package changed: linux (Ubuntu) => klibc (Ubuntu)

** Changed in: klibc (Ubuntu)
   Status: Triaged => Confirmed

** Changed in: klibc (Ubuntu)
 Assignee: (unassigned) => Jay Vosburgh (jvosburgh)

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to klibc in Ubuntu.
https://bugs.launchpad.net/bugs/1652348

Title:
  initrd dhcp fails / ignores valid response

Status in klibc package in Ubuntu:
  Confirmed
Status in klibc source package in Xenial:
  Triaged

Bug description:
  Between kernel versions 4.4.0-53 and 4.4.0-57 a bug has been
  (re?)introduced that is breaking dhcp booting in the initrd
  environment.  This is stopping instances that use iscsi storage from
  being able to connect.

  Over serial console it outputs:

  IP-Config: no response after 2 secs - giving up
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f1 hardware address 90:e2:ba:d1:36:39 mtu 1500 DHCP RARP
  IP-Config: no response after 3 secs - giving up

  with increasing delays until it fails.  At which point a simple
  ipconfig -t dhcp -d "ens2f0"  works.  The console output is slightly
  garbled but should give you an idea:

  (initramfs) ipconfig -t dhcp -[  728.379793] ixgbe :13:00.0 ens2f0: 
changing MTU from 1500 to 9000
  d "ens2f0"
  IP-Config: ens2f0 hardware address 90:e2:ba:d1:36:38 mtu 1500 DHCP RARP
  IP-Config: ens2f0 guessed broadcast address 10.0.1.255
  IP-Config: ens2f0 complete (dhcp from 169.254.169.254):
   addres[  728.980448] ixgbe :13:00.0 ens2f0: detected SFP+: 3
  s: 10.0.1.56broadcast: 10.0.1.255   netmask: 255.255.255.0
   gateway: 10.0.1.1   [  729.148410] ixgbe :13:00.0 ens2f0: NIC Link is Up 
10 Gbps, Flow Control: RX/TX
    dns0 : 169.254.169.254  dns1   : 0.0.0.0
   rootserver: 169.254.169.254 rootpath:
   filename  : /ipxe.efi

  tcpdumps show that dhcp requests are being received from the host, and
  responses sent, but not accepted by the host.  When the ipconfig
  command is issued manually, an identical dhcp request and response
  happens, only this time it is accepted.  It doesn't appear to be that
  the messages are being sent and received incorrectly, just silently
  ignored by ipconfig.

  I was seeing this behaviour earlier this year, which I was able to fix
  by specifying "ip=dhcp" as a kernel parameter.  About a month ago that
  was identified as causing us other problems (long story) and we
  dropped it, at which point we discovered the original bug was no
  longer an issue.

  Putting "ip=dhcp" back on with this kernel no longer fixes the
  problem.

  I've compared the two initrds and effectively the only thing that has
  changed between the two is the kernel components.

  Ubuntu