I reproduced this on an m8g Graviton instance with a delayed ENA probe. The issue is that PR #7065 restricted wait_for_candidate_nics() to hpc7a.96xlarge, so fast-booting Graviton instances (m8g, r8g, c8g) still hit the race and fail with DataSourceNone.
I submitted an upstream PR to expand the allowlist and make the timeout configurable: https://github.com/canonical/cloud-init/pull/7138 I also verified the fix on Ubuntu 26.04 and 26.10 test environments with 0 test regressions. ** Description changed: + [ Impact ] + On fast-booting AWS instances (e.g. Graviton 4 m8g/r8g), cloud-init-local starts before the ENA driver finishes probing PCI. + DataSourceEc2 finds no interfaces, logs "The instance must have at least one eligible NIC", and falls back to DataSourceNone without writing netplan configuration. Instances boot without networking and SSH access. + + [ Fix ] + Add wait_for_candidate_nics() to poll /sys/class/net in DataSourceEc2 when no interfaces are present at boot. + If interfaces are already present, it returns immediately with 0 delay. + + [ Test Plan ] + 1. Delay ENA probe in modprobe: + echo "install ena /usr/bin/sleep 2; /sbin/modprobe --ignore-install ena" > /etc/modprobe.d/ena-delay.conf + Run: cloud-init clean --logs && cloud-init init --local + Without fix: DataSourceEc2 exits False. + With fix: DataSourceEc2 waits for the interface and configures networking. + 2. Run unit tests: + pytest tests/unittests/net/test_init.py tests/unittests/sources/test_ec2.py + + [ Where problems could occur ] + If an instance has no network interfaces attached, cloud-init-local will wait up to 10s before bailing out. The timeout is configurable via datasource/Ec2/wait_for_nic_timeout. + + [ Other Info ] + - stonking debdiff: cloud-init 26.2-0ubuntu2 + - resolute debdiff: cloud-init 26.2-0ubuntu1~26.04.2 + - Upstream PR: https://github.com/canonical/cloud-init/pull/7138 + + --- [ Original Report ] 1) Ubuntu Release: Description: Ubuntu 26.04 LTS Release: 26.04 2) Package Version: cloud-init: Installed: 26.1-0ubuntu2 3) What I expected to happen: Cloud-init should successfully detect the Ec2 datasource and configure networking. Specifically, cloud-init- local.service should have a dependency on network hardware enumeration (udev) to ensure that the primary network interface (ena) is visible before attempting to identify the cloud platform. 4) What happened instead: On modern, fast-booting hardware (specifically Graviton 4 / m8g instances), or a minimalist OS image, or in scenarios where the ena driver is slightly delayed, cloud-init-local executes before the ENA driver has finished probing the PCI bus. Cloud-init sees only the 'lo' interface, fails to identify the Ec2 datasource, and falls back to DataSourceNone. This results in the instance booting without network connectivity, causing systemd-networkd-wait-online to time out. The cloud-init log explicitly shows the failure: "DataSourceEc2.py[ERROR]: The instance must have at least one eligible NIC" The systemd unit file for cloud-init-local.service in this release is missing any hardware or udev dependencies: [Unit] Description=Cloud- init: Local Stage (pre-network) DefaultDependencies=no Wants=network- pre.target After=hv_kvp_daemon.service Before=network-pre.target Before=sysinit.target STEPS TO REPRODUCE: The issue can be deterministically reproduced by artificially delaying the ena driver initialization to simulate a fast OS boot outrunning the hardware probe: Launch a stock Ubuntu 26.04 m8g.medium instance. Add a modprobe delay for the ena driver: echo "install ena /usr/bin/sleep 10; /sbin/modprobe --ignore-install ena" > /etc/modprobe.d/ena-delay.conf Clean cloud-init state and reboot: cloud-init clean --logs && reboot On the subsequent boot, cloud-init will fail to detect the EC2 datasource because the ENA driver will still be in its 10-second sleep when cloud-init-local executes. Thanks! ** Tags added: ec2 graviton resolute stonking ** Description changed: [ Impact ] On fast-booting AWS instances (e.g. Graviton 4 m8g/r8g), cloud-init-local starts before the ENA driver finishes probing PCI. DataSourceEc2 finds no interfaces, logs "The instance must have at least one eligible NIC", and falls back to DataSourceNone without writing netplan configuration. Instances boot without networking and SSH access. [ Fix ] - Add wait_for_candidate_nics() to poll /sys/class/net in DataSourceEc2 when no interfaces are present at boot. - If interfaces are already present, it returns immediately with 0 delay. + PR #7065 added wait_for_candidate_nics() to DataSourceEc2, but restricted polling to hpc7a.96xlarge. + The fix expands polling to include Graviton instance families (m8g, r8g, c8g, etc.) and makes wait_for_nic_timeout configurable via datasource/Ec2/wait_for_nic_timeout. [ Test Plan ] 1. Delay ENA probe in modprobe: echo "install ena /usr/bin/sleep 2; /sbin/modprobe --ignore-install ena" > /etc/modprobe.d/ena-delay.conf Run: cloud-init clean --logs && cloud-init init --local Without fix: DataSourceEc2 exits False. With fix: DataSourceEc2 waits for the interface and configures networking. 2. Run unit tests: - pytest tests/unittests/net/test_init.py tests/unittests/sources/test_ec2.py + pytest tests/unittests/sources/test_ec2.py [ Where problems could occur ] - If an instance has no network interfaces attached, cloud-init-local will wait up to 10s before bailing out. The timeout is configurable via datasource/Ec2/wait_for_nic_timeout. + If an instance has no network interfaces attached, cloud-init-local will wait up to the timeout before bailing out. The timeout is configurable via datasource/Ec2/wait_for_nic_timeout. [ Other Info ] - - stonking debdiff: cloud-init 26.2-0ubuntu2 - - resolute debdiff: cloud-init 26.2-0ubuntu1~26.04.2 - Upstream PR: https://github.com/canonical/cloud-init/pull/7138 + - Upstream issue: GH-6697 + - Verification: 0 regressions on Ubuntu 26.04 and 26.10 test suites. --- [ Original Report ] 1) Ubuntu Release: Description: Ubuntu 26.04 LTS Release: 26.04 2) Package Version: cloud-init: Installed: 26.1-0ubuntu2 3) What I expected to happen: Cloud-init should successfully detect the Ec2 datasource and configure networking. Specifically, cloud-init- local.service should have a dependency on network hardware enumeration (udev) to ensure that the primary network interface (ena) is visible before attempting to identify the cloud platform. 4) What happened instead: On modern, fast-booting hardware (specifically Graviton 4 / m8g instances), or a minimalist OS image, or in scenarios where the ena driver is slightly delayed, cloud-init-local executes before the ENA driver has finished probing the PCI bus. Cloud-init sees only the 'lo' interface, fails to identify the Ec2 datasource, and falls back to DataSourceNone. This results in the instance booting without network connectivity, causing systemd-networkd-wait-online to time out. The cloud-init log explicitly shows the failure: "DataSourceEc2.py[ERROR]: The instance must have at least one eligible NIC" The systemd unit file for cloud-init-local.service in this release is missing any hardware or udev dependencies: [Unit] Description=Cloud- init: Local Stage (pre-network) DefaultDependencies=no Wants=network- pre.target After=hv_kvp_daemon.service Before=network-pre.target Before=sysinit.target STEPS TO REPRODUCE: The issue can be deterministically reproduced by artificially delaying the ena driver initialization to simulate a fast OS boot outrunning the hardware probe: Launch a stock Ubuntu 26.04 m8g.medium instance. Add a modprobe delay for the ena driver: echo "install ena /usr/bin/sleep 10; /sbin/modprobe --ignore-install ena" > /etc/modprobe.d/ena-delay.conf Clean cloud-init state and reboot: cloud-init clean --logs && reboot On the subsequent boot, cloud-init will fail to detect the EC2 datasource because the ENA driver will still be in its 10-second sleep when cloud-init-local executes. Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2151279 Title: cloud-init-local executes before ENA network driver completes PCI probe, causing DataSourceEc2 failure To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2151279/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
