[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
** Also affects: cloud-init (Ubuntu) Importance: Undecided Status: New ** No longer affects: linux-azure (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in cloud-init package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
https://github.com/canonical/cloud-init/pull/539 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
Cloud-init logs on an instance with accelerated networking (bionic) exhibiting this delay across reboot ** Attachment added: "cloud-init.tar" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+attachment/5404235/+files/cloud-init.tar -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
This looks hypervisor/kernel related. Some observations: The cloud-init.log in the collect-logs shows cloud-init running twice. The first time, run-time is expected, approx 17s of cloud-init time, the second boot took much longer, but the bulk if the time is in udev 2019-06-20 18:09:18,951 - util.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True) 2019-06-20 18:10:48,209 - util.py[DEBUG]: Waiting for udev events to settle took 89.258 seconds Some kernel logs around this time show some strangeness: Jun 20 18:09:18.252780 cnewcomer-1561053730 systemd[1]: Reached target System Time Synchronized. Jun 20 18:10:05.561513 cnewcomer-1561053730 kernel: hv_balloon: Max. dynamic memory size: 262144 MB Jun 20 18:10:17.907484 cnewcomer-1561053730 systemd-udevd[770]: seq 9849 '/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/c204def2-185f-44d7-9540-5836f397c2de/pci0001:00/0001:00:02.0/net/eth1' is taking a long time Jun 20 18:10:48.179475 cnewcomer-1561053730 systemd-udevd[868]: error changing net interface name 'eth1' to 'eth0': File exists Jun 20 18:10:48.179510 cnewcomer-1561053730 systemd-udevd[868]: could not rename interface '3' from 'eth1' to 'eth0': File exists Jun 20 18:10:48.240895 cnewcomer-1561053730 kernel: mlx4_en: rename3: Steering Mode 2 Jun 20 18:10:48.336884 cnewcomer-1561053730 kernel: hv_netvsc 000d3a7a-59f0-000d-3a7a-59fd3a7a eth0: Data path switched to VF: rename3 Jun 20 18:10:48.337179 cnewcomer-1561053730 kernel: mlx4_en: rename3: Link Up -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
** Attachment added: "cloud-init collect-logs" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+attachment/5271884/+files/cloud-init.tar.gz -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
** Attachment added: "cloud-init analyze blame" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+attachment/5271879/+files/ci-analyze-blame.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
I created a test instance and was able to produce most of the data requested. Attachments incoming. ** Attachment added: "cloud-init analyze show" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+attachment/5271877/+files/ci-analyze-show.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
** Attachment added: "cloud-init analyze dump" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+attachment/5271878/+files/ci-analyze-dump.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
This sosreport contains the systemd analyze blame output. It is from a test instance that I was able to reproduce this issue on. ** Attachment added: "sosreport-cnewcomer-1559258602-20190530234322.tar.xz" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+attachment/5271880/+files/sosreport-cnewcomer-1559258602-20190530234322.tar.xz -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
Joe, can you also attach the gz file generated by `cloud-init collect- logs` and the output from `cloud-init analyze show`? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
Hi, Joe. Do you think you can provide the output of "systemd-analyse blame" on this system? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1830740] Re: [linux-azure] Delay during boot of some instance sizes
We will be investigation this situation ASAP but it is pending submission for SRU for other azure work. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1830740 Title: [linux-azure] Delay during boot of some instance sizes Status in linux-azure package in Ubuntu: New Bug description: We are seeing a long delay, about 100s, when booting new E32s_v3 VMs. We don’t see this delay with smaller sized Ubuntu VMs(D2s and NC6 sizes). I attached a screen shot of the boot delay. You can see between 7.022911 and 101.325958 seconds there is no activity. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
