[Kernel-packages] [Bug 2063315] Re: Suspend & Resume functionality broken/timesout in GCE
** Also affects: linux-gcp (Ubuntu Noble) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-gcp in Ubuntu. https://bugs.launchpad.net/bugs/2063315 Title: Suspend & Resume functionality broken/timesout in GCE Status in Release Notes for Ubuntu: New Status in linux-gcp package in Ubuntu: New Status in linux-gcp source package in Noble: New Bug description: [Impact] Suspend/Resume capability is broken in all noble images with kernel version 6.8.0-1007-gcp. GCE offers the capability to "Suspend" a VM to conserve power/lower costs when the instance is not in use [0]. It uses ACPI S3 signals to tell the guest to power down. This capability no longer works in the latest kernel with the following error: ``` Operation type [suspend] failed with message "Instance suspend failed due to guest timeout." ``` which points to the following [1]. Refs: [0]: https://cloud.google.com/compute/docs/instances/suspend-resume- instance [1]: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting- suspend-resume#there_was_a_guest_timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2063315/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2061851] Re: linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression with new apparmor profiles/features
** Changed in: snapd (Ubuntu Noble) Status: New => Invalid ** Changed in: linux-aws (Ubuntu Noble) Status: New => Fix Released ** Changed in: linux-azure (Ubuntu Noble) Status: New => Fix Released ** Changed in: linux-gcp (Ubuntu Noble) Status: New => Fix Released ** Changed in: linux-ibm (Ubuntu Noble) Status: New => Fix Released ** Changed in: linux-oracle (Ubuntu Noble) Status: New => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2061851 Title: linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression with new apparmor profiles/features Status in chrony package in Ubuntu: Invalid Status in linux package in Ubuntu: Fix Released Status in linux-aws package in Ubuntu: Fix Released Status in linux-azure package in Ubuntu: Fix Released Status in linux-gcp package in Ubuntu: Fix Released Status in linux-ibm package in Ubuntu: Fix Released Status in linux-oracle package in Ubuntu: Fix Released Status in snapd package in Ubuntu: Invalid Status in chrony source package in Noble: Invalid Status in linux source package in Noble: Fix Released Status in linux-aws source package in Noble: Fix Released Status in linux-azure source package in Noble: Fix Released Status in linux-gcp source package in Noble: Fix Released Status in linux-ibm source package in Noble: Fix Released Status in linux-oracle source package in Noble: Fix Released Status in snapd source package in Noble: Invalid Bug description: * Canonical Public Cloud discovered that `chronyc -c sources` now fails with `506 Cannot talk to daemon` with the latest kernels. We are seeing this in linux-azure and linux-gcp kernels (6.8.0-1005.5) * Disabling AppArmor (`sudo systemctl stop apparmor`) completely results in no regression and `chronyc -c sources` returns as expected * Disabling the apparmor profile for `chronyd` only results in no regression and `chronyc -c sources` returns as expected * There are zero entries in dmesg when this occurs * There are zero entries in dmesg when this occurs if the apparmor profile for `chronyd` is placed in complain mode instead of enforce mode * We changed the time server from the internal GCP metadata.google.internal to the ubuntu time server ntp.ubuntu.com with no change in behaviour We also noted issues with DNS resolution in snaps like `google-cloud-cli` in GCE images. * Disabling apparmor completely for snaps too (`sudo systemctl stop snapd.apparmor`) results in no regression and calling the snaps returns as expected. The same issues are present in azure kernel `linux-azure` `6.8.0-1005.5` and the -proposed `6.8.0-25.25` generic kernel. This is a release blocker for Noble release To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/chrony/+bug/2061851/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2061851] Re: linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression iwth new apparmor profiles/features
** Also affects: linux-gcp (Ubuntu) Importance: Undecided Status: New ** Also affects: linux-aws (Ubuntu) Importance: Undecided Status: New ** Also affects: linux-azure (Ubuntu) Importance: Undecided Status: New ** Also affects: linux-oracle (Ubuntu) Importance: Undecided Status: New ** Also affects: linux-ibm (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/2061851 Title: linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression iwth new apparmor profiles/features Status in chrony package in Ubuntu: New Status in linux package in Ubuntu: New Status in linux-aws package in Ubuntu: New Status in linux-azure package in Ubuntu: New Status in linux-gcp package in Ubuntu: New Status in linux-ibm package in Ubuntu: New Status in linux-oracle package in Ubuntu: New Status in snapd package in Ubuntu: New Status in chrony source package in Noble: New Status in linux source package in Noble: New Status in linux-aws source package in Noble: New Status in linux-azure source package in Noble: New Status in linux-gcp source package in Noble: New Status in linux-ibm source package in Noble: New Status in linux-oracle source package in Noble: New Status in snapd source package in Noble: New Bug description: * Canonical Public Cloud discovered that `chronyc -c sources` now fails with `506 Cannot talk to daemon` with the latest kernels. We are seeing this in linux-azure and linux-gcp kernels (6.8.0-1005.5) * Disabling AppArmor (`sudo systemctl stop apparmor`) completely results in no regression and `chronyc -c sources` returns as expected * Disabling the apparmor profile for `chronyd` only results in no regression and `chronyc -c sources` returns as expected * There are zero entries in dmesg when this occurs * There are zero entries in dmesg when this occurs if the apparmor profile for `chronyd` is placed in complain mode instead of enforce mode * We changed the time server from the internal GCP metadata.google.internal to the ubuntu time server ntp.ubuntu.com with no change in behaviour We also noted issues with DNS resolution in snaps like `google-cloud-cli` in GCE images. * Disabling apparmor completely for snaps too (`sudo systemctl stop snapd.apparmor`) results in no regression and calling the snaps returns as expected. The same issues are present in azure kernel `linux-azure` `6.8.0-1005.5` and the -proposed `6.8.0-25.25` generic kernel. This is a release blocker for Noble release To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/chrony/+bug/2061851/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2061851] Re: linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression iwth new apparmor profiles/features
** Also affects: chrony (Ubuntu) Importance: Undecided Status: New ** Also affects: snapd (Ubuntu) Importance: Undecided Status: New ** Also affects: chrony (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: snapd (Ubuntu Noble) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2061851 Title: linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression iwth new apparmor profiles/features Status in chrony package in Ubuntu: New Status in linux package in Ubuntu: New Status in snapd package in Ubuntu: New Status in chrony source package in Noble: New Status in linux source package in Noble: New Status in snapd source package in Noble: New Bug description: * Canonical Public Cloud discovered that `chronyc -c sources` now fails with `506 Cannot talk to daemon` with the latest kernels. We are seeing this in linux-azure and linux-gcp kernels (6.8.0-1005.5) * Disabling AppArmor (`sudo systemctl stop apparmor`) completely results in no regression and `chronyc -c sources` returns as expected * Disabling the apparmor profile for `chronyd` only results in no regression and `chronyc -c sources` returns as expected * There are zero entries in dmesg when this occurs * There are zero entries in dmesg when this occurs if the apparmor profile for `chronyd` is placed in complain mode instead of enforce mode * We changed the time server from the internal GCP metadata.google.internal to the ubuntu time server ntp.ubuntu.com with no change in behaviour We also noted issues with DNS resolution in snaps like `google-cloud-cli` in GCE images. * Disabling apparmor completely for snaps too (`sudo systemctl stop snapd.apparmor`) results in no regression and calling the snaps returns as expected. The same issues are present in azure kernel `linux-azure` `6.8.0-1005.5` and the -proposed `6.8.0-25.25` generic kernel. This is a release blocker for Noble release To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/chrony/+bug/2061851/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2061851] [NEW] linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression iwth new apparmor profiles/features
Public bug reported: * Canonical Public Cloud discovered that `chronyc -c sources` now fails with `506 Cannot talk to daemon` with the latest kernels. We are seeing this in linux-azure and linux-gcp kernels (6.8.0-1005.5) * Disabling AppArmor (`sudo systemctl stop apparmor`) completely results in no regression and `chronyc -c sources` returns as expected * Disabling the apparmor profile for `chronyd` only results in no regression and `chronyc -c sources` returns as expected * There are zero entries in dmesg when this occurs * There are zero entries in dmesg when this occurs if the apparmor profile for `chronyd` is placed in complain mode instead of enforce mode * We changed the time server from the internal GCP metadata.google.internal to the ubuntu time server ntp.ubuntu.com with no change in behaviour We also noted issues with DNS resolution in snaps like `google-cloud-cli` in GCE images. * Disabling apparmor completely for snaps too (`sudo systemctl stop snapd.apparmor`) results in no regression and calling the snaps returns as expected. The same issues are present in azure kernel `linux-azure` `6.8.0-1005.5` and the -proposed `6.8.0-25.25` generic kernel. This is a release blocker for Noble release ** Affects: chrony (Ubuntu) Importance: Undecided Status: New ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Affects: snapd (Ubuntu) Importance: Undecided Status: New ** Affects: chrony (Ubuntu Noble) Importance: Undecided Status: New ** Affects: linux (Ubuntu Noble) Importance: Undecided Status: New ** Affects: snapd (Ubuntu Noble) Importance: Undecided Status: New ** Tags: block-proposed block-proposed-noble -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2061851 Title: linux-gcp 6.8.0-1005.5 (+ others) Noble kernel regression iwth new apparmor profiles/features Status in chrony package in Ubuntu: New Status in linux package in Ubuntu: New Status in snapd package in Ubuntu: New Status in chrony source package in Noble: New Status in linux source package in Noble: New Status in snapd source package in Noble: New Bug description: * Canonical Public Cloud discovered that `chronyc -c sources` now fails with `506 Cannot talk to daemon` with the latest kernels. We are seeing this in linux-azure and linux-gcp kernels (6.8.0-1005.5) * Disabling AppArmor (`sudo systemctl stop apparmor`) completely results in no regression and `chronyc -c sources` returns as expected * Disabling the apparmor profile for `chronyd` only results in no regression and `chronyc -c sources` returns as expected * There are zero entries in dmesg when this occurs * There are zero entries in dmesg when this occurs if the apparmor profile for `chronyd` is placed in complain mode instead of enforce mode * We changed the time server from the internal GCP metadata.google.internal to the ubuntu time server ntp.ubuntu.com with no change in behaviour We also noted issues with DNS resolution in snaps like `google-cloud-cli` in GCE images. * Disabling apparmor completely for snaps too (`sudo systemctl stop snapd.apparmor`) results in no regression and calling the snaps returns as expected. The same issues are present in azure kernel `linux-azure` `6.8.0-1005.5` and the -proposed `6.8.0-25.25` generic kernel. This is a release blocker for Noble release To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/chrony/+bug/2061851/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2042564] Re: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel
I have reproduced with @amalmostafa's updated script with a separate disk too. I see no segfault and no EXT4 errors but the regression in performance is still present but not as great as in my previous tests. ``` ### Ubuntu 20.04 with 5.4 kernel and data disk ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdb fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][w=1162MiB/s][w=298k IOPS][eta 00m:00s] fiojob1: (groupid=0, jobs=8): err= 0: pid=2391: Tue Nov 21 10:34:57 2023 write: IOPS=284k, BW=1108MiB/s (1162MB/s)(320GiB/295713msec); 0 zone resets slat (nsec): min=751, max=115304k, avg=8630.05, stdev=124263.77 clat (nsec): min=391, max=239001k, avg=3598764.23, stdev=2429948.87 lat (usec): min=72, max=239002, avg=3607.70, stdev=2428.75 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 1434], 10.00th=[ 1778], 20.00th=[ 2212], | 30.00th=[ 2573], 40.00th=[ 2900], 50.00th=[ 3261], 60.00th=[ 3654], | 70.00th=[ 4080], 80.00th=[ 4686], 90.00th=[ 5669], 95.00th=[ 6587], | 99.00th=[ 9110], 99.50th=[10945], 99.90th=[26608], 99.95th=[43779], | 99.99th=[83362] bw ( MiB/s): min= 667, max= 1341, per=99.98%, avg=1107.88, stdev=13.07, samples=4728 iops: min=170934, max=343430, avg=283618.08, stdev=3346.84, samples=4728 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.04%, 500=0.43%, 750=0.66%, 1000=0.51% lat (msec) : 2=13.23%, 4=53.26%, 10=31.18%, 20=0.53%, 50=0.11% lat (msec) : 100=0.03%, 250=0.01% cpu : usr=3.10%, sys=7.36%, ctx=1105263, majf=0, minf=102 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,83886080,0,8 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=1108MiB/s (1162MB/s), 1108MiB/s-1108MiB/s (1162MB/s-1162MB/s), io=320GiB (344GB), run=295713-295713msec Disk stats (read/write): sdb: ios=96/33256749, merge=0/50606838, ticks=19/30836895, in_queue=971080, util=100.00% ubuntu@cloudimg:~$ ubuntu@cloudimg:~$ uname --kernel-release 5.4.0-164-generic ubuntu@cloudimg:~$ cat /etc/cloud/build.info build_name: server serial: 20231011 ubuntu@cloudimg:~$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; VERSION_CODENAME=focal UBUNTU_CODENAME=focal ### Ubuntu 20.04 with 5.15 kernel and data disk ubuntu@cloudimg:~$ uname --kernel-release 5.15.0-89-generic ubuntu@cloudimg:~$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; VERSION_CODENAME=focal UBUNTU_CODENAME=focal ubuntu@cloudimg:~$ cat /etc/cloud/build.info build_name: server serial: 20231011 ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdb fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][w=1071MiB/s][w=274k IOPS][eta 00m:00s] fiojob1: (groupid=0, jobs=8): err= 0: pid=1008: Tue Nov 21 12:19:56 2023 write: IOPS=258k, BW=1007MiB/s (1056MB/s)(320GiB/325284msec); 0 zone resets slat (nsec): min=931, max=36726k, avg=7936.15, stdev=120427.45 clat (nsec): min=1963, max=155870k, avg=3959799.65, stdev=2129472.51 lat (usec): min=55, max=155872, avg=3968.10, stdev=2128.87 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 1319], 10.00th=[ 1811], 20.00th=[ 2376], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3720], 60.00th=[ 4113], | 70.00th=[ 4621], 80.00th=[ 5211], 90.00th=[ 6390], 95.00th=[ 7439], | 99.00th=[10159], 99.50th=[11863], 99.90th=[19268], 99.95th=[23462], | 99.99th=[36439] bw ( KiB/s): min=715896, max=1190887, per=100.00%, avg=1031613.99, stdev=9298.50, samples=5200 iops:
[Kernel-packages] [Bug 2042564] Re: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel
I am NOT seeing the same on 22.04 ``` ubuntu@cloudimg:~$ uname --kernel-release 5.15.0-87-generic ubuntu@cloudimg:~$ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; UBUNTU_CODENAME=jammy ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.28 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][w=1180MiB/s][w=302k IOPS][eta 00m:00s] fiojob1: (groupid=0, jobs=8): err= 0: pid=2166: Thu Nov 9 07:30:05 2023 write: IOPS=315k, BW=1230MiB/s (1290MB/s)(320GiB/266343msec); 0 zone resets slat (nsec): min=662, max=37599k, avg=4806.87, stdev=81368.70 clat (nsec): min=656, max=87622k, avg=3241604.28, stdev=1888945.13 lat (usec): min=46, max=87623, avg=3246.74, stdev=1888.87 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 1139], 10.00th=[ 1483], 20.00th=[ 1893], | 30.00th=[ 2245], 40.00th=[ 2573], 50.00th=[ 2933], 60.00th=[ 3294], | 70.00th=[ 3752], 80.00th=[ 4359], 90.00th=[ 5276], 95.00th=[ 6194], | 99.00th=[ 9241], 99.50th=[11207], 99.90th=[19530], 99.95th=[24511], | 99.99th=[36439] bw ( MiB/s): min= 512, max= 1567, per=100.00%, avg=1232.42, stdev=21.42, samples=4248 iops: min=131165, max=401400, avg=315500.53, stdev=5483.45, samples=4248 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.02%, 500=1.41%, 750=0.91%, 1000=1.47% lat (msec) : 2=19.05%, 4=51.85%, 10=24.55%, 20=0.66%, 50=0.09% lat (msec) : 100=0.01% cpu : usr=3.39%, sys=7.68%, ctx=968250, majf=0, minf=108 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,83886080,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=1230MiB/s (1290MB/s), 1230MiB/s-1230MiB/s (1290MB/s-1290MB/s), io=320GiB (344GB), run=266343-266343msec Disk stats (read/write): sda: ios=230/34934643, merge=0/48920965, ticks=54/27582950, in_queue=27583025, util=100.00% ``` ... but I do see the following in journalctl ``` Nov 09 07:33:03 cloudimg kernel: EXT4-fs warning (device sda1): ext4_dirblock_csum_verify:404: inode #80703: comm journalctl: No space for directory leaf checksum. Please run e2fsck -D. Nov 09 07:33:03 cloudimg kernel: EXT4-fs error (device sda1): __ext4_find_entry:1689: inode #80703: comm journalctl: checksumming directory block 0 Nov 09 07:33:03 cloudimg kernel: Aborting journal on device sda1-8. Nov 09 07:33:03 cloudimg kernel: EXT4-fs error (device sda1): ext4_journal_check_start:83: comm rs:main Q:Reg: Detected aborted journal Nov 09 07:33:03 cloudimg kernel: EXT4-fs error (device sda1): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal Nov 09 07:33:03 cloudimg kernel: EXT4-fs (sda1): Remounting filesystem read-only Nov 09 07:33:03 cloudimg kernel: EXT4-fs warning (device sda1): ext4_dirblock_csum_verify:404: inode #80703: comm journalctl: No space for directory leaf checksum. Please run e2fsck -D. Nov 09 07:33:03 cloudimg kernel: EXT4-fs error (device sda1): __ext4_find_entry:1689: inode #80703: comm journalctl: checksumming directory block 0 Nov 09 07:33:03 cloudimg kernel: EXT4-fs warning (device sda1): ext4_dirblock_csum_verify:404: inode #80703: comm journalctl: No space for directory leaf checksum. Please run e2fsck -D. Nov 09 07:33:03 cloudimg kernel: EXT4-fs error (device sda1): __ext4_find_entry:1689: inode #80703: comm journalctl: checksumming directory block 0 Nov 09 07:33:03 cloudimg kernel: EXT4-fs warning (device sda1): ext4_dirblock_csum_verify:404: inode #80703: comm journalctl: No space for directory leaf checksum. Please run e2fsck -D. Nov 09 07:33:03 cloudimg kernel: EXT4-fs error (device sda1): __ext4_find_entry:1689: inode #80703: comm journalctl: checksumming directory block 0 Nov 09 07:33:03 cloudimg kernel: EXT4-fs warning (device sda1): ext4_dirblock_csum_verify:404: inode #80703: comm journalctl: No space for directory leaf checksum. Please run e2fsck -D. Nov 09 07:33:03 cloudimg kernel: EXT4-fs error (device sda1): __ext4_find_entry:1689: inode #80703: comm journalctl: checksumming directory block 0 Nov 09 07:33:03 cloudimg kernel:
[Kernel-packages] [Bug 2042564] Re: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel
@kamalmostafa indeed yes. I had missed this. See below for the output from 20.04 with 5.15 kernel. ``` ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 4 (f=4): [W(1),_(2),W(1),_(1),W(2),_(1)][99.6%][w=212MiB/s][w=54.3k IOPS][eta 00m:01s] fiojob1: (groupid=0, jobs=8): err= 0: pid=697: Thu Nov 9 06:30:01 2023 write: IOPS=319k, BW=1247MiB/s (1307MB/s)(320GiB/262803msec); 0 zone resets slat (nsec): min=652, max=55305k, avg=4699.55, stdev=76242.73 clat (usec): min=5, max=121658, avg=3187.92, stdev=1886.86 lat (usec): min=34, max=121660, avg=3192.96, stdev=1886.68 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 1254], 10.00th=[ 1565], 20.00th=[ 1909], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3228], | 70.00th=[ 3687], 80.00th=[ 4228], 90.00th=[ 5145], 95.00th=[ 5997], | 99.00th=[ 8717], 99.50th=[10683], 99.90th=[18744], 99.95th=[25822], | 99.99th=[55837] bw ( MiB/s): min= 353, max= 1690, per=100.00%, avg=1251.19, stdev=22.89, samples=4182 iops: min=90615, max=432642, avg=320303.28, stdev=5860.14, samples=4182 lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02% lat (usec) : 500=0.95%, 750=0.55%, 1000=1.01% lat (msec) : 2=20.21%, 4=53.10%, 10=23.55%, 20=0.52%, 50=0.07% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=3.42%, sys=7.71%, ctx=960980, majf=0, minf=86 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,83886080,0,8 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=1247MiB/s (1307MB/s), 1247MiB/s-1247MiB/s (1307MB/s-1307MB/s), io=320GiB (344GB), run=262803-262803msec Disk stats (read/write): sda: ios=251/31989123, merge=0/51876731, ticks=482/27581400, in_queue=27581891, util=100.00% Segmentation fault ubuntu@cloudimg:~$ uname --kernel-release 5.15.0-88-generic ubuntu@cloudimg:~$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; VERSION_CODENAME=focal UBUNTU_CODENAME=focal ubuntu@cloudimg:~$ ``` I also see the same details in `journalctl` ``` Nov 09 06:25:38 cloudimg sudo[692]: ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda Nov 09 06:25:38 cloudimg sudo[692]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0) Nov 09 06:30:01 cloudimg kernel: fio[693]: segfault at 7f62e5d78595 ip 7f62fa672a50 sp 7ffeaecfd2a8 error 6 in libglusterfs.so.0.0.1[7f62fa5b+c3000] Nov 09 06:30:01 cloudimg kernel: Code: f0 0e d0 dd 14 e5 db 95 d9 14 73 00 00 00 00 00 00 00 c2 11 ee 01 00 00 00 00 00 3a 21 70 00 00 00 00 4d f9 e0 18 e7 be 9b 09 <29> 9f 75 66 6c eb 9a 03 e5 33 3a bd 32 2a 9c 0e 7c c6 b4 3d a9 26 Nov 09 06:30:01 cloudimg kernel: Core dump to |/usr/share/apport/apport pipe failed Nov 09 06:30:01 cloudimg sudo[692]: pam_unix(sudo:session): session closed for user root Nov 09 06:30:01 cloudimg kernel: Core dump to |/usr/share/apport/apport pipe failed Nov 09 06:39:19 cloudimg systemd[1]: Starting Cleanup of Temporary Directories... Nov 09 06:39:19 cloudimg kernel: EXT4-fs warning (device sda1): ext4_dirblock_csum_verify:404: inode #108: comm systemd-tmpfile: No space for directory leaf checksum. Please run e2fsck -D. Nov 09 06:39:19 cloudimg kernel: EXT4-fs error (device sda1): htree_dirblock_to_tree:1080: inode #108: comm systemd-tmpfile: Directory block failed checksum Nov 09 06:39:19 cloudimg systemd[1]: systemd-tmpfiles-clean.service: Succeeded. ``` -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2042564 Title: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel Status in linux package in Ubuntu: New Status in linux source package in Focal: New Bug description: We in the Canonical Public Cloud team have received report from our colleagues in Google regarding a potential performance
[Kernel-packages] [Bug 2042564] Re: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel
Providing exact reproducer steps using Qemu locally - Launch script: https://gist.github.com/philroche/8242106415ef35b446d7e625b6d60c90 and cloud image I used for testing @ http://cloud- images.ubuntu.com/minimal/releases/focal/release/ ``` # Download the VM launch script wget --output-document=launch-qcow2-image-qemu-40G.sh https://gist.githubusercontent.com/philroche/8242106415ef35b446d7e625b6d60c90/raw/4a338b92301b6e08608e9345f85a50452ca5fa21/launch-qcow2-image-qemu-40G.sh chmod +x launch-qcow2-image-qemu-40G.sh # Download latest image wget --output-document=ubuntu-20.04-minimal-cloudimg-amd64.img http://cloud-images.ubuntu.com/minimal/releases/focal/release/ubuntu-20.04-minimal-cloudimg-amd64.img # launch the image - note that this provisions a 40GB disk. ./launch-qcow2-image-qemu-40G.sh --password passw0rd --image ./20231102-ubuntu-20.04-minimal-cloudimg-amd64.img # install fio sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo apt-get update sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo apt- get install --assume-yes fio # run the synthetic test and gather information sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda # kill VM (Use 'Ctrl-a x' key combination to exit emulator ) # start new VM to upgrade kernel and run the synthetic test again # launch the image - note that this provisions a 40GB disk. ./launch-qcow2-image-qemu-40G.sh --password passw0rd --image ./20231102-ubuntu-20.04-minimal-cloudimg-amd64.img # install fio sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo apt-get update sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo apt- get install --assume-yes fio # upgrade the kernel sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo apt-get install --assume-yes linux-generic-hwe-20.04 # wait until VM has rebooted sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo reboot # run the synthetic test again and gather the information sshpass -p passw0rd ssh -p -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no ubuntu@127.0.0.1 -- sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda ``` -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2042564 Title: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel Status in linux package in Ubuntu: New Status in linux source package in Focal: New Bug description: We in the Canonical Public Cloud team have received report from our colleagues in Google regarding a potential performance regression with the 5.15 kernel vs the 5.4 kernel on ubuntu 20.04. Their test were performed using the linux-gkeop and linux-gkeop-5.15 kernels. I have verified with the generic Ubuntu 20.04 5.4 linux-generic and the Ubuntu 20.04 5.15 linux-generic-hwe-20.04 kernels. The tests were run using `fio` fio commands: * 4k initwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` * 4k overwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` My reproducer was to launch an Ubuntu 20.04 cloud image locally with qemu the results are below: Using 5.4 kernel ``` ubuntu@cloudimg:~$ uname --kernel-release 5.4.0-164-generic ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 8 (f=8): [W(8)][99.6%][w=925MiB/s][w=237k IOPS][eta 00m:01s] fiojob1: (groupid=0, jobs=8): err= 0: pid=2443: Thu Nov 2 09:15:22 2023 write: IOPS=317k, BW=1237MiB/s (1297MB/s)(320GiB/264837msec); 0 zone resets
[Kernel-packages] [Bug 2042564] Re: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel
Google have provided a non synthetic `fio` impact > Performance was severally degraded when accessing Persistent Volumes provided by Portworx/PureStorage. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2042564 Title: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel Status in linux package in Ubuntu: New Status in linux source package in Focal: New Bug description: We in the Canonical Public Cloud team have received report from our colleagues in Google regarding a potential performance regression with the 5.15 kernel vs the 5.4 kernel on ubuntu 20.04. Their test were performed using the linux-gkeop and linux-gkeop-5.15 kernels. I have verified with the generic Ubuntu 20.04 5.4 linux-generic and the Ubuntu 20.04 5.15 linux-generic-hwe-20.04 kernels. The tests were run using `fio` fio commands: * 4k initwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` * 4k overwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` My reproducer was to launch an Ubuntu 20.04 cloud image locally with qemu the results are below: Using 5.4 kernel ``` ubuntu@cloudimg:~$ uname --kernel-release 5.4.0-164-generic ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 8 (f=8): [W(8)][99.6%][w=925MiB/s][w=237k IOPS][eta 00m:01s] fiojob1: (groupid=0, jobs=8): err= 0: pid=2443: Thu Nov 2 09:15:22 2023 write: IOPS=317k, BW=1237MiB/s (1297MB/s)(320GiB/264837msec); 0 zone resets slat (nsec): min=628, max=37820k, avg=7207.71, stdev=101058.61 clat (nsec): min=457, max=56099k, avg=340.45, stdev=1707823.38 lat (usec): min=23, max=56100, avg=3229.78, stdev=1705.80 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 1352], 10.00th=[ 1647], 20.00th=[ 2024], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 2933], 60.00th=[ 3261], | 70.00th=[ 3654], 80.00th=[ 4146], 90.00th=[ 5014], 95.00th=[ 5932], | 99.00th=[ 8979], 99.50th=[10945], 99.90th=[18220], 99.95th=[22676], | 99.99th=[32113] bw ( MiB/s): min= 524, max= 1665, per=100.00%, avg=1237.72, stdev=20.42, samples=4232 iops: min=134308, max=426326, avg=316855.16, stdev=5227.36, samples=4232 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.05%, 500=0.54%, 750=0.37%, 1000=0.93% lat (msec) : 2=17.40%, 4=58.02%, 10=22.01%, 20=0.60%, 50=0.07% lat (msec) : 100=0.01% cpu : usr=3.29%, sys=7.45%, ctx=1262621, majf=0, minf=103 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,83886080,0,8 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=1237MiB/s (1297MB/s), 1237MiB/s-1237MiB/s (1297MB/s-1297MB/s), io=320GiB (344GB), run=264837-264837msec Disk stats (read/write): sda: ios=36/32868891, merge=0/50979424, ticks=5/27498602, in_queue=1183124, util=100.00% ``` After upgrading to linux-generic-hwe-20.04 kernel and rebooting ``` ubuntu@cloudimg:~$ uname --kernel-release 5.15.0-88-generic ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 1 (f=1): [_(7),W(1)][100.0%][w=410MiB/s][w=105k IOPS][eta 00m:00s] fiojob1: (groupid=0, jobs=8): err= 0: pid=1438: Thu Nov 2 09:46:49 2023 write: IOPS=155k, BW=605MiB/s (634MB/s)(320GiB/541949msec); 0 zone resets slat (nsec): min=660, max=325426k, avg=10351.04, stdev=232438.50 clat (nsec): min=1100, max=782743k, avg=6595008.67, stdev=6290570.04 lat (usec): min=86, max=782748, avg=6606.08, stdev=6294.03 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 2180], 10.00th=[ 2802], 20.00th=[ 3556], | 30.00th=[
[Kernel-packages] [Bug 2042564] Re: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel
For reference, I also tried on a 22.04 cloud image with 5.15 kernel ``` ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.28 Starting 8 processes Jobs: 1 (f=1): [_(5),W(1),_(2)][100.0%][w=311MiB/s][w=79.7k IOPS][eta 00m:00s] fiojob1: (groupid=0, jobs=8): err= 0: pid=2198: Thu Nov 2 13:42:07 2023 write: IOPS=244k, BW=952MiB/s (998MB/s)(320GiB/344252msec); 0 zone resets slat (nsec): min=655, max=86425k, avg=6303.33, stdev=121079.62 clat (usec): min=5, max=124812, avg=4182.79, stdev=3154.25 lat (usec): min=61, max=124814, avg=4189.52, stdev=3155.57 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 1319], 10.00th=[ 1713], 20.00th=[ 2180], | 30.00th=[ 2638], 40.00th=[ 3064], 50.00th=[ 3523], 60.00th=[ 4015], | 70.00th=[ 4621], 80.00th=[ 5538], 90.00th=[ 7177], 95.00th=[ 9110], | 99.00th=[15270], 99.50th=[19792], 99.90th=[34866], 99.95th=[42206], | 99.99th=[64226] bw ( KiB/s): min=197710, max=1710611, per=100.00%, avg=978380.45, stdev=38019.38, samples=5483 iops: min=49424, max=427652, avg=244594.42, stdev=9504.89, samples=5483 lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec) : 500=0.79%, 750=0.82%, 1000=0.95% lat (msec) : 2=13.06%, 4=44.04%, 10=36.63%, 20=3.21%, 50=0.45% lat (msec) : 100=0.03%, 250=0.01% cpu : usr=3.34%, sys=7.56%, ctx=978696, majf=0, minf=122 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,83886080,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=952MiB/s (998MB/s), 952MiB/s-952MiB/s (998MB/s-998MB/s), io=320GiB (344GB), run=344252-344252msec Disk stats (read/write): sda: ios=336/33974149, merge=0/49890543, ticks=201/35842485, in_queue=35842765, util=100.00% ubuntu@cloudimg:~$ uname --kernel-release 5.15.0-87-generic ``` This shows bw=952MiB/s Summary: Ubuntu 22.04 5.15 kernel `bw=952MiB/s` Ubuntu 20.04 5.4 kernel `bw=1237MiB/s` Ubuntu 20.04 5.15 kernel `bw=605MiB/s` -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2042564 Title: Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel Status in linux package in Ubuntu: New Status in linux source package in Focal: New Bug description: We in the Canonical Public Cloud team have received report from our colleagues in Google regarding a potential performance regression with the 5.15 kernel vs the 5.4 kernel on ubuntu 20.04. Their test were performed using the linux-gkeop and linux-gkeop-5.15 kernels. I have verified with the generic Ubuntu 20.04 5.4 linux-generic and the Ubuntu 20.04 5.15 linux-generic-hwe-20.04 kernels. The tests were run using `fio` fio commands: * 4k initwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` * 4k overwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` My reproducer was to launch an Ubuntu 20.04 cloud image locally with qemu the results are below: Using 5.4 kernel ``` ubuntu@cloudimg:~$ uname --kernel-release 5.4.0-164-generic ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 8 (f=8): [W(8)][99.6%][w=925MiB/s][w=237k IOPS][eta 00m:01s] fiojob1: (groupid=0, jobs=8): err= 0: pid=2443: Thu Nov 2 09:15:22 2023 write: IOPS=317k, BW=1237MiB/s (1297MB/s)(320GiB/264837msec); 0 zone resets slat (nsec): min=628, max=37820k, avg=7207.71, stdev=101058.61 clat (nsec): min=457, max=56099k, avg=340.45, stdev=1707823.38 lat (usec): min=23, max=56100, avg=3229.78, stdev=1705.80 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 1352], 10.00th=[ 1647], 20.00th=[ 2024], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 2933], 60.00th=[ 3261], | 70.00th=[ 3654], 80.00th=[ 4146], 90.00th=[ 5014], 95.00th=[ 5932], |
[Kernel-packages] [Bug 2042564] [NEW] Performance regression in the 5.15 Ubuntu 20.04 kernel compared to 5.4 Ubuntu 20.04 kernel
Public bug reported: We in the Canonical Public Cloud team have received report from our colleagues in Google regarding a potential performance regression with the 5.15 kernel vs the 5.4 kernel on ubuntu 20.04. Their test were performed using the linux-gkeop and linux-gkeop-5.15 kernels. I have verified with the generic Ubuntu 20.04 5.4 linux-generic and the Ubuntu 20.04 5.15 linux-generic-hwe-20.04 kernels. The tests were run using `fio` fio commands: * 4k initwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` * 4k overwrite: `fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sdc` My reproducer was to launch an Ubuntu 20.04 cloud image locally with qemu the results are below: Using 5.4 kernel ``` ubuntu@cloudimg:~$ uname --kernel-release 5.4.0-164-generic ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 8 (f=8): [W(8)][99.6%][w=925MiB/s][w=237k IOPS][eta 00m:01s] fiojob1: (groupid=0, jobs=8): err= 0: pid=2443: Thu Nov 2 09:15:22 2023 write: IOPS=317k, BW=1237MiB/s (1297MB/s)(320GiB/264837msec); 0 zone resets slat (nsec): min=628, max=37820k, avg=7207.71, stdev=101058.61 clat (nsec): min=457, max=56099k, avg=340.45, stdev=1707823.38 lat (usec): min=23, max=56100, avg=3229.78, stdev=1705.80 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 1352], 10.00th=[ 1647], 20.00th=[ 2024], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 2933], 60.00th=[ 3261], | 70.00th=[ 3654], 80.00th=[ 4146], 90.00th=[ 5014], 95.00th=[ 5932], | 99.00th=[ 8979], 99.50th=[10945], 99.90th=[18220], 99.95th=[22676], | 99.99th=[32113] bw ( MiB/s): min= 524, max= 1665, per=100.00%, avg=1237.72, stdev=20.42, samples=4232 iops: min=134308, max=426326, avg=316855.16, stdev=5227.36, samples=4232 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.05%, 500=0.54%, 750=0.37%, 1000=0.93% lat (msec) : 2=17.40%, 4=58.02%, 10=22.01%, 20=0.60%, 50=0.07% lat (msec) : 100=0.01% cpu : usr=3.29%, sys=7.45%, ctx=1262621, majf=0, minf=103 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,83886080,0,8 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=1237MiB/s (1297MB/s), 1237MiB/s-1237MiB/s (1297MB/s-1297MB/s), io=320GiB (344GB), run=264837-264837msec Disk stats (read/write): sda: ios=36/32868891, merge=0/50979424, ticks=5/27498602, in_queue=1183124, util=100.00% ``` After upgrading to linux-generic-hwe-20.04 kernel and rebooting ``` ubuntu@cloudimg:~$ uname --kernel-release 5.15.0-88-generic ubuntu@cloudimg:~$ sudo fio --ioengine=libaio --blocksize=4k --readwrite=write --filesize=40G --end_fsync=1 --iodepth=128 --direct=1 --group_reporting --numjobs=8 --name=fiojob1 --filename=/dev/sda fiojob1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 ... fio-3.16 Starting 8 processes Jobs: 1 (f=1): [_(7),W(1)][100.0%][w=410MiB/s][w=105k IOPS][eta 00m:00s] fiojob1: (groupid=0, jobs=8): err= 0: pid=1438: Thu Nov 2 09:46:49 2023 write: IOPS=155k, BW=605MiB/s (634MB/s)(320GiB/541949msec); 0 zone resets slat (nsec): min=660, max=325426k, avg=10351.04, stdev=232438.50 clat (nsec): min=1100, max=782743k, avg=6595008.67, stdev=6290570.04 lat (usec): min=86, max=782748, avg=6606.08, stdev=6294.03 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 2180], 10.00th=[ 2802], 20.00th=[ 3556], | 30.00th=[ 4178], 40.00th=[ 4817], 50.00th=[ 5538], 60.00th=[ 6259], | 70.00th=[ 7177], 80.00th=[ 8455], 90.00th=[ 10683], 95.00th=[ 13566], | 99.00th=[ 26870], 99.50th=[ 34866], 99.90th=[ 63177], 99.95th=[ 80217], | 99.99th=[145753] bw ( KiB/s): min=39968, max=1683451, per=100.00%, avg=619292.10, stdev=26377.19, samples=8656 iops: min= 9990, max=420862, avg=154822.58, stdev=6594.34, samples=8656 lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=0.05%, 750=0.48%, 1000=0.65% lat (msec) : 2=2.79%, 4=23.00%, 10=60.93%, 20=10.08%, 50=1.83% lat (msec) :
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
https://bugs.launchpad.net/cloud-images/+bug/2038894 is a related bug to track specifically the introduction of listening port 5353 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2038567] Re: Disable restricting unprivileged change_profile by default, due to LXD latest/stable not yet compatible with this new apparmor feature
cloud minimized and non minimized images have now been tested with 6.5.0-9 kernel from -proposed and pass our lxd-start-stop test suite which was failing and which is the test suite which prompted this whole thread. +1 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Disable restricting unprivileged change_profile by default, due to LXD latest/stable not yet compatible with this new apparmor feature Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Fix Committed Status in lxd package in Ubuntu: Triaged Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2038567/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2038567] [NEW] Mantic 6.5.0-7 kernel causes regression in LXD container usage
Public bug reported: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. >From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. ** Affects: ubuntu-release-notes Importance: Undecided Status: New ** Affects: linux (Ubuntu) Importance: Critical Status: Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in linux package in Ubuntu: Incomplete Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to:
[Kernel-packages] [Bug 2036968] Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online
I have also successfully verified that -proposed amd64 kernel `6.5.0-7-generic` results in successful network configuration when tested using qemu on an amd64 host with older hardware (ThinkPad T460 with 6th gen intel i5 which is the same hardware which we were able to reproduce the issue on previously). See https://people.canonical.com/~philroche/20231003-mantic-minimal- proposed-kernel/amd64/ for cloud-init logs, some debug output and test image. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2036968 Title: Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online Status in cloud-images: New Status in linux package in Ubuntu: Fix Committed Status in systemd package in Ubuntu: Triaged Bug description: Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images, there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on amd64 host. The bug is a race condition, so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud- images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This but is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud- init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1 valid_lft 86391sec preferred_lft 86391sec inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute valid_lft 86393sec preferred_lft 14393sec inet6 fe80::5054:ff:fe12:3456/64 scope link valid_lft forever preferred_lft forever ``` The bug is also reproducible with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2036968/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2036968] Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online
@xnox I have successfully verified that -proposed arm64 kernel `6.5.0-7-generic` results in successful network configuration when tested using qemu on an amd64 host. See https://people.canonical.com/~philroche/20231003-mantic-minimal- proposed-kernel/ for cloud-init logs, some debug output and test image. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2036968 Title: Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online Status in cloud-images: New Status in linux package in Ubuntu: Fix Committed Status in systemd package in Ubuntu: Triaged Bug description: Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images, there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on amd64 host. The bug is a race condition, so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud- images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This but is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud- init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1 valid_lft 86391sec preferred_lft 86391sec inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute valid_lft 86393sec preferred_lft 14393sec inet6 fe80::5054:ff:fe12:3456/64 scope link valid_lft forever preferred_lft forever ``` The bug is also reproducible with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2036968/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2037398] Re: kexec enable to load/kdump zstd compressed zimg
I have confirmed that this issue with not being able to capture kernel dump with a mantic arm64 kernel is not new. using the arm64 6.5 kernel (6.5.0-5) in the release pocket I captured the following during test. ``` ubuntu@cloudimg:~$ sudo kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_COREDIR: /var/crash crashkernel addr: 0xde00 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-5-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.5.0-5-generic current state:Not ready to kdump kexec command: no kexec command recorded ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-5-generic #5-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 6 15:36:23 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux ubuntu@cloudimg:~$ sudo cat /proc/cmdline BOOT_IMAGE=/vmlinuz-6.5.0-5-generic root=UUID=e7604ab2-200c-4f34-ab11-47e78ac4b8bd ro console=tty1 console=ttyS0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M ubuntu@cloudimg:~$ sudo dmesg | grep -i crash [0.00] crashkernel reserved: 0xde00 - 0xfe00 (512 MB) [0.00] Kernel command line: BOOT_IMAGE=/vmlinuz-6.5.0-5-generic root=UUID=e7604ab2-200c-4f34-ab11-47e78ac4b8bd ro console=tty1 console=ttyS0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M [ 87.054802] pstore: Using crash dump compression: deflate ubuntu@cloudimg:~$ sudo cat /proc/sys/kernel/sysrq 176 ubuntu@cloudimg:~$ sudo service kdump-tools status â— kdump-tools.service - Kernel crash dump capture service Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; preset: enabled) Active: active (exited) since Tue 2023-09-26 13:44:22 UTC; 7min ago Process: 514 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/SUCCESS) Main PID: 514 (code=exited, status=0/SUCCESS) CPU: 4min 46.461s Sep 26 13:38:21 cloudimg systemd[1]: Starting kdump-tools.service - Kernel crash dump capture service... Sep 26 13:38:32 cloudimg kdump-tools[514]: Starting kdump-tools: Sep 26 13:38:32 cloudimg kdump-tools[535]: * Creating symlink /var/lib/kdump/vmlinuz Sep 26 13:38:41 cloudimg kdump-tools[577]: kdump-tools: Generating /var/lib/kdump/initrd.img-6.5.0-5-generic Sep 26 13:44:11 cloudimg kdump-tools[535]: * Creating symlink /var/lib/kdump/initrd.img Sep 26 13:44:22 cloudimg kdump-tools[535]: * failed to load kdump kernel Sep 26 13:44:22 cloudimg kdump-tools[5536]: failed to load kdump kernel Sep 26 13:44:22 cloudimg systemd[1]: Finished kdump-tools.service - Kernel crash dump capture service. ubuntu@cloudimg:~$ sudo ls -al /var/lib/kdump/vmlinuz lrwxrwxrwx 1 root root 29 Sep 26 13:38 /var/lib/kdump/vmlinuz -> /boot/vmlinuz-6.5.0-5-generic ubuntu@cloudimg:~$ sudo file /boot/vmlinuz-6.5.0-5-generic /boot/vmlinuz-6.5.0-5-generic: gzip compressed data, was "vmlinuz-6.5.0-5-generic.efi.signed", last modified: Thu Sep 7 10:43:26 2023, max compression, from Unix, original size modulo 2^32 54989192 ``` .. but the `Cannot determine the file type of /var/lib/kdump/vmlinuz` in the `sudo service kdump-tools status` as @xnox has also reproduced with `kexec` is new. I have also confirmed that capturing kernel crash dump with the amd64 kernel 6.5.0-5 & 6.5.0-6 works as expected. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to kexec-tools in Ubuntu. https://bugs.launchpad.net/bugs/2037398 Title: kexec enable to load/kdump zstd compressed zimg Status in kdump-tools package in Ubuntu: Invalid Status in kexec-tools package in Ubuntu: Triaged Status in linux package in Ubuntu: Triaged Bug description: While testing the 6.5.0-6-generic proposed arm64 generic kernel I encountered issues being able to use kdump. After enabling -proposed and installing the -proposed 6.5.0-6 kernel and rebooting I encountered the following: `kdump-config show` shows `current state:Not ready to kdump` and looking at the status of the kdump-tools services I see `Cannot determine the file type of /var/lib/kdump/vmlinuz` Full output: ``` ubuntu@cloudimg:~$ sudo kdump-config show DUMP_MODE:kdump USE_KDUMP:1 KDUMP_COREDIR:/var/crash crashkernel addr: 0xde00 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-6-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.5.0-6-generic current state:Not ready to kdump kexec command: no kexec command recorded ubuntu@cloudimg:~$ sudo service kdump-tools status ● kdump-tools.service - Kernel crash dump capture service Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; preset: enabled) Active: active (exited) since Tue 2023-09-26 09:21:44 UTC; 5min ago Process: 515 ExecStart=/etc/init.d/kdump-tools start (code=exited,
[Kernel-packages] [Bug 2037398] [NEW] Unable to capture kernel crash dump using arm64 mantic 6.5.0-6-generic -proposed kernel
Public bug reported: While testing the 6.5.0-6-generic proposed arm64 generic kernel I encountered issues being able to use kdump. After enabling -proposed and installing the -proposed 6.5.0-6 kernel and rebooting I encountered the following: `kdump-config show` shows `current state:Not ready to kdump` and looking at the status of the kdump-tools services I see `Cannot determine the file type of /var/lib/kdump/vmlinuz` Full output: ``` ubuntu@cloudimg:~$ sudo kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_COREDIR: /var/crash crashkernel addr: 0xde00 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-6-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.5.0-6-generic current state:Not ready to kdump kexec command: no kexec command recorded ubuntu@cloudimg:~$ sudo service kdump-tools status ● kdump-tools.service - Kernel crash dump capture service Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; preset: enabled) Active: active (exited) since Tue 2023-09-26 09:21:44 UTC; 5min ago Process: 515 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/SUCCESS) Main PID: 515 (code=exited, status=0/SUCCESS) CPU: 4min 21.329s Sep 26 09:16:14 cloudimg systemd[1]: Starting kdump-tools.service - Kernel crash dump capture service... Sep 26 09:16:24 cloudimg kdump-tools[515]: Starting kdump-tools: Sep 26 09:16:24 cloudimg kdump-tools[537]: * Creating symlink /var/lib/kdump/vmlinuz Sep 26 09:16:32 cloudimg kdump-tools[580]: kdump-tools: Generating /var/lib/kdump/initrd.img-6.5.0-6-generic Sep 26 09:21:42 cloudimg kdump-tools[537]: * Creating symlink /var/lib/kdump/initrd.img Sep 26 09:21:43 cloudimg kdump-tools[5538]: Cannot determine the file type of /var/lib/kdump/vmlinuz Sep 26 09:21:44 cloudimg kdump-tools[537]: * failed to load kdump kernel Sep 26 09:21:44 cloudimg kdump-tools[5539]: failed to load kdump kernel Sep 26 09:21:44 cloudimg systemd[1]: Finished kdump-tools.service - Kernel crash dump capture service. ubuntu@cloudimg:~$ ls -al /var/lib/kdump/vmlinuz lrwxrwxrwx 1 root root 29 Sep 26 09:16 /var/lib/kdump/vmlinuz -> /boot/vmlinuz-6.5.0-6-generic ubuntu@cloudimg:~$ file /var/lib/kdump/vmlinuz /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-6-generic ubuntu@cloudimg:~$ sudo file /boot/vmlinuz-6.5.0-6-generic /boot/vmlinuz-6.5.0-6-generic: PE32+ executable (EFI application) Aarch64 (stripped to external PDB), for MS Windows, 2 sections ``` The reboot with 6.5.0-6 was successful and the reboot after linux-crashdump install was successful too. I used https://ubuntu.com/server/docs/kernel-crash-dump guide for installing linux-crashdump and attempting to trigger a dump. I used arm64 qcow cloud image from http://cloud- images.ubuntu.com/mantic/20230925/mantic-server-cloudimg-arm64.img to test the above emulated on amd64. ** Affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2037398 Title: Unable to capture kernel crash dump using arm64 mantic 6.5.0-6-generic -proposed kernel Status in linux package in Ubuntu: New Bug description: While testing the 6.5.0-6-generic proposed arm64 generic kernel I encountered issues being able to use kdump. After enabling -proposed and installing the -proposed 6.5.0-6 kernel and rebooting I encountered the following: `kdump-config show` shows `current state:Not ready to kdump` and looking at the status of the kdump-tools services I see `Cannot determine the file type of /var/lib/kdump/vmlinuz` Full output: ``` ubuntu@cloudimg:~$ sudo kdump-config show DUMP_MODE:kdump USE_KDUMP:1 KDUMP_COREDIR:/var/crash crashkernel addr: 0xde00 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-6-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.5.0-6-generic current state:Not ready to kdump kexec command: no kexec command recorded ubuntu@cloudimg:~$ sudo service kdump-tools status ● kdump-tools.service - Kernel crash dump capture service Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; preset: enabled) Active: active (exited) since Tue 2023-09-26 09:21:44 UTC; 5min ago Process: 515 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/SUCCESS) Main PID: 515 (code=exited, status=0/SUCCESS) CPU: 4min 21.329s Sep 26 09:16:14 cloudimg systemd[1]: Starting kdump-tools.service - Kernel crash dump capture service... Sep 26 09:16:24 cloudimg kdump-tools[515]: Starting kdump-tools: Sep 26 09:16:24 cloudimg kdump-tools[537]: * Creating symlink /var/lib/kdump/vmlinuz Sep 26 09:16:32 cloudimg
[Kernel-packages] [Bug 2036968] Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning
cloud-init bug filed @ https://github.com/canonical/cloud- init/issues/4451 ** Bug watch added: github.com/canonical/cloud-init/issues #4451 https://github.com/canonical/cloud-init/issues/4451 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2036968 Title: Mantic minimized/minimal cloud images do not receive IP address during provisioning Status in cloud-images: New Status in linux package in Ubuntu: Incomplete Bug description: Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images, there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on amd64 host. The bug is a race condition, so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud- images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This but is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud- init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1 valid_lft 86391sec preferred_lft 86391sec inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute valid_lft 86393sec preferred_lft 14393sec inet6 fe80::5054:ff:fe12:3456/64 scope link valid_lft forever preferred_lft forever ``` The bug is also reproducible with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2036968/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2036968] Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning
** Description changed: Following a recent change from linux-kvm kernel to linux-generic kernel - in the mantic minimized images there is a reproducable bug where a guest - VM does not have an IP address assigned as part of cloud-init + in the mantic minimized images, there is a reproducable bug where a + guest VM does not have an IP address assigned as part of cloud-init provisioning. - This is easiest to reproduce when emulating arm64 on adm64 host. The bug - is a race condition so there could exist fast enough virtualisation on + This is easiest to reproduce when emulating arm64 on amd64 host. The bug + is a race condition, so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud- images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. - This bug is not present in the non minimized/base images @ http://cloud- + This but is not present in the non minimized/base images @ http://cloud- images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` - This is because when cloud-init-local.service is trying to configure - network interfaces it doesn't find any so it doesn't configure any. But - by the time boot is complete the network interface is present but cloud- - init provisioning has already completed. + This is because when cloud-init is trying to configure network + interfaces it doesn't find any so it doesn't configure any. But by the + time boot is complete the network interface is present but cloud-init + provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud-init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1 valid_lft 86391sec preferred_lft 86391sec inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute valid_lft 86393sec preferred_lft 14393sec inet6 fe80::5054:ff:fe12:3456/64 scope link valid_lft forever preferred_lft forever ``` - The bug is also reproducable with amd64 guest on adm64 host on + The bug is also reproducible with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2036968 Title: Mantic minimized/minimal cloud images do not receive IP address during provisioning Status in cloud-images: New Status in linux package in Ubuntu: Incomplete Bug description: Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images, there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
There is a related bug @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036968 which might have affected boot speed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2036968] Re: Mantic minimized/minimal cloud images do not receive IP address during provisioning
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2036968 Title: Mantic minimized/minimal cloud images do not receive IP address during provisioning Status in cloud-images: New Status in linux package in Ubuntu: New Bug description: Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on adm64 host. The bug is a race condition so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud- images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This bug is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud- init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1 valid_lft 86391sec preferred_lft 86391sec inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute valid_lft 86393sec preferred_lft 14393sec inet6 fe80::5054:ff:fe12:3456/64 scope link valid_lft forever preferred_lft forever ``` The bug is also reproducable with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2036968/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@paelzer agreed. Good plan. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@paelzer given the above findings and discussion, I would like to mark this as Invalid for cloud-images project and continue the conversation in the context of kernel only. +1 / -1 ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
** Description changed: - The Mantic (Ubuntu 23.10) images are undergoing some big changes prior - to 23.10 release in October. + The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ + are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages + * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) download/qcow2 images available @ https://cloud-images.ubuntu.com/minimal/ are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * This is during image build only and will not affect any subsequent package installs * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
I have uploaded further data now to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/server-metrics/ with kernelmodules, kernelconfig, services, timers etc. for each of the three images being inspected. This additional data was gathered with a modified fork of the `server-test- scripts` repo @ https://github.com/philroche/server-test- scripts/blob/feature/local-lxc-image-execution-additional-data- gathering/metric-server-simple/metric-server-simple.sh. It seems that most of the mem and process increase is attributed to the kernel change and we know that this was a conscious decision with the following reported bugs supporting that decision. * https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2006488 * https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931841 * https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1685291 Given the above I feel what we can work on is whether any of the process/modules introduced by the switch to the generic kernel should be omitted for the minimal images. The best, easiest source of this information is the data gathered from the latest image with both the generic kernel and the switch to the new minimal seed - https://people.canonical.com/~philroche/20230824-manticl- minimal-LP2032933/server-metrics/20230821.1-after-kernel-change-after- seed-change-mantic-minimal-cloudimg-amd64-data-f93870221eb8/ @seth-arnold You highlighted `ksmd`. Are there any others that concern you. @paelzer Are you happy to adjust your regression testing/metrics gathering to increase the memory required knowing that it was a conscious decision to switch kernel and incur the performance hit for the benefit of using a kernel with more support and less reported bugs? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
@paelzer > The change of the image build sadly combined it all See the description noting https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ which should help in determining where the changes were introduced as I have provided three images across the various stages of changes - no change -> new kernel -> new kernel + new seed For example https://pastebin.ubuntu.com/p/sJ5wGk4G7h/ show the process diff between previous images and images with new kernel only -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2032933] Re: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running
The diff in process count from kernel change image -> kernel change + seed change image is actually a reduction in processes - see diff @ https://pastebin.ubuntu.com/p/PXtQM9gB2K/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2032933 Title: Mantic (23.10) minimal images increase in memory consumption, port usage and processes running Status in cloud-images: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: The Mantic (Ubuntu 23.10) images are undergoing some big changes prior to 23.10 release in October. This is a devel release so this is the perfect time to be making these changes but we are noticing some changes that were not expected. This bug is to track the unexpected changes and discuss/resolve these. The changes that have been made to mantic minimal: * Move to the linux-generic kernel from the linux-kvm kernel * This also involved removal of the virtio-blk driver, which is the default for QEMU and OpenStack, but this is being restored in an upcoming 6.5 mantic kernel and is being trakced @ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2030745 * Move to using minimal-cloud seed - see https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.mantic/cloud-minimal * No longer installing Recommends packages * No initramfs fallback for boot - only initramfsless boot The latest mantic minimal images are available @ http://cloud- images.ubuntu.com/minimal/daily/mantic/ and are also available in the public clouds. A package name manifest diff can be seen @ https://pastebin.ubuntu.com/p/rRd6STnNmK/ We have had reports of higher memory usage on an idle system, higher number of ports open on an idle system and higher number of process running on a idle system. To help with debugging I have built and uploaded the following images and package manifests to https://people.canonical.com/~philroche/20230824-manticl-minimal- LP2032933/ * 20230618-before-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * Before kernel change and before seed change * 20230824-after-kernel-change-before-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and before seed change * 20230821.1-after-kernel-change-after-seed-change-mantic-minimal-cloudimg-amd64 * After kernel change and after seed change To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2032933/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1882955] Re: LXD 4.2 broken on linux-kvm due to missing VLAN filtering
CPC are seeing this issue in _all_ minimal cloud images testing with LXD snap version 4.2 or greater. This blocks promotion of all minimal cloud download images and blocks build and publication of both daily and release cloud images. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1882955 Title: LXD 4.2 broken on linux-kvm due to missing VLAN filtering Status in linux-kvm package in Ubuntu: Triaged Bug description: This is another case of linux-kvm having unexplained differences compared to linux-generic in areas that aren't related to hardware drivers (see other bug we filed for missing nft). This time, CPC is reporting that LXD no longer works on linux-kvm as we now set vlan filtering on our bridges to prevent containers from escaping firewalling through custom vlan tags. This relies on CONFIG_BRIDGE_VLAN_FILTERING which is a built-in on the generic kernel but is apparently missing on linux-kvm (I don't have any system running that kernel to confirm its config, but the behavior certainly matches that). We need this fixed in focal and groovy. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-kvm/+bug/1882955/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1861704] Re: ubuntu-fan recommends netcat package in universe
It appears that a bug was already filed against netcat https://bugs.launchpad.net/ubuntu/+source/netcat-openbsd/+bug/1780316 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to ubuntu-fan in Ubuntu. https://bugs.launchpad.net/bugs/1861704 Title: ubuntu-fan recommends netcat package in universe Status in ubuntu-fan package in Ubuntu: New Bug description: Following some debug of the docker.io package in universe, we (Canonical CPC) discovered that the ubuntu-fan package from main and the netcat-traditional package from universe were being installed. This was due to the following dependency/recommends tree: * docker.io (in universe) recommends ubuntu-fan (main) * ubuntu-fan (main) recommends netcat (universe) * netcat (universe) is a transitional package and depends on netcat-traditional (universe) Our concern is that this might be a packaging violation as ubuntu-fan is recommending a package not in main. > In addition, the packages in main > > must not require a package outside of main for compilation or execution (thus, the package must > not declare a "Depends", "Recommends", or "Build-Depends" relationship on a non-main package), Source: https://people.canonical.com/~cjwatson/ubuntu- policy/policy.html/ch-archive.html#s-main I will file a bug against netcat too to start a discussion on netcat being built from netcat-openbsd (main) instead of netcat-traditional (universe). Our feeling is that netcat is such a frequently depended on or recommended package that it being present in main would benefit Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-fan/+bug/1861704/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1861704] [NEW] ubuntu-fan recommends netcat package in universe
Public bug reported: Following some debug of the docker.io package in universe, we (Canonical CPC) discovered that the ubuntu-fan package from main and the netcat- traditional package from universe were being installed. This was due to the following dependency/recommends tree: * docker.io (in universe) recommends ubuntu-fan (main) * ubuntu-fan (main) recommends netcat (universe) * netcat (universe) is a transitional package and depends on netcat-traditional (universe) Our concern is that this might be a packaging violation as ubuntu-fan is recommending a package not in main. > In addition, the packages in main > > must not require a package outside of main for compilation or execution > (thus, the package must > not declare a "Depends", "Recommends", or "Build-Depends" relationship on a > non-main package), Source: https://people.canonical.com/~cjwatson/ubuntu-policy/policy.html /ch-archive.html#s-main I will file a bug against netcat too to start a discussion on netcat being built from netcat-openbsd (main) instead of netcat-traditional (universe). Our feeling is that netcat is such a frequently depended on or recommended package that it being present in main would benefit Ubuntu. ** Affects: ubuntu-fan (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to ubuntu-fan in Ubuntu. https://bugs.launchpad.net/bugs/1861704 Title: ubuntu-fan recommends netcat package in universe Status in ubuntu-fan package in Ubuntu: New Bug description: Following some debug of the docker.io package in universe, we (Canonical CPC) discovered that the ubuntu-fan package from main and the netcat-traditional package from universe were being installed. This was due to the following dependency/recommends tree: * docker.io (in universe) recommends ubuntu-fan (main) * ubuntu-fan (main) recommends netcat (universe) * netcat (universe) is a transitional package and depends on netcat-traditional (universe) Our concern is that this might be a packaging violation as ubuntu-fan is recommending a package not in main. > In addition, the packages in main > > must not require a package outside of main for compilation or execution (thus, the package must > not declare a "Depends", "Recommends", or "Build-Depends" relationship on a non-main package), Source: https://people.canonical.com/~cjwatson/ubuntu- policy/policy.html/ch-archive.html#s-main I will file a bug against netcat too to start a discussion on netcat being built from netcat-openbsd (main) instead of netcat-traditional (universe). Our feeling is that netcat is such a frequently depended on or recommended package that it being present in main would benefit Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-fan/+bug/1861704/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1795017] Re: linux-azure on cosmic minimal image boot speed
Myself and @kamalmostafa spoke and the issue is not obvious especially as linux-azure is installed in the base image too which does not exhibit the issue. Although not obvious @kamalmostafa did suspect the issue to lie somewhere with random number generation and a related package/kernel patch. One line of investigation is to check the package diff between the minimal image with initramfs-tools installed and the image without that installed. That diff is @ https://pastebin.canonical.com/p/7hJTSQHbyD/ Another possibility is to check the package diff between base and minimal. That diff is @ https://pastebin.canonical.com/p/B9m7GbsFhs/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1795017 Title: linux-azure on cosmic minimal image boot speed Status in linux-azure package in Ubuntu: New Bug description: Hi, We're planning to share a new Cosmic minimal image for Azure including the linux-azure custom kernel. I have found boot speed increase from ~55s in non minimal image to over 6mins for minimal image. I have gathered boot log, systemd logs and cloud-init logs for this instance @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/. You can see the same logs for the non minimal image @ https://private-fileshare.canonical.com/~philroche /azure-cosmic-20180927/base Most of the time seems to be spent in crng and "task swapper/0:1" (?) [5.774760] sd 3:0:0:0: [sdc] Attached SCSI disk [ 71.388989] random: crng init done [ 242.656057] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.688434] Not tainted 4.15.0-1018-azure #18-Ubuntu Source: https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/iengal3seeSee-vm-imagedebug-boot.log Please let me know if you need any more information or would like me to launch an instance for testing. I understand that minimal images are intended to boot without initramfs but I did install initramfs-tools in another test image which did result in much better boot times. See https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180928/ for logs. Minimal image boot time 48.784s To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1795017/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1795017] Re: linux-azure on cosmic minimal image boot speed
@kamalmostafa @mhcerri I noticed that linux-azure 4.18.0.1003.3 was uploaded to cosmic-proposed so I built a new test image. For convenience I have imported your SSH keys to 'ssh ubuntu@51.140.217.58'. All logs and VHD etc. have been uploaded to https://private- fileshare.canonical.com/~philroche/azure-cosmic-20181011/ ** Attachment added: "cosmic minimal image with linux-azure 4.18.0.1003.3 (from cosmic-proposed) boot log" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1795017/+attachment/5199957/+files/eeDaquoh3aiqu-vm-imagedebug-boot.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1795017 Title: linux-azure on cosmic minimal image boot speed Status in linux-azure package in Ubuntu: New Bug description: Hi, We're planning to share a new Cosmic minimal image for Azure including the linux-azure custom kernel. I have found boot speed increase from ~55s in non minimal image to over 6mins for minimal image. I have gathered boot log, systemd logs and cloud-init logs for this instance @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/. You can see the same logs for the non minimal image @ https://private-fileshare.canonical.com/~philroche /azure-cosmic-20180927/base Most of the time seems to be spent in crng and "task swapper/0:1" (?) [5.774760] sd 3:0:0:0: [sdc] Attached SCSI disk [ 71.388989] random: crng init done [ 242.656057] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.688434] Not tainted 4.15.0-1018-azure #18-Ubuntu Source: https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/iengal3seeSee-vm-imagedebug-boot.log Please let me know if you need any more information or would like me to launch an instance for testing. I understand that minimal images are intended to boot without initramfs but I did install initramfs-tools in another test image which did result in much better boot times. See https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180928/ for logs. Minimal image boot time 48.784s To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1795017/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1795017] Re: linux-azure on cosmic minimal image boot speed
On request on IRC I have tried a new image with linux-azure 4.18.0.1002.2 from cosmic-proposed. This too exhibited the slow boot 5min 19.070s (kernel) + 41.724s (userspace) = 6min 794ms Full logs @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20181001/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1795017 Title: linux-azure on cosmic minimal image boot speed Status in linux-azure package in Ubuntu: New Bug description: Hi, We're planning to share a new Cosmic minimal image for Azure including the linux-azure custom kernel. I have found boot speed increase from ~55s in non minimal image to over 6mins for minimal image. I have gathered boot log, systemd logs and cloud-init logs for this instance @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/. You can see the same logs for the non minimal image @ https://private-fileshare.canonical.com/~philroche /azure-cosmic-20180927/base Most of the time seems to be spent in crng and "task swapper/0:1" (?) [5.774760] sd 3:0:0:0: [sdc] Attached SCSI disk [ 71.388989] random: crng init done [ 242.656057] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.688434] Not tainted 4.15.0-1018-azure #18-Ubuntu Source: https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/iengal3seeSee-vm-imagedebug-boot.log Please let me know if you need any more information or would like me to launch an instance for testing. I understand that minimal images are intended to boot without initramfs but I did install initramfs-tools in another test image which did result in much better boot times. See https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180928/ for logs. Minimal image boot time 48.784s To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1795017/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1795017] Re: linux-azure on cosmic minimal image boot speed
** Attachment added: "cosmic minimal image with linux-azure 4.18.0.1002.2 (from cosmic-proposed) boot log" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1795017/+attachment/5195337/+files/chohv4Va7Ahx7-vm-imagedebug-boot.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1795017 Title: linux-azure on cosmic minimal image boot speed Status in linux-azure package in Ubuntu: New Bug description: Hi, We're planning to share a new Cosmic minimal image for Azure including the linux-azure custom kernel. I have found boot speed increase from ~55s in non minimal image to over 6mins for minimal image. I have gathered boot log, systemd logs and cloud-init logs for this instance @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/. You can see the same logs for the non minimal image @ https://private-fileshare.canonical.com/~philroche /azure-cosmic-20180927/base Most of the time seems to be spent in crng and "task swapper/0:1" (?) [5.774760] sd 3:0:0:0: [sdc] Attached SCSI disk [ 71.388989] random: crng init done [ 242.656057] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.688434] Not tainted 4.15.0-1018-azure #18-Ubuntu Source: https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/iengal3seeSee-vm-imagedebug-boot.log Please let me know if you need any more information or would like me to launch an instance for testing. I understand that minimal images are intended to boot without initramfs but I did install initramfs-tools in another test image which did result in much better boot times. See https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180928/ for logs. Minimal image boot time 48.784s To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1795017/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1795017] [NEW] linux-azure on cosmic minimal image boot speed
Public bug reported: Hi, We're planning to share a new Cosmic minimal image for Azure including the linux-azure custom kernel. I have found boot speed increase from ~55s in non minimal image to over 6mins for minimal image. I have gathered boot log, systemd logs and cloud-init logs for this instance @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/. You can see the same logs for the non minimal image @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/base Most of the time seems to be spent in crng and "task swapper/0:1" (?) [5.774760] sd 3:0:0:0: [sdc] Attached SCSI disk [ 71.388989] random: crng init done [ 242.656057] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.688434] Not tainted 4.15.0-1018-azure #18-Ubuntu Source: https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/iengal3seeSee-vm-imagedebug-boot.log Please let me know if you need any more information or would like me to launch an instance for testing. I understand that minimal images are intended to boot without initramfs but I did install initramfs-tools in another test image which did result in much better boot times. See https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180928/ for logs. Minimal image boot time 48.784s ** Affects: linux-azure (Ubuntu) Importance: Undecided Status: New ** Attachment added: "cosmic minimal image with linux-azure boot log" https://bugs.launchpad.net/bugs/1795017/+attachment/5194121/+files/iengal3seeSee-vm-imagedebug-boot.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-azure in Ubuntu. https://bugs.launchpad.net/bugs/1795017 Title: linux-azure on cosmic minimal image boot speed Status in linux-azure package in Ubuntu: New Bug description: Hi, We're planning to share a new Cosmic minimal image for Azure including the linux-azure custom kernel. I have found boot speed increase from ~55s in non minimal image to over 6mins for minimal image. I have gathered boot log, systemd logs and cloud-init logs for this instance @ https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/. You can see the same logs for the non minimal image @ https://private-fileshare.canonical.com/~philroche /azure-cosmic-20180927/base Most of the time seems to be spent in crng and "task swapper/0:1" (?) [5.774760] sd 3:0:0:0: [sdc] Attached SCSI disk [ 71.388989] random: crng init done [ 242.656057] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.688434] Not tainted 4.15.0-1018-azure #18-Ubuntu Source: https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180927/minimal/iengal3seeSee-vm-imagedebug-boot.log Please let me know if you need any more information or would like me to launch an instance for testing. I understand that minimal images are intended to boot without initramfs but I did install initramfs-tools in another test image which did result in much better boot times. See https://private-fileshare.canonical.com/~philroche/azure- cosmic-20180928/ for logs. Minimal image boot time 48.784s To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1795017/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1779961] Re: Boot delays with 4.15.0-24 based kernels in images
The meta-packages have been rolled back and the fix is in progress of being applied and uploaded. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1779961 Title: Boot delays with 4.15.0-24 based kernels in images Status in linux package in Ubuntu: Triaged Bug description: We (Canonical CPC) are seeing an issue with today's build of the bionic minimal cloud images. Boot stops at "Starting Initial cloud- init job (metadata service crawler)..." for nearly 5 minutes (booting on scalingstack and locally with kvm) this is with a new kernel, but otherwise no changes over the prior image. Image (manifest and logs) exhibiting this behaviour @ https://private- fileshare.canonical.com/~philroche/bionic-minimal-20180703/ With server teams's help we have narrowed it down to "Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com)." taking over 4 minutes. On Kamal's instruction we also tried linux-kvm 4.15.0-1013.13 and bionic GA linux (4.15.0-24.26). Both of which also exhibit this regression. For reference: Kamal noted on IRC that the problem was possibly introduced in the GA kernel and is not specific to the kvm kernel. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779961/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1779961] [NEW] Bionic minimal image boot hang with linux-image-4.15.0-1012-kvm kernel
Public bug reported: We (Canonical CPC) are seeing an issue with today's build of the bionic minimal cloud images. Boot stops at "Starting Initial cloud-init job (metadata service crawler)..." for nearly 5 minutes (booting on scalingstack and locally with kvm) this is with a new kernel, but otherwise no changes over the prior image. Image (manifest and logs) exhibiting this behaviour @ https://private- fileshare.canonical.com/~philroche/bionic-minimal-20180703/ With server teams's help we have narrowed it down to "Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com)." taking over 4 minutes. On Kamal's instruction we also tried linux-kvm 4.15.0-1013.13 and bionic GA linux (4.15.0-24.26). Both of which also exhibit this regression. For reference: Kamal noted on IRC that the problem was possibly introduced in the GA kernel and is not specific to the kvm kernel. ** Affects: linux (Ubuntu) Importance: Undecided Status: Incomplete ** Tags: bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1779961 Title: Bionic minimal image boot hang with linux-image-4.15.0-1012-kvm kernel Status in linux package in Ubuntu: Incomplete Bug description: We (Canonical CPC) are seeing an issue with today's build of the bionic minimal cloud images. Boot stops at "Starting Initial cloud- init job (metadata service crawler)..." for nearly 5 minutes (booting on scalingstack and locally with kvm) this is with a new kernel, but otherwise no changes over the prior image. Image (manifest and logs) exhibiting this behaviour @ https://private- fileshare.canonical.com/~philroche/bionic-minimal-20180703/ With server teams's help we have narrowed it down to "Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com)." taking over 4 minutes. On Kamal's instruction we also tried linux-kvm 4.15.0-1013.13 and bionic GA linux (4.15.0-24.26). Both of which also exhibit this regression. For reference: Kamal noted on IRC that the problem was possibly introduced in the GA kernel and is not specific to the kvm kernel. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779961/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
