[Touch-packages] [Bug 1729836] Re: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook
Thank you for taking the time to report this bug and helping to make Ubuntu better. Ubuntu 17.10 (artful) reached end-of-life on July 19, 2018. See this document for currently supported Ubuntu releases: https://wiki.ubuntu.com/Releases We appreciate that this bug may be old and you might not be interested in discussing it anymore. But if you are then please upgrade to the latest Ubuntu version and re-test. If you then find the bug is still present in the newer Ubuntu version, please add a comment here telling us which new version it is in. ** Changed in: initramfs-tools (Ubuntu) Status: Triaged => Incomplete -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to initramfs-tools in Ubuntu. https://bugs.launchpad.net/bugs/1729836 Title: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook Status in initramfs-tools package in Ubuntu: Incomplete Bug description: After running 'update-initramfs -u -k $(uname -r)', Linux will not boot. With 'quiet splash' set in grub, it appears to freeze after the 'Loading ramdisk...' message, and if booted in recovery mode, I see the following kernel panic: Freeing unused kernel memory: 948k Freeing unused kernel memory: 104K x86/mm: Chcked W+X mappings: passed, no W+X pages found. Failed to execute /init (error -2) Kernel panic - not syncing: No working init found. Try passing init= optin to kernel. See Linux Documentation/admin-guide/init.rst for guidance. CPU: 6 PID: 1 Comm: swapper/0 Not tainted 4.13.0-16-generic #19-Ubuntu Hardware name: Dell Inc. OptiPlex 990/0D6H9T, BIOS A07 09/10/2011 Call Trace: dump_stack+0x63/0x8b ? rest_init+0xb0/0xc0 panic+0xe4/0x23d ? putname+0x4b/0x50 ? rest_init+0xc0/0xc0 kernel_init+0xeb/0xfc ret_from_fork0x25/0x30 Kernel Offset: 0x1fc0 from 0x8a00 (relocation range: 0x8000-0xbfff) ---[ end Kernel panic - no syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. (full screenshot at https://photos.app.goo.gl/XLNKOM3cDNHsfyFt2). Examining the initrd files, here is a good one, as installed by the Ubuntu 17.10 installer: # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: ASCII cpio archive (SVR4 with no CRC) # ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50764087 Nov 3 21:02 /boot/initrd.img-4.13.0-16-generic and here is the file after 'update-initramfs -u -k 4.13.0-16-generic': # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: gzip compressed data, last modified: Thu Nov 2 10:51:42 2017, from Unix root@jturner-home:/boot# ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50895514 Nov 3 21:01 /boot/initrd.img-4.13.0-16-generic Good and bad initrds can be downloaded from https://drive.google.com/drive/folders/1reMWj1TohrQ4K5EbBMJPhw4xXUdBi0K9?usp=sharing After downloading initrd.img-4.13.0-16-generic-broken, the panic can be replicated in qemu with: kvm -m 1G -kernel /boot/vmlinuz-4.13.0-16-generic -initrd /boot/initrd.img-4.13.0-16-generic-broken (tip obtained from https://lists.debian.org/debian- kernel/2016/02/msg00323.html) I've experienced this problem with Ubuntu 17.04 and 17.10 on two separate PCs (both Dell OptiPlex 990s). Ubuntu boots fine initially, until one day I update the kernel or something, and a .deb postinst script calls update-initramfs and trashes that kernel's initrd file. If I run 'update-initramfs -u -k all', all my old initrd files are toast. The "good" initrd format "ASCII cpio archive (SVR4 with no CRC)" is strange. I'm not getting many google hits on it. None of my other Ubuntu/Debian servers use that format in their initrds. ProblemType: Bug DistroRelease: Ubuntu 17.10 Package: initramfs-tools 0.125ubuntu12 ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4 Uname: Linux 4.13.0-16-generic x86_64 ApportVersion: 2.20.7-0ubuntu3.1 Architecture: amd64 Date: Fri Nov 3 21:33:12 2017 InstallationDate: Installed on 2017-08-05 (90 days ago) InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412) PackageArchitecture: all SourcePackage: initramfs-tools UpgradeStatus: Upgraded to artful on 2017-10-24 (10 days ago) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1729836/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1729836] Re: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook
** Changed in: initramfs-tools (Ubuntu) Status: Incomplete => Triaged ** Changed in: initramfs-tools (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to initramfs-tools in Ubuntu. https://bugs.launchpad.net/bugs/1729836 Title: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook Status in initramfs-tools package in Ubuntu: Triaged Bug description: After running 'update-initramfs -u -k $(uname -r)', Linux will not boot. With 'quiet splash' set in grub, it appears to freeze after the 'Loading ramdisk...' message, and if booted in recovery mode, I see the following kernel panic: Freeing unused kernel memory: 948k Freeing unused kernel memory: 104K x86/mm: Chcked W+X mappings: passed, no W+X pages found. Failed to execute /init (error -2) Kernel panic - not syncing: No working init found. Try passing init= optin to kernel. See Linux Documentation/admin-guide/init.rst for guidance. CPU: 6 PID: 1 Comm: swapper/0 Not tainted 4.13.0-16-generic #19-Ubuntu Hardware name: Dell Inc. OptiPlex 990/0D6H9T, BIOS A07 09/10/2011 Call Trace: dump_stack+0x63/0x8b ? rest_init+0xb0/0xc0 panic+0xe4/0x23d ? putname+0x4b/0x50 ? rest_init+0xc0/0xc0 kernel_init+0xeb/0xfc ret_from_fork0x25/0x30 Kernel Offset: 0x1fc0 from 0x8a00 (relocation range: 0x8000-0xbfff) ---[ end Kernel panic - no syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. (full screenshot at https://photos.app.goo.gl/XLNKOM3cDNHsfyFt2). Examining the initrd files, here is a good one, as installed by the Ubuntu 17.10 installer: # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: ASCII cpio archive (SVR4 with no CRC) # ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50764087 Nov 3 21:02 /boot/initrd.img-4.13.0-16-generic and here is the file after 'update-initramfs -u -k 4.13.0-16-generic': # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: gzip compressed data, last modified: Thu Nov 2 10:51:42 2017, from Unix root@jturner-home:/boot# ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50895514 Nov 3 21:01 /boot/initrd.img-4.13.0-16-generic Good and bad initrds can be downloaded from https://drive.google.com/drive/folders/1reMWj1TohrQ4K5EbBMJPhw4xXUdBi0K9?usp=sharing After downloading initrd.img-4.13.0-16-generic-broken, the panic can be replicated in qemu with: kvm -m 1G -kernel /boot/vmlinuz-4.13.0-16-generic -initrd /boot/initrd.img-4.13.0-16-generic-broken (tip obtained from https://lists.debian.org/debian- kernel/2016/02/msg00323.html) I've experienced this problem with Ubuntu 17.04 and 17.10 on two separate PCs (both Dell OptiPlex 990s). Ubuntu boots fine initially, until one day I update the kernel or something, and a .deb postinst script calls update-initramfs and trashes that kernel's initrd file. If I run 'update-initramfs -u -k all', all my old initrd files are toast. The "good" initrd format "ASCII cpio archive (SVR4 with no CRC)" is strange. I'm not getting many google hits on it. None of my other Ubuntu/Debian servers use that format in their initrds. ProblemType: Bug DistroRelease: Ubuntu 17.10 Package: initramfs-tools 0.125ubuntu12 ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4 Uname: Linux 4.13.0-16-generic x86_64 ApportVersion: 2.20.7-0ubuntu3.1 Architecture: amd64 Date: Fri Nov 3 21:33:12 2017 InstallationDate: Installed on 2017-08-05 (90 days ago) InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412) PackageArchitecture: all SourcePackage: initramfs-tools UpgradeStatus: Upgraded to artful on 2017-10-24 (10 days ago) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1729836/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1729836] Re: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook
Logged at https://github.com/sickill/stderred/issues/63 ** Bug watch added: github.com/sickill/stderred/issues #63 https://github.com/sickill/stderred/issues/63 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to initramfs-tools in Ubuntu. https://bugs.launchpad.net/bugs/1729836 Title: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook Status in initramfs-tools package in Ubuntu: Incomplete Bug description: After running 'update-initramfs -u -k $(uname -r)', Linux will not boot. With 'quiet splash' set in grub, it appears to freeze after the 'Loading ramdisk...' message, and if booted in recovery mode, I see the following kernel panic: Freeing unused kernel memory: 948k Freeing unused kernel memory: 104K x86/mm: Chcked W+X mappings: passed, no W+X pages found. Failed to execute /init (error -2) Kernel panic - not syncing: No working init found. Try passing init= optin to kernel. See Linux Documentation/admin-guide/init.rst for guidance. CPU: 6 PID: 1 Comm: swapper/0 Not tainted 4.13.0-16-generic #19-Ubuntu Hardware name: Dell Inc. OptiPlex 990/0D6H9T, BIOS A07 09/10/2011 Call Trace: dump_stack+0x63/0x8b ? rest_init+0xb0/0xc0 panic+0xe4/0x23d ? putname+0x4b/0x50 ? rest_init+0xc0/0xc0 kernel_init+0xeb/0xfc ret_from_fork0x25/0x30 Kernel Offset: 0x1fc0 from 0x8a00 (relocation range: 0x8000-0xbfff) ---[ end Kernel panic - no syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. (full screenshot at https://photos.app.goo.gl/XLNKOM3cDNHsfyFt2). Examining the initrd files, here is a good one, as installed by the Ubuntu 17.10 installer: # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: ASCII cpio archive (SVR4 with no CRC) # ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50764087 Nov 3 21:02 /boot/initrd.img-4.13.0-16-generic and here is the file after 'update-initramfs -u -k 4.13.0-16-generic': # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: gzip compressed data, last modified: Thu Nov 2 10:51:42 2017, from Unix root@jturner-home:/boot# ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50895514 Nov 3 21:01 /boot/initrd.img-4.13.0-16-generic Good and bad initrds can be downloaded from https://drive.google.com/drive/folders/1reMWj1TohrQ4K5EbBMJPhw4xXUdBi0K9?usp=sharing After downloading initrd.img-4.13.0-16-generic-broken, the panic can be replicated in qemu with: kvm -m 1G -kernel /boot/vmlinuz-4.13.0-16-generic -initrd /boot/initrd.img-4.13.0-16-generic-broken (tip obtained from https://lists.debian.org/debian- kernel/2016/02/msg00323.html) I've experienced this problem with Ubuntu 17.04 and 17.10 on two separate PCs (both Dell OptiPlex 990s). Ubuntu boots fine initially, until one day I update the kernel or something, and a .deb postinst script calls update-initramfs and trashes that kernel's initrd file. If I run 'update-initramfs -u -k all', all my old initrd files are toast. The "good" initrd format "ASCII cpio archive (SVR4 with no CRC)" is strange. I'm not getting many google hits on it. None of my other Ubuntu/Debian servers use that format in their initrds. ProblemType: Bug DistroRelease: Ubuntu 17.10 Package: initramfs-tools 0.125ubuntu12 ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4 Uname: Linux 4.13.0-16-generic x86_64 ApportVersion: 2.20.7-0ubuntu3.1 Architecture: amd64 Date: Fri Nov 3 21:33:12 2017 InstallationDate: Installed on 2017-08-05 (90 days ago) InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412) PackageArchitecture: all SourcePackage: initramfs-tools UpgradeStatus: Upgraded to artful on 2017-10-24 (10 days ago) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1729836/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1729836] Re: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook
I think I figured out the cause. Basically PEBCAK.. There's a program called stderred (https://github.com/sickill/stderred) which colorizes stderr to differentiate it from stdout, via a LD_PRELOAD hook. I have had this set for years in my ~/.bashrc: root@jturner-home:~# echo $LD_PRELOAD /home/jturner/src/github/stderred/build/libstderred.so When I examine the initrd file as follows, I see half the binaries in bin/ contain ANSI escape codes, and the libstderred.so library has been pulled in: root@jturner-home:/boot# update-initramfs -u -k 4.13.0-25-generic update-initramfs: Generating /boot/initrd.img-4.13.0-25-generic root@jturner-home:/boot# mkdir bad root@jturner-home:/boot# ( cd bad; zcat ../initrd.img-4.13.0-25-generic | cpio -idmv; ) root@jturner-home:/boot# ls bad/home/jturner/src/github/stderred/build/libstderred.so bad/home/jturner/src/github/stderred/build/libstderred.so root@jturner-home:/boot# ls -1 bad/bin ?[0m ?[0m?[1m?[31m[[?[0m?[1m?[31m ?[0m?[1m?[31macpid?[0m?[1m?[31m ?[0m?[1m?[31mash?[0m?[1m?[31m ?[0m?[1m?[31mawk?[0m?[1m?[31m ?[0m?[1m?[31mbasename?[0m?[1m?[31m ?[0m?[1m?[31mblockdev?[0m?[1m?[31m ?[0m?[1m?[31mcat?[0m?[1m?[31m ... ?[1m?[31m[?[0m?[1m?[31m busybox cat chroot cpio dash date dd ... The output of update-initramfs goes red halfway. Perhaps update-initramfs could sanitize its environment, or explicitly unset LD_PRELOAD. Hard to be idiot-proof when there's such sophisticated idiots :) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to initramfs-tools in Ubuntu. https://bugs.launchpad.net/bugs/1729836 Title: update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook Status in initramfs-tools package in Ubuntu: Incomplete Bug description: After running 'update-initramfs -u -k $(uname -r)', Linux will not boot. With 'quiet splash' set in grub, it appears to freeze after the 'Loading ramdisk...' message, and if booted in recovery mode, I see the following kernel panic: Freeing unused kernel memory: 948k Freeing unused kernel memory: 104K x86/mm: Chcked W+X mappings: passed, no W+X pages found. Failed to execute /init (error -2) Kernel panic - not syncing: No working init found. Try passing init= optin to kernel. See Linux Documentation/admin-guide/init.rst for guidance. CPU: 6 PID: 1 Comm: swapper/0 Not tainted 4.13.0-16-generic #19-Ubuntu Hardware name: Dell Inc. OptiPlex 990/0D6H9T, BIOS A07 09/10/2011 Call Trace: dump_stack+0x63/0x8b ? rest_init+0xb0/0xc0 panic+0xe4/0x23d ? putname+0x4b/0x50 ? rest_init+0xc0/0xc0 kernel_init+0xeb/0xfc ret_from_fork0x25/0x30 Kernel Offset: 0x1fc0 from 0x8a00 (relocation range: 0x8000-0xbfff) ---[ end Kernel panic - no syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. (full screenshot at https://photos.app.goo.gl/XLNKOM3cDNHsfyFt2). Examining the initrd files, here is a good one, as installed by the Ubuntu 17.10 installer: # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: ASCII cpio archive (SVR4 with no CRC) # ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50764087 Nov 3 21:02 /boot/initrd.img-4.13.0-16-generic and here is the file after 'update-initramfs -u -k 4.13.0-16-generic': # file /boot/initrd.img-4.13.0-16-generic /boot/initrd.img-4.13.0-16-generic: gzip compressed data, last modified: Thu Nov 2 10:51:42 2017, from Unix root@jturner-home:/boot# ls -la /boot/initrd.img-4.13.0-16-generic -rw-r--r-- 1 root root 50895514 Nov 3 21:01 /boot/initrd.img-4.13.0-16-generic Good and bad initrds can be downloaded from https://drive.google.com/drive/folders/1reMWj1TohrQ4K5EbBMJPhw4xXUdBi0K9?usp=sharing After downloading initrd.img-4.13.0-16-generic-broken, the panic can be replicated in qemu with: kvm -m 1G -kernel /boot/vmlinuz-4.13.0-16-generic -initrd /boot/initrd.img-4.13.0-16-generic-broken (tip obtained from https://lists.debian.org/debian- kernel/2016/02/msg00323.html) I've experienced this problem with Ubuntu 17.04 and 17.10 on two separate PCs (both Dell OptiPlex 990s). Ubuntu boots fine initially, until one day I update the kernel or something, and a .deb postinst script calls update-initramfs and trashes that kernel's initrd file. If I run 'update-initramfs -u -k all', all my old initrd files are toast. The "good" initrd format "ASCII cpio archive (SVR4 with no CRC)" is strange. I'm not getting many google hits on it. None of my other Ubuntu/Debian servers use that format in their initrds. ProblemType: Bug DistroRelease: Ubuntu 17.10 Package: initramfs-tools 0.125ubuntu12 ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4 Uname: Linux 4.13.0-16-generic x86_64 ApportVersion: 2.20.7-0ubuntu3.1 Architecture: amd64 Da