[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
This bug was fixed in the package grub2-signed - 1.66.23 --- grub2-signed (1.66.23) xenial; urgency=medium * Rebuild against grub2 2.02~beta2-36ubuntu3.23. (LP: #1840686) -- Eric Desrochers Tue, 05 Nov 2019 00:43:00 + ** Changed in: grub2-signed (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
This bug was fixed in the package grub2 - 2.02~beta2-36ubuntu3.23 --- grub2 (2.02~beta2-36ubuntu3.23) xenial; urgency=medium * d/p/fix_booting_for_large_root_volumes.patch: Cherry pick upstream fix for booting on systems with large root volumes, either by default or from resizing. (LP: #1840686) -- Matthew Ruffell Sat, 19 Oct 2019 17:47:16 +1300 ** Changed in: grub2 (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
I have run our tests and am satisfied with the results. Our tests very closely matches what mruffell posted. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
As per Łukasz's request, I have performed verification testing across a wide range of affected and unaffected systems, with the grub2-2.02~beta2-36ubuntu3.23 package in -proposed. All tests were performed on GCP. Test case one: Summary: efi enabled image with 3072gb disk. Disk: 3072gb Bios/efi: efi Image: daily-ubuntu-1604-xenial-v20190731, ubuntu-os-cloud-devel Affected: Yes Behaviour with grub2-2.02~beta2-36ubuntu3.22 from -updates: Fails to reboot - as expected because of this bug. Log: https://paste.ubuntu.com/p/KKH7r3vdrC/ Behaviour with grub2-2.02~beta2-36ubuntu3.23 from -proposed: If you follow the instructions in the test section, the instance reboots successfully. Note, grub-install must be explicitly called. If you do not manually run grub-install, rebooting will fail. Log: https://paste.ubuntu.com/p/7KxmDCjFSG/ Log (apt upgrade, no manual grub-install): https://paste.ubuntu.com/p/j3VK5PV7GR/ Test case two: Summary: efi enabled image with 10gb disk. Disk: 10gb Bios/efi: efi Image: daily-ubuntu-1604-xenial-v20190731, ubuntu-os-cloud-devel Affected: No Behaviour with grub2-2.02~beta2-36ubuntu3.22 from -updates: Reboots successfully. Behaviour with grub2-2.02~beta2-36ubuntu3.23 from -proposed: Reboots successfully. Logs: https://paste.ubuntu.com/p/c65jjYjj6S/ Note, grub-install was not invoked manually, and represents a typical user apt upgrade with no interaction. Test case three: Summary: bios enabled image with 10gb disk. Disk: 10gb Bios/efi: bios Image: ubuntu-1604-xenial-v20191024 Affected: No Behaviour with grub2-2.02~beta2-36ubuntu3.22 from -updates: Reboots successfully. Behaviour with grub2-2.02~beta2-36ubuntu3.23 from -proposed: Reboots successfully. Logs: https://paste.ubuntu.com/p/N3QFX63WCS/ Note, grub-install was not invoked manually, and represents a typical user apt upgrade with no interaction. Test case four: Summary: bios enabled image with 3072gb disk. Disk: 3072gb Bios/efi: bios Image: ubuntu-1604-xenial-v20191024 Affected: No Behaviour with grub2-2.02~beta2-36ubuntu3.22 from -updates: Reboots successfully. Behaviour with grub2-2.02~beta2-36ubuntu3.23 from -proposed: Reboots successfully. Logs: https://paste.ubuntu.com/p/KZw4kcD6pS/ Note, grub-install was not invoked manually, and represents a typical user apt upgrade with no interaction. Log (apt upgrade, WITH manual grub-install): https://paste.ubuntu.com/p/RFBR5BbbtH/ Conclusion: grub2-2.02~beta2-36ubuntu3.23 from -proposed fixes this bug. It does not introduce any regressions for non-affected use cases. The only thing to note, is that for efi based images with disk > 2tb, if the user does not manually run grub-install after installing the package in -proposed then rebooting will fail. This is no better than the current situation of failing to reboot regardless, and because of this, there is unlikely to be any users out there who are running an image with disk > 2tb and have never rebooted ever, so it is unlikely this will be a problem. This will however, fix all images and new instances using the fixed version of grub moving forward. Taking this into consideration, I am happy to mark this as verified. Pat, feel free to also test. I will also see if the customer is interested in testing as well. ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Changed in: grub2-signed (Ubuntu Xenial) Assignee: (unassigned) => Eric Desrochers (slashd) ** Changed in: grub2-signed (Ubuntu Xenial) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Hello Pat, or anyone else affected, Accepted grub2-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.66.23 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: grub2-signed (Ubuntu Xenial) Status: New => Fix Committed ** Changed in: grub2 (Ubuntu Xenial) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Package changed: grub (Ubuntu) => grub2 (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Uploaded grub2-signed "Rebuild against grub2 2.02~beta2-36ubuntu3.23. (LP: #1840686)" ** Changed in: grub2-signed (Ubuntu) Status: In Progress => Fix Released ** Changed in: grub2-signed (Ubuntu) Assignee: Eric Desrochers (slashd) => (unassigned) ** Changed in: grub2-signed (Ubuntu) Importance: High => Undecided -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Also affects: grub2-signed (Ubuntu) Importance: Undecided Status: New ** Changed in: grub2-signed (Ubuntu) Status: New => In Progress ** Changed in: grub2-signed (Ubuntu) Importance: Undecided => High ** Changed in: grub2-signed (Ubuntu) Assignee: (unassigned) => Eric Desrochers (slashd) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Hello Pat, or anyone else affected, Accepted grub2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-36ubuntu3.23 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Tags added: verification-needed verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Since the change seems to be quite invasive, I would like to ask to perform some additional testing on non-affected, regular ext* systems before we promote this to -updates. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Sponsored for Xenial. The package is now waiting for SRU approval in order to start building in xenial-proposed for the testing phase of the SRU. Thanks for your contribution Matthew ! ** Tags removed: sts-sponsor sts-sponsor-slashd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Tags added: sts-sponsor-slashd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Description changed: [Impact] - GCP wishes for xenial images to use GPT instead of MBR as a part of - their efforts to change to efi based booting, but they have hit an issue - where after booting an instance that has a disk size of 2049 GB or - higher, we hang on the next subsequent boot (Logs indicate it hanging on - "Booting Hard Disk 0"). + On Xenial images which use GPT instead of MBR to enable efi based + booting, there is an issue where after booting an instance that has a + disk size of 2049 GB or higher, we hang on the next subsequent boot + (Logs indicate it hanging on "Booting Hard Disk 0"). This is a problem in grub2 where the system would become unbootable after ext* online resize if no resize_inode was created at ext* format time. [Test Case] To reproduce: 1) Create an image with a disk size of 3072 GB using a serial that has GPT: gcloud compute instances create test-3072-xenial --image daily- ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 2) Reboot the instance The instance will hang on reboot and you cannot connect. If you go to GCP console and select Logs > Serial port 1 (console), you will see the boot process has stopped at "Booting Hard Disk 0". I have built a test package, which is available here: https://launchpad.net/~mruffell/+archive/ubuntu/lp1840686-test If you do step 1) but do not reboot, and instead add the PPA, install the new grub like so: 1) gcloud compute instances create test-3072-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 2) sudo add-apt-repository ppa:mruffell/lp1840686-test 3) sudo apt-get update 4) sudo apt remove grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common 5) sudo apt install grub-common grub-efi-amd64 grub-efi-amd64-bin grub-pc-bin grub2-common 6) sudo grub-install /dev/sda 7) sudo reboot The instance will boot successfully and you will be able to connect. Note, we must use "daily-ubuntu-1604-xenial-v20190731" as the image, as it is enabled for GPT and efi. GCP was reverted back to MBR and bios booting because of this bug, so the latest images will not reproduce the problem. [Regression Potential] Grub is a core package and every care must be taken in order to not introduce any regressions. The commit is present in B, D, E and F, and is considered well tested and widely adopted by the community. The commit comes with its own testcase, to test the ext4_metabg fix. The changes are localised to ext* based filesystems, although since they are the most popular family of filesystems used by the community, this does not reduce risk of breakage by much. If a regression were to happen, a regression would have a large impact, and in the worst case, can lead to unbootable systems and data loss for users who are not technical enough to reinstall grub from a working package inside the broken system chroot. [Other Info] In comment #4, Sultan identifies the fix as: commit e20aa39ea4298011ba716087713cff26c6c52006 Author: Vladimir Serbinenko Date: Mon Feb 16 20:53:26 2015 +0100 Subject: ext2: Support META_BG. This commit is from upstream grub2, and can be found here: https://git.savannah.gnu.org/cgit/grub.git/commit/?id=e20aa39ea4298011ba716087713cff26c6c52006 Looking at when this was merged: $ git describe --contains e20aa39ea4298011ba716087713cff26c6c52006 2.02-beta3~429 This commit is present in B, D, E and F, leaving X as the only version needing an SRU. The commit cleanly cherry picks to X, because the delta from 2.02~beta2-36ubuntu3.22 to 2.02-beta3~429 is small. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Thanks for testing Pat. Attached is the debdiff for xenial to fix this bug. ** Patch added: "grub2 debdiff for xenial" https://bugs.launchpad.net/cloud-init/+bug/1840686/+attachment/5301264/+files/lp1840686_xenial.debdiff ** Tags added: sts-sponsor -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
I have re-ran my test cases and the package you provided fixes the original issue that we saw. Thank you -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Description changed: [Impact] GCP wishes for xenial images to use GPT instead of MBR as a part of their efforts to change to efi based booting, but they have hit an issue where after booting an instance that has a disk size of 2049 GB or higher, we hang on the next subsequent boot (Logs indicate it hanging on - "Booting Hard Disk 0". + "Booting Hard Disk 0"). This is a problem in grub2 where the system would become unbootable after ext* online resize if no resize_inode was created at ext* format time. [Test Case] To reproduce: - 1) Create an image with a disk size of 3072 using a serial that has GPT: + 1) Create an image with a disk size of 3072 GB using a serial that has + GPT: gcloud compute instances create test-3072-xenial --image daily- ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 2) Reboot the instance The instance will hang on reboot and you cannot connect. If you go to GCP console and select Logs > Serial port 1 (console), you will see the boot process has stopped at "Booting Hard Disk 0". I have built a test package, which is available here: https://launchpad.net/~mruffell/+archive/ubuntu/lp1840686-test If you do step 1) but do not reboot, and instead add the PPA, install the new grub like so: 1) gcloud compute instances create test-3072-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 2) sudo add-apt-repository ppa:mruffell/lp1840686-test 3) sudo apt-get update 4) sudo apt remove grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common 5) sudo apt install grub-common grub-efi-amd64 grub-efi-amd64-bin grub-pc-bin grub2-common 6) sudo grub-install /dev/sda 7) sudo reboot The instance will boot successfully and you will be able to connect. Note, we must use "daily-ubuntu-1604-xenial-v20190731" as the image, as it is enabled for GPT and efi. GCP was reverted back to MBR and bios booting because of this bug, so the latest images will not reproduce the problem. [Regression Potential] Grub is a core package and every care must be taken in order to not introduce any regressions. The commit is present in B, D, E and F, and is considered well tested and widely adopted by the community. The commit comes with its own testcase, to test the ext4_metabg fix. The changes are localised to ext* based filesystems, although since they are the most popular family of filesystems used by the community, this does not reduce risk of breakage by much. If a regression were to happen, a regression would have a large impact, and in the worst case, can lead to unbootable systems and data loss for users who are not technical enough to reinstall grub from a working package inside the broken system chroot. [Other Info] In comment #4, Sultan identifies the fix as: commit e20aa39ea4298011ba716087713cff26c6c52006 Author: Vladimir Serbinenko Date: Mon Feb 16 20:53:26 2015 +0100 Subject: ext2: Support META_BG. This commit is from upstream grub2, and can be found here: https://git.savannah.gnu.org/cgit/grub.git/commit/?id=e20aa39ea4298011ba716087713cff26c6c52006 Looking at when this was merged: $ git describe --contains e20aa39ea4298011ba716087713cff26c6c52006 2.02-beta3~429 This commit is present in B, D, E and F, leaving X as the only version needing an SRU. The commit cleanly cherry picks to X, because the delta from 2.02~beta2-36ubuntu3.22 to 2.02-beta3~429 is small. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Description changed: - CPC team has recently converted Xenial images to use GPT instead of MBR. - However, after booting an instance that has a disk size of 2049 GB or + [Impact] + + GCP wishes for xenial images to use GPT instead of MBR as a part of + their efforts to change to efi based booting, but they have hit an issue + where after booting an instance that has a disk size of 2049 GB or higher, we hang on the next subsequent boot (Logs indicate it hanging on "Booting Hard Disk 0". - This works on Bionic, but what makes it strange is that they have the - same kernel revision - 4.15.0-1-37. + This is a problem in grub2 where the system would become unbootable + after ext* online resize if no resize_inode was created at ext* format + time. - patrick_viafore@patviafore-test-3072-xenial:~$ lsb_release -rd - Description:Ubuntu 16.04.6 LTS - Release:16.04 - patrick_viafore@patviafore-test-3072-xenial:~$ sudo dpkg -l | grep linux-gcp - ii linux-gcp4.15.0.1037.51 amd64Complete Google Cloud Platform (GCP) Linux kernel and headers - ii linux-gcp-headers-4.15.0-10374.15.0-1037.39~16.04.1 amd64Header files related to Linux kernel version 4.15.0 + [Test Case] To reproduce: - 1) Create an image with a disk size of 3072 using a serial that has GPT - gcloud compute instances create test-3072-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 + 1) Create an image with a disk size of 3072 using a serial that has GPT: - Reboot the instance + gcloud compute instances create test-3072-xenial --image daily- + ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel + --boot-disk-size 3072 - 2) It will hang on reboot and you cannot connect + 2) Reboot the instance - 3) Please note that later serials have the GPT change reverted. + The instance will hang on reboot and you cannot connect. If you go to + GCP console and select Logs > Serial port 1 (console), you will see the + boot process has stopped at "Booting Hard Disk 0". - You can replace xenial with bionic in the above commands to get a bionic - instance instead. + I have built a test package, which is available here: - To test this out in a more slower fashion: + https://launchpad.net/~mruffell/+archive/ubuntu/lp1840686-test - 1) Create an image with a disk size of 2048 using a serial that has GPT - gcloud compute instances create test-2048-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 2048 + If you do step 1) but do not reboot, and instead add the PPA, install + the new grub like so: - 2) Resize the disk to 3072 + 1) gcloud compute instances create test-3072-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 + 2) sudo add-apt-repository ppa:mruffell/lp1840686-test + 3) sudo apt-get update + 4) sudo apt remove grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common + 5) sudo apt install grub-common grub-efi-amd64 grub-efi-amd64-bin grub-pc-bin grub2-common + 6) sudo grub-install /dev/sda + 7) sudo reboot - 3) Issue growpart /dev/sda 1 + The instance will boot successfully and you will be able to connect. - 4) Issue resize2fs /dev/sda1 + Note, we must use "daily-ubuntu-1604-xenial-v20190731" as the image, as + it is enabled for GPT and efi. GCP was reverted back to MBR and bios + booting because of this bug, so the latest images will not reproduce the + problem. - 5) Issue rsize2fs /dev/sda1 again + [Regression Potential] - On the second resize2fs, it tries to resize again, but on a working - instance, it says there's nothing to resize. + Grub is a core package and every care must be taken in order to not + introduce any regressions. - I've tried starting from a Xenial instance and doing a do-release- - upgrade to get to bionic and then doing the growpart/resize2fs, but the - issue still shows up. + The commit is present in B, D, E and F, and is considered well tested + and widely adopted by the community. + + The commit comes with its own testcase, to test the ext4_metabg fix. + + The changes are localised to ext* based filesystems, although since they + are the most popular family of filesystems used by the community, this + does not reduce risk of breakage by much. + + If a regression were to happen, a regression would have a large impact, + and in the worst case, can lead to unbootable systems and data loss for + users who are not technical enough to reinstall grub from a working + package inside the broken system chroot. + + [Other Info] + + In comment #4, Sultan identifies the fix as: + + commit e20aa39ea4298011ba716087713cff26c6c52006 + Author: Vladimir Serbinenko + Date: Mon Feb 16 20:53:26 2015 +0100 + Subject: ext2: Support META_BG. + + This commit is fr
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
I have built a test package for grub2, which is available here: https://launchpad.net/~mruffell/+archive/ubuntu/lp1840686-test It contains http://git.savannah.gnu.org/cgit/grub.git/commit/?id=e20aa39ea4298011ba716087713cff26c6c52006 mentioned in comment #4. I did the following to test that the test package fixes the problem: 1) gcloud compute instances create test-3072-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 2) sudo add-apt-repository ppa:mruffell/lp1840686-test 3) sudo apt-get update 4) sudo apt remove grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common 5) sudo apt install grub-common grub-efi-amd64 grub-efi-amd64-bin grub-pc-bin grub2-common 6) sudo grub-install /dev/sda 7) sudo reboot The instance started like normal and you can connect to it. ** Changed in: grub (Ubuntu Xenial) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Changed in: grub (Ubuntu Xenial) Status: Triaged => In Progress ** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
This is fixed in the version of grub2 in Eoan which will become Ubuntu 19.10. ** Changed in: grub (Ubuntu) Status: New => Fix Released ** Changed in: grub (Ubuntu Xenial) Status: New => Triaged ** Changed in: grub (Ubuntu Xenial) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
This is an old bug in GRUB. This commit fixes it: http://git.savannah.gnu.org/cgit/grub.git/commit/?id=e20aa39ea4298011ba716087713cff26c6c52006 To test it, apply it to a GRUB source tree, compile it, install it, and then reinstall the bootloader with "sudo grub-install /dev/sda". ** Also affects: grub Importance: Undecided Status: New ** Changed in: linux-gcp (Ubuntu) Status: New => Won't Fix ** Also affects: grub (Ubuntu) Importance: Undecided Status: New ** No longer affects: grub ** Also affects: grub (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: linux-gcp (Ubuntu Xenial) Importance: Undecided Status: New ** No longer affects: linux-gcp (Ubuntu) ** No longer affects: linux-gcp (Ubuntu Xenial) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Based on IRC conversation, I don't believe this is a cloud-init bug. Please set the task back to New if I'm mistaken! ** Changed in: cloud-init Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Description changed: CPC team has recently converted Xenial images to use GPT instead of MBR. However, after booting an instance that has a disk size of 2049 GB or higher, we hang on the next subsequent boot (Logs indicate it hanging on "Booting Hard Disk 0". This works on Bionic, but what makes it strange is that they have the same kernel revision - 4.15.0-1-37. patrick_viafore@patviafore-test-3072-xenial:~$ lsb_release -rd Description:Ubuntu 16.04.6 LTS Release:16.04 patrick_viafore@patviafore-test-3072-xenial:~$ sudo dpkg -l | grep linux-gcp ii linux-gcp4.15.0.1037.51 amd64Complete Google Cloud Platform (GCP) Linux kernel and headers ii linux-gcp-headers-4.15.0-10374.15.0-1037.39~16.04.1 amd64Header files related to Linux kernel version 4.15.0 To reproduce: 1) Create an image with a disk size of 3072 using a serial that has GPT gcloud compute instances create test-3072-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 3072 Reboot the instance 2) It will hang on reboot and you cannot connect 3) Please note that later serials have the GPT change reverted. You can replace xenial with bionic in the above commands to get a bionic instance instead. - To test this out in a more slower fashion: - 1) Create an image with a disk size of 2048 using a serial that has GPT gcloud compute instances create test-2048-xenial --image daily-ubuntu-1604-xenial-v20190731 --image-project ubuntu-os-cloud-devel --boot-disk-size 2048 2) Resize the disk to 3072 3) Issue growpart /dev/sda 1 4) Issue resize2fs /dev/sda1 - 5) Issue rsize2fs /dev/sda1 instead + 5) Issue rsize2fs /dev/sda1 again On the second resize2fs, it tries to resize again, but on a working instance, it says there's nothing to resize. - - I've tried starting from a Xenial instance and doing a do-release-upgrade to get to bionic and then doing the growpart/resize2fs, but the issue still shows up. + I've tried starting from a Xenial instance and doing a do-release- + upgrade to get to bionic and then doing the growpart/resize2fs, but the + issue still shows up. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Tags added: id-5d484a6466c79944a30e4644 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
At first I thought it was related to https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=1e19b01be31fc5264a84d246023ecf29e44949df&context=25&ignorews=0&dt=0, and later I thought it was related to https://bugs.launchpad.net/bugs/1762748. However, I added the bionic archives to my Xenial instance and updated e2fsprogs and cloud-utils, then tried to grow the disk past 2048, and ran into the reboot issue again. I agree it was very close to, and the recreate information in that bug helped me narrow down what I was seeing. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
Seems related or at least "close to" bug 1762748. If nothing else, that bug has nice local recreate information. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1840686] Re: Xenial images won't reboot if disk size is > 2TB when using GPT
** Also affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1840686 Title: Xenial images won't reboot if disk size is > 2TB when using GPT To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1840686/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
