Re: /boot disk partition size

[Michael Mikowski]

> We considered adapting the kernel autoremoval code in apt to consider free
space, but decided against it, as it removes safety/redundancy. If users
install more kernels than their boot partition supports, upgrades
*should* fail, rather than providing less safety by removing kernels.

> I'm strongly opposed to anyone shipping different implementations of
kernel autoremoval code, we really ought to clean all that up and
move everyone to the new code inside libapt-pkg.

Hi Julian:

I agree that it makes sense to upstream any improvements. I hope you can
appreciate we had to take immediate action due to a packagekit bug where
kernel cleanup stopped working.  It was either that or face a tsunami of of
broken systems and 2-hour support calls for over-full /boot partitions.
Thanks to the hard work of Matthias Klumpp, it looks now like the package
kit bug has been resolved, but the total time from initial report to
distribution will be perhaps 18 months. The kennel cleaner tool was created
and distributed in 2 days, and it works. We also increased /boot size in
new installation to provide a margin of safety.

When the /boot partition over fills, it doesn't always break cleanly. While
packaging improvements may have helped recently, in our own testing and in
the field we saw apt left in an inconsistent state and even the inability
to boot. Repair in that situation requires a rescue disk, and knowledge of
chroot, LUKS, lvm, apt, grub, and other advanced Linux topics. And hours of
time on the phone walking an often-casual user through all those steps. Ugh.

Even when kernel auto remove is working correctly, our assessment was it
wasn't sufficient. It doesn't consider remaining disk space, and it
actively protects up to 4 kernel version. The default boot size only has
room for 3.

I would appreciate an opportunity to participate in the specifications for
an improved kennel cleaning algorithm. I certainly have hands-on experience
both developing effective specs and deep knowledge and experience with this
issue. This is perhaps a good start:
https://github.com/Bleuzen/ubuntu-kernel-autoremove/issues/1#issuecomment-1028425717

Sincerely, Mike

Packagekit bug: https://github.com/PackageKit/PackageKit/issues/450
Kernel clean cli: https://github.com/Bleuzen/ubuntu-kernel-autoremove



On Tue, Feb 22, 2022, 01:20 Julian Andres Klode 
wrote:

> On Mon, Feb 21, 2022 at 02:44:14PM -0800, Michael Mikowski wrote:
> > Hi Brian:
> >
> > First, let me state the title of the bug I filed was unfortunate and I’ve
> > changed it accordingly. The body of the request shows that it is a
> proposal
> > to increase the /boot partition size, but the title sounded demanding.
> > Also, the file sizes were ambiguous. I have updated them both.
> >
> > https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1960089
> >
> > > A consequence of Ubuntu being open and customizable is that users can
> do
> > many things that are not ones that we endorse.
> >
> > Agreed, but I believe Ubuntu should support multiple kernels for a few
> > reasons:
> >
> > 1. We agree that the outcome of an overfull /boot partition can be
> > catastrophic for less-than-expert users. It seems, therefore, that the
> > system should help avoid this situation commiserate with its potential
> cost.
> >
> > 2. There are no obvious mechanisms to guide novice users with kernel
> > management. Unless I’m missing something (always a possibility) there is
> no
> > system that actively monitors the /boot partition *size* and warns users
> of
> > filling disk (impending doom?). We did this (see
> > https://kfocus.org/wf/tools#kclean) but we don’t know if or when this
> will
> > be upstreamed.
>
> We considered adapting the kernel autoremoval code in apt to consider free
> space, but decided against it, as it removes safety/redundancy. If users
> install more kernels than their boot partition supports, upgrades
> *should* fail, rather than providing less safety by removing kernels.
>
> I'm strongly opposed to anyone shipping different implementations of
> kernel autoremoval code, we really ought to clean all that up and
> move everyone to the new code inside libapt-pkg.
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer  i speak de, ena
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: /boot disk partition size

[Julian Andres Klode]

On Mon, Feb 21, 2022 at 02:44:14PM -0800, Michael Mikowski wrote:
> Hi Brian:
> 
> First, let me state the title of the bug I filed was unfortunate and I’ve
> changed it accordingly. The body of the request shows that it is a proposal
> to increase the /boot partition size, but the title sounded demanding.
> Also, the file sizes were ambiguous. I have updated them both.
> 
> https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1960089
> 
> > A consequence of Ubuntu being open and customizable is that users can do
> many things that are not ones that we endorse.
> 
> Agreed, but I believe Ubuntu should support multiple kernels for a few
> reasons:
> 
> 1. We agree that the outcome of an overfull /boot partition can be
> catastrophic for less-than-expert users. It seems, therefore, that the
> system should help avoid this situation commiserate with its potential cost.
> 
> 2. There are no obvious mechanisms to guide novice users with kernel
> management. Unless I’m missing something (always a possibility) there is no
> system that actively monitors the /boot partition *size* and warns users of
> filling disk (impending doom?). We did this (see
> https://kfocus.org/wf/tools#kclean) but we don’t know if or when this will
> be upstreamed.

We considered adapting the kernel autoremoval code in apt to consider free
space, but decided against it, as it removes safety/redundancy. If users
install more kernels than their boot partition supports, upgrades
*should* fail, rather than providing less safety by removing kernels.

I'm strongly opposed to anyone shipping different implementations of
kernel autoremoval code, we really ought to clean all that up and
move everyone to the new code inside libapt-pkg.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: /boot disk partition size

[Michael Mikowski]

Hi Brian:

First, let me state the title of the bug I filed was unfortunate and I’ve
changed it accordingly. The body of the request shows that it is a proposal
to increase the /boot partition size, but the title sounded demanding.
Also, the file sizes were ambiguous. I have updated them both.

https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1960089

> A consequence of Ubuntu being open and customizable is that users can do
many things that are not ones that we endorse.

Agreed, but I believe Ubuntu should support multiple kernels for a few
reasons:

1. We agree that the outcome of an overfull /boot partition can be
catastrophic for less-than-expert users. It seems, therefore, that the
system should help avoid this situation commiserate with its potential cost.

2. There are no obvious mechanisms to guide novice users with kernel
management. Unless I’m missing something (always a possibility) there is no
system that actively monitors the /boot partition *size* and warns users of
filling disk (impending doom?). We did this (see
https://kfocus.org/wf/tools#kclean) but we don’t know if or when this will
be upstreamed.

3. Users install different kernel flavors for many legitimate workflows.
Three examples immediately come to mind: 3.1) People who do low-latency A/V
work frequently switch between the generic-hwe and lowlatency-hwe kernels;
3.2) The community often encourages users to explore different kernel
flavors to improve hardware support; and 3.3) Developers switch to stock
generic to test for deployment, then back to a later kernel, like hwe or
oem, for better hardware support.

> We are all in agreement that the /boot partition size needs increasing

Agreed, and the increase you have made to 1.5G is much better.

> For us to move forward in the discussion about increasing it to 2G it
would really help for you to answer my previous questions regarding the use
cases for having multiple kernels installed.

>> I’d really like more information on the use case where multiple kernel
flavors will be installed. You mention switching from HWE to OEM, given
that this is a switch wouldn’t the HWE kernel be removed and then you are
left with just one flavor of kernel?

Kernel packages rarely replace one another and so it is trivial to install
generic (stock), generic-hwe, lowlatency-hwe, oem-20.04d, and others
concurrently. This is a welcome feature as discussed in (3) above.

The calculations provided in
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1960089 suggest
that Ubuntu should support up to three flavors at once. For example, a user
may have generic-hwe, lowlatency-hwe, and a third kernel flavor for
hardware support (oem-20.04d) or deployment testing (linux-generic).

If we agree that supporting 3 concurrent kernel flavors is good, then what
is a reasonable space requirement? During an upgrade, the /boot partition
is first filled up with new package files before any cleanup by autoremove
or unattended-upgrades can occur. Also, Ubuntu often updates multiple
kernel flavors on the same day. For example, generic-hwe and
lowlatency-hwe are almost always released concurrently. Thus, we can
reasonably assume there will be times when the /boot partition will need to
accommodate the files for 3 flavors and 2 versions each. The total,
therefore, is 2 x 3 = 6 kernels.

We have found the files in /boot take over 180MiB for each kernel for a
system with Nvidia hardware. Rounding up to 200MiB to accommodate for
future growth, we get 6 x 200MiB = 1.2GiB minimum. Add space for 4
additional kernel images as a safety factor and we get 2.0GiB.

One might ask if maintaining room for 4 extra kernels is a reasonable
safety factor, and that’s a fair question. Here are some edge cases to
consider that can keep additional kernel images around: a) The running
kernel package is never removed (see
/etc/apt/apt.conf.d/01autoremove-kernels) which could keep an extra kernel;
b) A metapackage has been constructed to transition to newer flavor (for
example, upgrading oem-20.04c to oem-20.04d); c) The system has not had a
cleaning event occur for a while and kernel packages have accumulated. My
feeling is this safety factor is warranted given how expensive recovery is
for most users.

I hope you find that useful, Brian, and thank you for your consideration.

Sincerely, Mike



On Mon, Feb 21, 2022 at 12:19 PM Brian Murray  wrote:

> On Tue, Feb 15, 2022 at 10:47:59PM -0800, Michael Mikowski wrote:
> > Hi Everyone:
> >
> > Aaron: On an encrypted Ubuntu system, the EFI partition mounted on
> > /boot/efi is 512M too. But here we are talking about the /boot partition
> > which is distinct. It is typically 705M and formatted with ext4, although
> > subsequent released will likely be larger. The question here is by how
> much.
> >
> > We have offered this suggestion as means  to move a proven solution
> > component upstream for the benefit of everyone using 22.04. We
> implemented
> > an enlarged /boot partition and improved 

Re: /boot disk partition size

[Brian Murray]

On Tue, Feb 15, 2022 at 10:47:59PM -0800, Michael Mikowski wrote:
> Hi Everyone:
> 
> Aaron: On an encrypted Ubuntu system, the EFI partition mounted on
> /boot/efi is 512M too. But here we are talking about the /boot partition
> which is distinct. It is typically 705M and formatted with ext4, although
> subsequent released will likely be larger. The question here is by how much.
> 
> We have offered this suggestion as means  to move a proven solution
> component upstream for the benefit of everyone using 22.04. We implemented
> an enlarged /boot partition and improved kernel management and cleaning
> over a year ago because our customers needed it. See the support cost
> calculations below about why we moved quickly when this became an issue. We
> expected to see much more of if we didn't take action. The problem was made
> more urgent by a packagekit bug.
> 
> Using an additional 0.5% (1.295G costing < $0.25) of a 250G NVMe drive is a
> very small price to pay to help avoid a catastrophic failure mode from
> which most users cannot recover.

When interviewing candidates for positions at Canonical I've had the
opportunity to speak to Ubuntu users from around the world. Sometimes
these are people who installed Ubuntu on whatever system they had lying
around and they weren't in a position to buy a new PC or drive. So while
the additional space in /boot might cost $0.25 for you, the cost for
other people may be much greater. When developing Ubuntu we need to take
into consideration the fact that our users are located in a variety of
different countries and subsequently have access to different resources.

> And users *do* install multiple kernels images for many legitimate and
> exploratory reasons, and this *does* cause full disks on the default
> partition size. If people are doing this, and the OS allows it (and it
> should), then how can it not be a supported use case?

A consequence of Ubuntu being open and customizable is that users can do
many things that are not ones that we endorse.

> Should we remove safety belts from cars because we arbitrarily decide that
> braking hard is not a supported use case? Where are Ubuntu's supported use
> cases, DFMEAs, and KPCs available for review?
> 
> Compare the cost of $0.25 disk space versus the cost helping one novice
> user recover an overfull /boot partition. This can easily exceed $200 in
> direct support and lost productivity.  That's 800 *times* more expensive.
> Worse, if the user can't find help expert-level assistance, they are highly
> motivated to abandon Ubuntu altogether.
> 
> If Ubuntu is for everyone, then we should be catering to the vast majority
> to whom the $0.25 of disk space doesn't matter. These users also often lack
> the deep skill set to recover from this failure mode. Conversely, the tiny
> fraction who want to optimize /boot size are those best equipped to do so -
> they are typically on a much smaller disk and are creating custom
> installations for things like embedded or edge systems. The installer is no
> impediment for these professionals.
> 
> We could continue discussing minutia about why this may or may not be a
> good idea, however, I believe the cost / benefit analysis above is highly
> convincing. We are confident from experience that a 2G /boot partition is a
> very good choice that works with best practice for all common and readable
> use cases. A quick web search also shows this size is frequently
> recommended by other experienced desktop Linux users.

We are all in agreement that the /boot partition size needs / needed
increasing. For us to move forward in the discussion about increasing it
to 2G it would really help for you to answer my previous questions
regarding the use cases for having multiple kernels installed. Those
questions were in the following email:

https://lists.ubuntu.com/archives/ubuntu-devel/2022-February/041856.html

Cheers,
--
Brian Murray

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: /boot disk partition size

[Ian Bruntlett]

Hi,

Quick question.
I'm using an install of Ubuntu 20.04.3 LTS x86_64 with one partition for /
and another partition for swap. All my /boot files are in a subdirectory of
my / partition.

My drive is 1TB in size.

Am I doing something wrong?

TIA,


Ian

-- 
-- ACCU - Professionalism in programming - http://www.accu.org
-- My writing - https://sites.google.com/site/ianbruntlett/
-- Free Software page -
https://sites.google.com/site/ianbruntlett/home/free-software
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: /boot disk partition size

[Michael Mikowski]

Hi Everyone:

Aaron: On an encrypted Ubuntu system, the EFI partition mounted on
/boot/efi is 512M too. But here we are talking about the /boot partition
which is distinct. It is typically 705M and formatted with ext4, although
subsequent released will likely be larger. The question here is by how much.

We have offered this suggestion as means  to move a proven solution
component upstream for the benefit of everyone using 22.04. We implemented
an enlarged /boot partition and improved kernel management and cleaning
over a year ago because our customers needed it. See the support cost
calculations below about why we moved quickly when this became an issue. We
expected to see much more of if we didn't take action. The problem was made
more urgent by a packagekit bug.

Using an additional 0.5% (1.295G costing < $0.25) of a 250G NVMe drive is a
very small price to pay to help avoid a catastrophic failure mode from
which most users cannot recover. And users *do* install multiple kernels
images for many legitimate and exploratory reasons, and this *does* cause
full disks on the default partition size. If people are doing this, and the
OS allows it (and it should), then how can it not be a supported use case?
Should we remove safety belts from cars because we arbitrarily decide that
braking hard is not a supported use case? Where are Ubuntu's supported use
cases, DFMEAs, and KPCs available for review?

Compare the cost of $0.25 disk space versus the cost helping one novice
user recover an overfull /boot partition. This can easily exceed $200 in
direct support and lost productivity.  That's 800 *times* more expensive.
Worse, if the user can't find help expert-level assistance, they are highly
motivated to abandon Ubuntu altogether.

If Ubuntu is for everyone, then we should be catering to the vast majority
to whom the $0.25 of disk space doesn't matter. These users also often lack
the deep skill set to recover from this failure mode. Conversely, the tiny
fraction who want to optimize /boot size are those best equipped to do so -
they are typically on a much smaller disk and are creating custom
installations for things like embedded or edge systems. The installer is no
impediment for these professionals.

We could continue discussing minutia about why this may or may not be a
good idea, however, I believe the cost / benefit analysis above is highly
convincing. We are confident from experience that a 2G /boot partition is a
very good choice that works with best practice for all common and readable
use cases. A quick web search also shows this size is frequently
recommended by other experienced desktop Linux users.

I hope you find this useful.

Sincerely, Mike







On Tue, Feb 15, 2022, 19:59 athoneycutt 
wrote:

> I second this as we are at the point where this size increase won't affect
> most folks. Pop!_OS uses a 512MB /boot (/boot/efi) partition which works
> well for dual boot setups as well.
>
>
> Sent from ProtonMail mobile
>
>
>
>  Original Message 
> On Feb 14, 2022, 03:45, Michael Mikowski < mmikow...@kfocus.org> wrote:
>
>
> Hi Everyone:
>
> Members of the ubuntu-meeting asked me to clarify the issue with /boot.
> You can see the bugs filed at https://launchpad.net/bugs/1959971 and
> https://launchpad.net/bugs/1960089.
> *Personal Background:*
> I currently lead the Kubuntu Focus  engineering
> efforts, and have been a professional product engineer specializing in
> mission-critical HP/HA/HV systems for decades. You can see a overview of my
> experience at https://michaelmikowski.com.
>
> *My Assessments:*
>
> *0. This is a low-cost, high-return proposal.* People have pushed back
> against increasing the /boot partition, but I find most of the reasoning
> does not consider the true impact of a small /boot partition to the
> complete product.
>
> *1. The cost to provide the space needed is minimal for almost all FDE
> systems. *So why not just do it? Of course, there is more work to be done
> for the rest of the product (guide users on kernel selection; improve the
> kernel cleaning logic), but this is an important component that is required
> in any complete solution. It would provide substantial relief to many users
> immediately.
>
> *2. An overfull /boot partition is catastrophic for many users.* Many
> cannot recover their system because they don't have a rescue disk or
> knowledge of ext4, chroot, LUKS, lvm, cryptesetup, package management, and
> more. These people either reload the OS or give up.
>
> *3. This happens frequently, even when everything works as designed*, and
> even when just using a single kernel flavor. While on IRC yesterday
> discussing this, 3 unsolicited individuals stepped in to comment about how
> this is needed. And those are advanced developers who know better. Search
> for 'ubuntu full /boot partition' and check out how frequently this
> continues to occur.
>
> *4. There are many use cases where multiple kernel flavors will occur*,
> 

Re: /boot disk partition size

[eeickmeyer]

Hi Everyone,

Just forwarding this since it's still getting moderated. I talked to
Mike and he didn't mean for it to sound snarky in any way in the 3rd
paragraph with the reference to car safety belts, and he wanted to
apologize.

I'd very much like to encourage participation in this thread,
especially from those on the Foundations Team that requested this
thread in the first place.

Thanks!
Erich
--
Erich Eickmeyer
Project Leader, Ubuntu Studio
Member, Ubuntu Community Council

On Wed, 2022-02-16 at 11:26 -0800, Michael Mikowski wrote:
> 
> Hi Everyone:
> 
> Aaron: On an encrypted Ubuntu system, the EFI partition mounted on
> /boot/efi is 512M too. But here we are talking about the /boot
> partition which is distinct. It is typically 705M and formatted with
> ext4, although subsequent released will likely be larger. The
> question here is by how much.
> 
> We have offered this suggestion as means  to move a proven solution
> component upstream for the benefit of everyone using 22.04. We
> implemented an enlarged /boot partition and improved kernel
> management and cleaning over a year ago because our customers needed
> it. See the support cost calculations below about why we moved
> quickly when this became an issue. We expected to see much more of if
> we didn't take action. The problem was made more urgent by a
> packagekit bug.
> 
> Using an additional 0.5% (1.295G costing < $0.25) of a 250G NVMe
> drive is a very small price to pay to help avoid a catastrophic
> failure mode from which most users cannot recover. And users *do*
> install multiple kernels images for many legitimate and exploratory
> reasons, and this *does* cause full disks on the default partition
> size. If people are doing this, and the OS allows it (and it should),
> then how can it not be a supported use case? Should we remove safety
> belts from cars because we arbitrarily decide that braking hard is
> not a supported use case? Where are Ubuntu's supported use cases,
> DFMEAs, and KPCs available for review?
> 
> Compare the cost of $0.25 disk space versus the cost helping one
> novice user recover an overfull /boot partition. This can easily
> exceed $200 in direct support and lost productivity.  That's 800
> *times* more expensive. Worse, if the user can't find help expert-
> level assistance, they are highly motivated to abandon Ubuntu
> altogether.
> 
> If Ubuntu is for everyone, then we should be catering to the vast
> majority to whom the $0.25 of disk space doesn't matter. These users
> also often lack the deep skill set to recover from this failure mode.
> Conversely, the tiny fraction who want to optimize /boot size are
> those best equipped to do so - they are typically on a much smaller
> disk and are creating custom installations for things like embedded
> or edge systems. The installer is no impediment for these
> professionals.
> 
> We could continue discussing minutia about why this may or may not be
> a good idea, however, I believe the cost / benefit analysis above is
> highly convincing. We are confident from experience that a 2G /boot
> partition is a very good choice that works with best practice for all
> common and readable use cases. A quick web search also shows this
> size is frequently recommended by other experienced desktop Linux
> users.
> 
> I hope you find this useful.
> 
> Sincerely, Mike
> 
> 
> On Tue, Feb 15, 2022, 19:59 athoneycutt
>  wrote:
> > I second this as we are at the point where this size increase won't
> > affect most folks. Pop!_OS uses a 512MB /boot (/boot/efi) partition
> > which works well for dual boot setups as well. 
> > 
> > 
> > Sent from ProtonMail mobile
> > 
> > 
> > 
> >  Original Message 
> > On Feb 14, 2022, 03:45, Michael Mikowski < mmikow...@kfocus.org>
> > wrote:
> > > 
> > > Hi Everyone: 
> > > Members of the ubuntu-meeting asked me to clarify the issue with
> > > /boot. You can see the bugs filed at
> > > https://launchpad.net/bugs/1959971 and
> > > https://launchpad.net/bugs/1960089.
> > > Personal Background:
> > > I currently lead the Kubuntu Focus engineering efforts, and have
> > > been a professional product engineer specializing in mission-
> > > critical HP/HA/HV systems for decades. You can see a overview of
> > > my experience at https://michaelmikowski.com. 
> > > 
> > > My Assessments:
> > > 
> > > 0. This is a low-cost, high-return proposal. People have pushed
> > > back against increasing the /boot partition, but I find most of
> > > the reasoning does not consider the true impact of a small /boot
> > > partition to the complete product. 
> > > 
> > > 1. The cost to provide the space needed is minimal for almost all
> > > FDE systems. So why not just do it? Of course, there is more work
> > > to be done for the rest of the product (guide users on kernel
> > > selection; improve the kernel cleaning logic), but this is an
> > > important component that is required in any complete solution. It
> > > would provide substantial relief to many 

Re: /boot disk partition size

[Brian Murray]

On Tue, Feb 15, 2022 at 06:36:09PM +, Ian Bruntlett wrote:
> Hi,
> 
> Quick question.
> I'm using an install of Ubuntu 20.04.3 LTS x86_64 with one partition for /
> and another partition for swap. All my /boot files are in a subdirectory of
> my / partition.
> 
> My drive is 1TB in size.
> 
> Am I doing something wrong?

A wrong install of Ubuntu is a subjective thing! Having said that the
default installation of Ubuntu does not create a separate /boot
partition so no you didn't miss anything. A separate /boot partition is
used if you perform an install with encryption. But your question brings
up a good point - the majority of users installing Ubuntu will not have
a separate /boot partition.

--
Brian Murray

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: /boot disk partition size

[Brian Murray]

On Mon, Feb 14, 2022 at 01:20:36PM -0800, eeickme...@ubuntu.com wrote:
> Hi all,
> 
> I thought I'd provide some context on this, reiterating what I wrote on
> Friday.
> 
> Summary: We would like to see some discussion about increasing the
> /boot partition for automatic partitioning on LVM/Encrypted systems to
> 2GB for 22.04. We believe this would be a relatively inexpensive
> change, and would be much better than the 1.5GB set to be a part of
> 20.04.4.
> 
> Background:
> 
> I'm part of the team that makes the Kubuntu Focus line of laptop
> computers. On encrypted LVM machines, using the default disk setup, we
> have found the /boot partition is way too small.  We discovered that
> the /boot partition gets overfilled while apt is installing updates.
> Part of this problem is that Kubuntu uses Discover to do updating (not
> update-manager like other flavors excluding Ubuntu Studio which also
> uses Discover), which uses PackageKit as a backend.
> 
> PackageKit has a bug in which it individually installs packages when
> updating, causing those packages to be marked as manually installed and
> therefore not letting 'apt autoremove' do its job. As you can imagine,
> this breaks the old kernel removal process that update-manager does on
> other flavors. This was recently fixed upstream[1] in PackageKit and
> hopefully will release in time to beat the Feature Freeze / Debian
> Import Freeze. 

From what you've said this sounds like something worth carrying as a
debdiff in packagekit rather than waiting for Debian to fix it.
Additionally, it'd make sense to me to get this sorted out for stable
releases of Ubuntu too. Is there any work being done on that front?

> We have created a script to mitigate this that automatically cleans
> kernels. We also increased the /boot partition size on our OEM-
> installed systems. We want to move our findings upstream to the OS.
> With the default 768MB /boot partition, sometimes even then three
> kernels would be installed and overfill with a new kernel update before
> it could be cleaned, especially if the user has more than one kernel
> flavor. A use case for two kernel flavors would be using 'lowlatency'
> for lower hardware latency, and using 'generic' for power-saving
> attributes when compared to 'lowlatency'. This would be especially
> advantageous on desktop-replacement-style laptop computers, such as the
> Kubuntu Focus M2[2].
> 
> Bear in mind also that the Nvidia drivers make the initrd for any given
> kernel significantly larger than it would be otherwise. With this in
> mind, we filed LP: #1960089[3], not knowing that Brian Murray had
> already patched and uploaded a larger boot partition prior [4]. Seeing
> Brian's patch, we don't believe it still makes the /boot partition
> large enough, especially considering the Nvidia drivers.

My changes to partman-auto specifically took into an initramfs with the
Nvidia drivers and that is documented in
http://launchpad.net/bugs/1959971.

Cheers,
--
Brian Murray

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: /boot disk partition size

[Brian Murray]

On Thu, Feb 10, 2022 at 06:38:54PM -0800, Michael Mikowski wrote:
> Hi Everyone:
> 
> Members of the ubuntu-meeting asked me to clarify the issue with /boot. You
> can see the bugs filed at https://launchpad.net/bugs/1959971 and
> https://launchpad.net/bugs/1960089.
> *Personal Background:*
> I currently lead the Kubuntu Focus  engineering
> efforts, and have been a professional product engineer specializing in
> mission-critical HP/HA/HV systems for decades. You can see a overview of my
> experience at https://michaelmikowski.com.
> 
> *My Assessments:*
> 
> *0. This is a low-cost, high-return proposal.* People have pushed back
> against increasing the /boot partition, but I find most of the reasoning
> does not consider the true impact of a small /boot partition to the
> complete product.

Increasing the size of the /boot partition comes at the cost of
decreasing the size of the / partition though and I suspect there are
people just as passionate about the usable space they have available for
their operating system and files as your are about free space in /boot.
I'm not disputing the impact of a small /boot partition but there is a
balancing act for us to provide a default install that is best for the
majority of Ubuntu users.

> *1. The cost to provide the space needed is minimal for almost all FDE
> systems. *So why not just do it? Of course, there is more work to be done
> for the rest of the product (guide users on kernel selection; improve the
> kernel cleaning logic), but this is an important component that is required
> in any complete solution. It would provide substantial relief to many users
> immediately.

The reason we just don't do it is because it requires careful
consideration as I've indicated above. What number of users have you
encountered who have filled their /boot partition? What release of
Ubuntu had they installed and what was the size of their /boot
partition? (The size of the /boot partition created by the installer has
changed a number of times.)

> *2. An overfull /boot partition is catastrophic for many users.* Many
> cannot recover their system because they don't have a rescue disk or
> knowledge of ext4, chroot, LUKS, lvm, cryptesetup, package management, and
> more. These people either reload the OS or give up.

Agreed.

> *3. This happens frequently, even when everything works as designed*, and
> even when just using a single kernel flavor.

I've seen references to bugs about packagekit and unattended-upgrades
and their handling of the quantity of kernels and space management in
/boot. Those are bugs that should be resolved in those packages rather
than increasing the size of /boot (and taking away from /) to compensate
for those bugs.

> While on IRC yesterday discussing this, 3 unsolicited individuals
> stepped in to comment about how this is needed. And those are advanced
> developers who know better. Search for 'ubuntu full /boot partition'
> and check out how frequently this continues to occur.

For reference the conversation where 2 unsolicited individuals commented
is here:

https://irclogs.ubuntu.com/2022/02/08/%23ubuntu-devel.html

Additionally, both of those developers have been able to install Ubuntu
in a way that meets their specific needs.

> *4. There are many use cases where multiple kernel flavors will occur*,
> such as when the users switches from HWE to OEM to address hardware issues,
> or they install lowlatency for studio work. When this happens, the current
> size boot partition is highly likely to fill. This can corrupt the system
> and require a rescue disk for recovery.

I'd really like more information on the use case where multiple kernel
flavors will be installed. You mention switching from HWE to OEM, given
that this is a switch wouldn't the HWE kernel be removed and then you
are left with just one flavor of kernel?

I understand the studio use case to be having a "regular" kernel
flavor installed and a low latency flavor one installed so that one can
boot between one or the other depending on their work load. Is that a
correct understanding?

> Check out the DFMEA which is used to assess the severity of a failure mode:
> https://bugs.launchpad.net/ubuntu/+source/partman-auto/+bug/1959971/comments/7
> 
> Finally, I assure you the calculations for kernel boot-file-set sizes (180M
> with Nvidia GPU) are correct for 20.04, and will be correct for 22.04
> unless the compression technique has been changed.

Instead of assuring use your calculations are correct it would have
helped if you were to show your math. However, I think I've sorted it out.

My Jammy installation of Ubuntu (nvidia and cryptfs) has the following
in /boot:

165M initrd.img-5.15.0-18-generic
  6M System.map-5.15.0-18-generic
 11M vmlinuz-5.15.0-18-generic

Adding those together we have 182M. These are the sizes I used when I
updated partman-auto for http://launchpad.net/bugs/1959971 which is
close to what you are looking for but with only one kernel flavor

Re: /boot disk partition size

[eeickmeyer]

Hi all,

I thought I'd provide some context on this, reiterating what I wrote on
Friday.

Summary: We would like to see some discussion about increasing the
/boot partition for automatic partitioning on LVM/Encrypted systems to
2GB for 22.04. We believe this would be a relatively inexpensive
change, and would be much better than the 1.5GB set to be a part of
20.04.4.

Background:

I'm part of the team that makes the Kubuntu Focus line of laptop
computers. On encrypted LVM machines, using the default disk setup, we
have found the /boot partition is way too small.  We discovered that
the /boot partition gets overfilled while apt is installing updates.
Part of this problem is that Kubuntu uses Discover to do updating (not
update-manager like other flavors excluding Ubuntu Studio which also
uses Discover), which uses PackageKit as a backend.

PackageKit has a bug in which it individually installs packages when
updating, causing those packages to be marked as manually installed and
therefore not letting 'apt autoremove' do its job. As you can imagine,
this breaks the old kernel removal process that update-manager does on
other flavors. This was recently fixed upstream[1] in PackageKit and
hopefully will release in time to beat the Feature Freeze / Debian
Import Freeze. 

We have created a script to mitigate this that automatically cleans
kernels. We also increased the /boot partition size on our OEM-
installed systems. We want to move our findings upstream to the OS.
With the default 768MB /boot partition, sometimes even then three
kernels would be installed and overfill with a new kernel update before
it could be cleaned, especially if the user has more than one kernel
flavor. A use case for two kernel flavors would be using 'lowlatency'
for lower hardware latency, and using 'generic' for power-saving
attributes when compared to 'lowlatency'. This would be especially
advantageous on desktop-replacement-style laptop computers, such as the
Kubuntu Focus M2[2].

Bear in mind also that the Nvidia drivers make the initrd for any given
kernel significantly larger than it would be otherwise. With this in
mind, we filed LP: #1960089[3], not knowing that Brian Murray had
already patched and uploaded a larger boot partition prior [4]. Seeing
Brian's patch, we don't believe it still makes the /boot partition
large enough, especially considering the Nvidia drivers.

>From LP: #1960089 (sizes here reported in MiB and GiB):

   Summary:
   
   We propose to increase the LVM /boot partition to 2.0 GB. This provides
   the space needed so advanced users can use best practice to manage up
   to 3 kernel flavors. The current /boot partition on 20.04 and 22.04 is
   limited to just 705M, which allows only 3 concurrent kernels before
   filling and sometimes locking the system (each image set takes 180M
   total; 4 x 180 = 720M > 705M).
   
   Reasoning:
   
   Best practice recommends users keep at least two version of each kernel
   flavor in the /boot directory. If a user has 3 kernel flavors installed
   (e.g. oem, generic-hwe, and lowlatency-hwe), then one needs to reserve
   room for 2 x 3 = 6 kernels.
   The system needs the headroom of at least two additional kernels during
   any automated clean-up process due to package removal scheduling. I
   propose to also reserve room for 2 additional kernels as a safety
   measure. Thus the total recommend available space should accommodate 10
   kernels.
   
   Each kernel file set takes up 180M in the /boot partition when used
   with Nvidia driver modules. These files include initrd.img, system.map,
   and vmlinuz. With future kernel and module growth, this may surpass
   200M soon. Therefore, we suggest planning for 200M for each kernel.
   
   We therefore request a total LVM /boot partition size of 10 image x
   200M = 2.0 GB.
   
   Other Considerations:
   
   When unattended-upgrades works correctly (which does not yet employ
   best practice), we have seen users with just a single kernel flavor
   over-fill their /boot partitions. This is because unattended-upgrades
   can retain up to 4 kernels, while the /boot partition is only large
   enough for 3. I am currently working with others to improve the
   unattended-upgrades algorithm to use best practice.
   
   The installer could allow users to resize the /boot partition during
   installation. In this case, we highly recommend a 2.0 GB default for
   the reasons outlined above.

After discovering LP: #1959971, Mike commented:

   Hi Łukasz and Brian: I have been doing quite a bit of research in this
   area recently, and also am advocating to get kernel management and
   cleanup improved, especially for users who must have separate /boot
   partitions. This means professional users who are required to have full
   disk encryption according to company IT policies.
   
   Using best practice to manage 3 kernel packages (e.g. oem, lowlatency-
   hwe, generic-hwe) results in the need to maintain 6 kernel file sets