Re: New tzdata release for ubuntu 20.04 lts (and higher)

[Dimitri John Ledkov]

tzdata should be updated in legacy releases via Ubuntu Pro, which you
should be able to gain access to via ubuntu.com/pro

On Mon, 19 Feb 2024, 17:19 Dauren Sarsenov,  wrote:

> Hi, guys.
>
> Good news, I can see that an updated version of tzdata has been released
> for Ubuntu 24.04 LTS.
>
> Can you suggest, how can i backport it to Ubuntu 18.04, 16.04 (as they are
> retired, i guess they will never see the update otherwise)?
>
>
>
> Best
>
> Dauren
>
> > On 8 Feb 2024, at 16:45, Adrien Nader  wrote:
> >
> > Hi,
> >
> > On Sat, Feb 03, 2024, Dauren Sarsenov wrote:
> >> Hi, guys.
> >>
> >> I wonder, how much time does it usually take to release an update
> version of tzdata package? Asking, because there is a new upstream build (
> https://www.iana.org/time-zones), and rather critical one to us, who live
> in Kazakhstan. The clock is to be moved 1 hour back on the 1st of March
> 2024. The new build of tzdata contains the appropriate rule, thus it is
> highly anticipated currently.
> >
> > On the Ubuntu side, we are aware of the update but the person usually
> > doing the updates has not been able to do it. This is something that we
> > are planning to address in the next two weeks (regardless of their
> > availability) although I cannot fully guarantee it at the moment.
> >
> > Best,
> >
> > --
> > Adrien
>
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Bumping apt RSA key length requirements to 3072-bit (2048 w/ warning) for 24.04

[Dimitri John Ledkov]

Hi,

On Thu, 18 Jan 2024 at 18:02, Julian Andres Klode
 wrote:
>
> Hi,
>
> we just noticed again that we are still trusting 1024R keys for
> signing repositories in APT, arguably because we do not have a
> means to tell gpgv the minimum key size.
>
> While the upstream bug[0] is being worked on,
> I have written a hack[1] that - if APT_SIGNING_REQUIREMENTS_HACK
> environment variable is set - makes gpgv error out on keys smaller
> than 2048R and warn on keys smaller than 3072R (following the
> current OpenPGP draft size length requirements, 3072 is a SHOULD,
> 2048 a MUST).

Separately we also care about NIST FIPS recommendations, for RSA it is
2048 until 2030, and with an option to bump it to 3072 from 2030.
Thus one can scope this as 2048 until 2030, and 3072 from 2030 already.

Also, given the performance penalty involved we should also consider
to support and accept ECC - as per NIST Ed25519 (more popular in new
deployments) is now approved as is P-256 (wider compat and support)
both of which are post-2030 acceptable to NIST, the wider internet
(TLS authorities) and us.

>
> I have also written code in APT to actually parse GPG error and
> warning status messages, and set the environment variable.[2]
>
> Sadly shipping this in 24.04 means that PPAs owned by user
> accounts created prior to 2014-03-11[3] until the key rotation
> mechanism(s) [4][5] have been implemented.
>

I do wonder how many active old PPA owners remain in action.

And if we can reset per-series signing keys on all of those for any
new PPAs, and noble series (meaning single signe, new key for noble+).

I have personally created a new team for myself, only added myself to
be a member of said team, to gain access to PPAs signed with 4k RSA
key, as I can no longer use my own ppas. I guess I should ask to
delete them all, and request removal of the signing key to gain back
personal PPAs with 4k signing key.

> However given that (I've been informed) ~800 bits were already cracked about 
> 5 years
> ago, and we are planning to support 24.04 for 12 years, I believe
> that this is necessary and it's better to take the pain now then
> do an SRU to disable 1024R keys on existing systems.
>
> This is more painful than the digest transition because we have
> reason to believe that 1024R keys are potentially unsafe *now*
> and we need to stop trusting them, whereas when we deprecated
> MD5 and SHA1 we were able to have a deprecation period of a
> stable release.
>
> [0] https://dev.gnupg.org/T6946
> [1] https://gist.github.com/julian-klode/fbc56278cd0bdcd305f825479b094fad
> [2] https://salsa.debian.org/apt-team/apt/-/merge_requests/322
> [3] https://code.launchpad.net/~wgrant/launchpad/4096r-ppa-keys/+merge/210336
> [4] https://bugs.launchpad.net/launchpad/+bug/1331914
> [5] https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1461834
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer  i speak de, en
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
Dimitri

Sent from Ubuntu Pro
https://ubuntu.com/pro

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Request for Update: NVIDIA Driver Version 535.146.02 for APT

[Dimitri John Ledkov]

Hi,

This version of this driver has been available in the proposed pocket
since 2023-12-15 and is undergoing testing. It will be released to
updates once regression and certification testing is completed. Note
we do not release routine updates like these over the winter holidays
period which was between 16th of December and today.

On Sun, 31 Dec 2023 at 17:29, Владислав  wrote:
>
> Dear Ubuntu APT Package Manager Maintainers,
>
> I hope this message finds you well. I am writing to bring to your attention 
> the recent release of the NVIDIA display driver version 535.146.02, 
> specifically designed for Linux x64 (AMD64/EM64T) systems on December 7, 2023.
>
> As an Ubuntu user and part of the community that greatly values the seamless 
> functionality and performance of our systems, I wish to highlight the 
> importance of updating the NVIDIA package within the Ubuntu APT repositories 
> to include this latest driver version.
>
> The advancements and improvements made in the latest NVIDIA driver iteration, 
> version 535.146.02, offer enhanced features, stability, and compatibility 
> with a wide range of hardware configurations and the most important they 
> fixed some bugs related to OS that uses several GPUs, so I had this error and 
> the driver version (535.129.03) didn't work well when system has several 
> GPUs. These updates are crucial for optimizing the performance and 
> compatibility of NVIDIA GPUs with Ubuntu systems, ensuring a better user 
> experience for all Ubuntu users who rely on NVIDIA graphics hardware.
>
> Looking forward to seeing the latest NVIDIA driver version integrated into 
> the Ubuntu repositories soon.
>
> Best regards,
>
> Vladyslav Matsapura.
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss



-- 
Dimitri

Sent from Ubuntu Pro
https://ubuntu.com/pro

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Choice of the openssl version for 23.10 and 24.04

[Dimitri John Ledkov]

On Mon, 4 Dec 2023 at 12:34, Adrien Nader  wrote:
>
> (stripping the quotes a bit)
>
> On Mon, Dec 04, 2023, Dimitri John Ledkov wrote:
> > On Mon, 4 Dec 2023 at 09:28, Adrien Nader  wrote:
> > > The issue is that we do not know when will be the next openssl LTS. We
> > That's irrelevant. Once Ubuntu release goes stable, we no longer apply
> > full upstream point releases, and create our own stream of security
> > and stable fixes anyway.
> > Our timelines of support are also longer than either shorter or longer
> > upstream support timelines.
>
> The point is that security updates require work and using a version that
> no other distribution uses (which will always be the case unless other
> distribution release dates align with Ubuntu's) prevents sharing the
> work with other distributions. This was stated earlier (months ago) in
> this thread. I don't know how that played out in hindsight. That would
> be an interesting thing to know.
>
> Even if we're not on the same point version as other distributions,
> there is still a lot of common things, especially for typical security
> patches, and lots of testing in common. I don't think the patches are
> very difficult to port across versions (except the ones which porting is
> horribly painful) but QA and verification are another matter. Keep in
> mind that 3.0 was a big release with many architectural changes and this
> has continued in 3.1 and 3.2. There are certainly fewer and fewer
> changes but I don't think I would call these changes uncommon yet.
>
> Finally, Ubuntu is not the only distribution with longer support than
> openssl's five years for LTS: there can still be cross-distro
> cooperation after these years, and it will actually be even more
> valuable.
>
> > > can safely bet there will be one before 26.04 and update to the most
> > > recent version for 24.10 (since by 26.04, the current LTS won't be
> > > supported anymore). However we can't really bet there will be one by
> > > 24.04 and even if there were, there probably wouldn't be a new one in
> > > time by 26.04.
> > >
> > > What you are asking for is equivalent to not using openssl LTS releases
> > > for Ubuntu LTS releases.
> >
> > Yes.
> >
> > > There are many more non-LTS releases so we're
> > > more likely to end up with a non-LTS version.
> > >
> > > Openssl time-based releases are very very recent.
> >
> > We did ship the first KDE time-based release when it came out for the
> > first time.
>
> KDE, especially in Ubuntu, is far less risky than openssl. It is also
> updated far more easily.
>
> I've been trying to make an SRU for openssl for several months now and
> it hasn't landed yet. I'm not blaming anyone here: there are tons of
> reasons for that (including the fact that I don't have the bandwidth to
> re-spin something at the moment).
>
> The fact is that today we're not able to update openssl for anything
> except security updates. In other words, no matter what the openssl
> maintainer puts in a release, that maintainer won't have anything to do
> with it after the release: only the security team will. That makes me
> completely uncomfortable with using a release without some kind of
> agreement with the security team. Using the most recent version would
> make my life easier but I don't think it's the right trade-off here.
>
> By the way, I don't know how that would play with FIPS: I'm not sure it
> would be manageable.
>
> > > This was announced
> > > late August and only one version was released under this model so far.
> > > It wasn't even on time. The delay was fairly small, especially for a
> > > first version using this model (and a change mid-cycle). I think they
> > > need a bit more time. Will they be late again? Will they continue this
> > > scheme? What else? I don't expect openssl upstream can answer these;
> > > they don't have enough hindsight yet.
> > >
> > > We talked about creating a new "openssl" package that is whatever the
> > > most recent version is (in universe, and probably with no ESM-guarantee
> > > attached somehow). This might need a bit of fiddling with packaging
> > > though and in any case, I've had absolutely no time to do that so far.
> > >
> > > Would such a package fit the needs you see?
> >
> > Only one version, the latest openssl, in every Ubuntu release, going 
> > forward.
> >
> > Shipping two openssl in the archive is extremely difficult and ends up
> > being unusable. Devel packages previously have not been coinstallable
>

Re: Choice of the openssl version for 23.10 and 24.04

[Dimitri John Ledkov]

On Mon, 4 Dec 2023 at 12:34, Adrien Nader  wrote:
>
> (stripping the quotes a bit)
>
> On Mon, Dec 04, 2023, Dimitri John Ledkov wrote:
> > On Mon, 4 Dec 2023 at 09:28, Adrien Nader  wrote:
> > > The issue is that we do not know when will be the next openssl LTS. We
> > That's irrelevant. Once Ubuntu release goes stable, we no longer apply
> > full upstream point releases, and create our own stream of security
> > and stable fixes anyway.
> > Our timelines of support are also longer than either shorter or longer
> > upstream support timelines.
>
> The point is that security updates require work and using a version that
> no other distribution uses (which will always be the case unless other
> distribution release dates align with Ubuntu's) prevents sharing the
> work with other distributions. This was stated earlier (months ago) in
> this thread. I don't know how that played out in hindsight. That would
> be an interesting thing to know.
>
> Even if we're not on the same point version as other distributions,
> there is still a lot of common things, especially for typical security
> patches, and lots of testing in common. I don't think the patches are
> very difficult to port across versions (except the ones which porting is
> horribly painful) but QA and verification are another matter. Keep in
> mind that 3.0 was a big release with many architectural changes and this
> has continued in 3.1 and 3.2. There are certainly fewer and fewer
> changes but I don't think I would call these changes uncommon yet.
>
> Finally, Ubuntu is not the only distribution with longer support than
> openssl's five years for LTS: there can still be cross-distro
> cooperation after these years, and it will actually be even more
> valuable.
>
> > > can safely bet there will be one before 26.04 and update to the most
> > > recent version for 24.10 (since by 26.04, the current LTS won't be
> > > supported anymore). However we can't really bet there will be one by
> > > 24.04 and even if there were, there probably wouldn't be a new one in
> > > time by 26.04.
> > >
> > > What you are asking for is equivalent to not using openssl LTS releases
> > > for Ubuntu LTS releases.
> >
> > Yes.
> >
> > > There are many more non-LTS releases so we're
> > > more likely to end up with a non-LTS version.
> > >
> > > Openssl time-based releases are very very recent.
> >
> > We did ship the first KDE time-based release when it came out for the
> > first time.
>
> KDE, especially in Ubuntu, is far less risky than openssl. It is also
> updated far more easily.
>
> I've been trying to make an SRU for openssl for several months now and
> it hasn't landed yet. I'm not blaming anyone here: there are tons of
> reasons for that (including the fact that I don't have the bandwidth to
> re-spin something at the moment).
>
> The fact is that today we're not able to update openssl for anything
> except security updates. In other words, no matter what the openssl
> maintainer puts in a release, that maintainer won't have anything to do
> with it after the release: only the security team will. That makes me
> completely uncomfortable with using a release without some kind of
> agreement with the security team. Using the most recent version would
> make my life easier but I don't think it's the right trade-off here.
>
> By the way, I don't know how that would play with FIPS: I'm not sure it
> would be manageable.

We do our own FIPS certification, and we do it based on whichever
version we ship, with staff & cert team working on that. We have
certified OpenSSL as needed for our purposes, irrespective of the
upstream FIPS efforts, due to the ways we make api & abi compatible
drop-in replacement without any need to change any runtime configs.

So far FIPS team will certify whatever noble will ship.

>
> > > This was announced
> > > late August and only one version was released under this model so far.
> > > It wasn't even on time. The delay was fairly small, especially for a
> > > first version using this model (and a change mid-cycle). I think they
> > > need a bit more time. Will they be late again? Will they continue this
> > > scheme? What else? I don't expect openssl upstream can answer these;
> > > they don't have enough hindsight yet.
> > >
> > > We talked about creating a new "openssl" package that is whatever the
> > > most recent version is (in universe, and probably with no ESM-guarantee
> > > attached somehow). This might need a bit of fiddling with packaging
> > > though and in any case, I've had absolu

Re: Choice of the openssl version for 23.10 and 24.04

[Dimitri John Ledkov]

On Mon, 4 Dec 2023 at 09:28, Adrien Nader  wrote:
>
> On Mon, Dec 04, 2023, Dimitri John Ledkov wrote:
> > Hi,
> >
> > On Fri, 20 Oct 2023 at 15:35, Adrien Nader  wrote:
> > >
> > > On Fri, Oct 20, 2023, Adrien Nader wrote:
> > > > Hi,
> > > >
> > > > A few weeks ago, openssl maintainers announced moving to a time-based
> > > > release (April and October):
> > > >
> > > > https://www.openssl.org/blog/blog/2023/09/29/OpenSSL-Update-ICMC23/
> > > >
> > > > > Key takeaway 3 : Time Based Release Policy
> > > > > We’re transitioning to time-based releases. This shift ensures
> > > > > predictability, allowing our users and developers to plan better and
> > > > > benefit from timely updates. The releases will be scheduled every
> > > > > April and October.
> > > >
> > > > Based on this and the openssl 3.0 release date, I'd expect a new LTS
> > > > version to be released (almost) in time for 26.04 but not for 24.04.
> > > >
> > > > *IF* an openssl LTS release is out in April 26.04, we might want to
> > > > track the corresponding openssl git branch during the 26.04 release in
> > > > order to be able to ship it. This is more than two years away however
> > > > and a lot can happen until then. I don't have a crystal ball
> > > > unfortunately. In any case, we'll know if the planned and the actual
> > > > release cadence and calendar match.
> > >
> > > Dimitri asked me for some more details so I dug a bit more. It's
> > > actually better explained in a better blog post from late August:
> > >
> > > https://www.openssl.org/blog/blog/2023/08/27/steps-forward
> > >
> > > > We’re also shifting how we release the OpenSSL library. We’ve adopted
> > > > a time-based release policy, with releases every April and October.
> > > > After our 3.2 release in October, our 3.3 release in April next year
> > > > will be our first time-based release, marking our initial venture into
> > > > this approach.
> > >
> > > And the release policy has been updated too:
> > >
> > > https://www.openssl.org/policies/general/release-policy.html
> > >
> > > > Planning: Continuous process, provides input to the Release Definition 
> > > > phase.
> > > > Release Definition: Defines release backlog, lasts up to 4 weeks.
> > > > Development: Execution of the release backlog, spans from 20 to 24 
> > > > weeks.
> > > > Release: Addressing issues discovered by the community in pre-releases. 
> > > > Up to 6 weeks.
> > > > Support: A support phase.
> > >
> > > If they follow their plan, we'd therefore have pre-release versions
> > > several weeks before Ubuntu releases. Of course, feature freeze
> > > concerns apply if the pre-release isn't out in time.
> > >
> > > That's all I've seen so far (OK, I didn't dig that much). We'll see very
> > > soon how that turns out in practice for the 3.2 release.
> >
> > With every other major piece of our dependencies, we have committed to
> > picking up regular time based releases which fit our schedule.
> > This includes things like GNOME, KDE, OpenStack, GCC.
> > I think we should commit to picking up the latest OpenSSL for every
> > Ubuntu release going forward.
> > 3.2 has been released, albeit in late November, and we should show
> > good will and encourage OpenSSL to stick to their new time based
> > releases.
> >
> > Can we please start the upgrade of OpenSSL to 3.2?
> >
> > And then if OpenSSL 3.3 is released in early April, pick that up as
> > well for 24.04. If the new cadence for 3.3 means May, pick it up for
> > the 24.10 instead.
>
> The issue is that we do not know when will be the next openssl LTS. We

That's irrelevant. Once Ubuntu release goes stable, we no longer apply
full upstream point releases, and create our own stream of security
and stable fixes anyway.
Our timelines of support are also longer than either shorter or longer
upstream support timelines.

> can safely bet there will be one before 26.04 and update to the most
> recent version for 24.10 (since by 26.04, the current LTS won't be
> supported anymore). However we can't really bet there will be one by
> 24.04 and even if there were, there probably wouldn't be a new one in
> time by 26.04.
>
> What you are asking for is equivalent to not using openssl LTS releases
> for Ubuntu LTS 

Re: Choice of the openssl version for 23.10 and 24.04

[Dimitri John Ledkov]

Hi,

On Fri, 20 Oct 2023 at 15:35, Adrien Nader  wrote:
>
> On Fri, Oct 20, 2023, Adrien Nader wrote:
> > Hi,
> >
> > A few weeks ago, openssl maintainers announced moving to a time-based
> > release (April and October):
> >
> > https://www.openssl.org/blog/blog/2023/09/29/OpenSSL-Update-ICMC23/
> >
> > > Key takeaway 3 : Time Based Release Policy
> > > We’re transitioning to time-based releases. This shift ensures
> > > predictability, allowing our users and developers to plan better and
> > > benefit from timely updates. The releases will be scheduled every
> > > April and October.
> >
> > Based on this and the openssl 3.0 release date, I'd expect a new LTS
> > version to be released (almost) in time for 26.04 but not for 24.04.
> >
> > *IF* an openssl LTS release is out in April 26.04, we might want to
> > track the corresponding openssl git branch during the 26.04 release in
> > order to be able to ship it. This is more than two years away however
> > and a lot can happen until then. I don't have a crystal ball
> > unfortunately. In any case, we'll know if the planned and the actual
> > release cadence and calendar match.
>
> Dimitri asked me for some more details so I dug a bit more. It's
> actually better explained in a better blog post from late August:
>
> https://www.openssl.org/blog/blog/2023/08/27/steps-forward
>
> > We’re also shifting how we release the OpenSSL library. We’ve adopted
> > a time-based release policy, with releases every April and October.
> > After our 3.2 release in October, our 3.3 release in April next year
> > will be our first time-based release, marking our initial venture into
> > this approach.
>
> And the release policy has been updated too:
>
> https://www.openssl.org/policies/general/release-policy.html
>
> > Planning: Continuous process, provides input to the Release Definition 
> > phase.
> > Release Definition: Defines release backlog, lasts up to 4 weeks.
> > Development: Execution of the release backlog, spans from 20 to 24 weeks.
> > Release: Addressing issues discovered by the community in pre-releases. Up 
> > to 6 weeks.
> > Support: A support phase.
>
> If they follow their plan, we'd therefore have pre-release versions
> several weeks before Ubuntu releases. Of course, feature freeze
> concerns apply if the pre-release isn't out in time.
>
> That's all I've seen so far (OK, I didn't dig that much). We'll see very
> soon how that turns out in practice for the 3.2 release.

With every other major piece of our dependencies, we have committed to
picking up regular time based releases which fit our schedule.
This includes things like GNOME, KDE, OpenStack, GCC.
I think we should commit to picking up the latest OpenSSL for every
Ubuntu release going forward.
3.2 has been released, albeit in late November, and we should show
good will and encourage OpenSSL to stick to their new time based
releases.

Can we please start the upgrade of OpenSSL to 3.2?

And then if OpenSSL 3.3 is released in early April, pick that up as
well for 24.04. If the new cadence for 3.3 means May, pick it up for
the 24.10 instead.

-- 
Dimitri

Sent from Ubuntu Pro
https://ubuntu.com/pro

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Intermittent superblock checksum mismatch during resize2fs

[Dimitri John Ledkov]

This was causing us headaches like since forever. Our reproducers were very
intermittent to catch it. I think we might want to backport this everywhere
we can.

On Thu, 5 Oct 2023, 17:50 Krister Johansen,  wrote:

> Hi,
> My team runs Ubuntu 20.04 on EC2.  We use the cloud images that
> Canonical and AWS publish for Ubuntu as our base image.  As part of
> the first boot of one of these images, cloud-init runs resiz2fs in order
> to make the root filesystem match the size of the root volume on which the
> instance is provisioned.
>
> A few times a week, this would fail for us, which results in instances
> that have an unexpectedly small root filesystem.  We were able to build
> a reproducer for the problem, debug it, and get it fixed upstream.
> There's a patch in the main branch of e2fsprogs with the fix now.
>
> We've had to fork the Ubuntu version of e2fsprogs to carry this patch,
> but from our analysis it ought to impact (potentially) any customer
> running on EC2.  Would Ubuntu be willing to backport this patch to the
> versions for which they build EC2 AMIs?  (We're specifically interested
> in any LTS Ubuntu releases).
>
> I logged a ticket for this on launchpad here (it includes the
> reproducer):
>
> https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/2036467
>
> The patch itself is a 2-line change:
>
>
> https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=43a498e938887956f393b5e45ea6ac79cc5f4b84
>
> The thread where this was discussed with the upstream maintainer is
> here:
>
> https://lore.kernel.org/linux-ext4/20230609042239.ga1436...@mit.edu/
>
> Thanks very much,
>
> -K
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Amendment: Update on reducing initramfs size and speed up

[Dimitri John Ledkov]

On Mon, 31 Jul 2023 at 20:41, Benjamin Drung  wrote:
>
> On Thu, 2023-07-27 at 11:51 +1200, Michael Hudson-Doyle wrote:
> >
> >
> > On Thu, 27 Jul 2023 at 09:21, Benjamin Drung 
> > wrote:
> > > On Wed, 2023-07-26 at 17:53 +0200, Benjamin Drung wrote:
> > > > Hi all,
> > > >
> > > > A few weeks ago, I posted an idea how to reduce the initramfs size
> > > > and
> > > > speed up the generation:
> > > >
> > > > https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html
> > > >
> > > > This post sparked a lively discussion. The initial idea was
> > > > ditched for
> > > > a better solution: mkinitramfs will put all compressed files
> > > > (kernel
> > > > modules and firmware files) into a cpio archive that is not
> > > > compressed
> > > > (because compressing compressed files makes no sense). All other
> > > > files
> > > > will be added to a cpio archive that gets compressed. As next
> > > > steps, the
> > > > kernel modules and firmware files need to be shipped compressed.
> > > >
> > > > After several iterations for the implementation and review by
> > > > Daves
> > > > Jones, I just uploaded initramfs-tools 0.142ubuntu8 to mantic that
> > > > puts
> > > > compressed kernel modules and firmware files in an uncompressed
> > > > cpio
> > > > (https://launchpad.net/bugs/2028567).
> > > >
> > > > I created/updated the follow-up tickets and added my patches to
> > > > them:
> > > >
> > > > Ship kernel modules Zstd compressed
> > > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568
> > > >
> > > > compress firmware in /lib/firmware
> > > > https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1942260
> > > >
> > > > And without further ado, here come the benchmark results:
> > > >
> > > > The benchmarks were done either on an AMD Ryzen 7 5700G with
> > > > schroot and
> > > > overlay on tmpfs or on the hardware mentioned. All tests were
> > > > running
> > > > the latest Ubuntu mantic development release.
> > > >
> > > > * minimal: schroot with linux-image-generic initramfs-tools zstd
> > > > * full: minimal + busybox-initramfs cryptsetup-initramfs
> > > >isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme-
> > > > spinner
> > > > * nvidia: full + linux-headers-generic nvidia-driver-525
> > > > * nvidia fw: nvidia + compressed /lib/firmware/nvidia/525.125.06/
> > > > * VisionFive 2: VisionFive 2 RISC-V board
> > > > * RPi Zero 2: Raspberry Pi Zero 2 ARM board (running armhf)
> > > >
> > > > "next" means using kernel/firmware/initramfs from ppa:bdrung/ppa
> > > > i.e.
> > > > * initramfs-tools 0.142ubuntu7bd4
> > > > * linux 6.3.0-7.7bd2
> > > > * linux-firmware 20230629.gitee91452d-0ubuntu1bd1
> > > >
> > > > > || build   | size   | uncompressed
> > > > > size  |
> > > > > | test   | time| in bytes  | in MiB | in bytes  | in
> > > > > MiB |
> > > > > ||-|---||---
> > > > > -|
> > > > > | minimal| 4.30 s  |  66701818 |  63.6  | 224087608 |
> > > > > 213.7  |
> > > > > | minimal next   | 4.54 s  |  59935186 |  57.2  |  67738810 |
> > > > > 64.6  |
> > > > > | full   | 7.15 s  | 118007038 | 112.5  | 387976843 |
> > > > > 370.0  |
> > > > > | full next  | 7.29 s  | 106937908 | 102.0  | 128610985 |
> > > > > 122.7  |
> > > > > | nvidia | 7.04 s  | 209200523 | 199.5  | 513554279 |
> > > > > 489.8  |
> > > > > | nvidia next| 7.21 s  | 195246287 | 186.2  | 235288174 |
> > > > > 224.4  |
> > > > > | nvidia fw next | 7.16 s  | 191329102 | 182.5  | 213078234 |
> > > > > 203.2  |
> > > > > | VisionFive 2   | 142.9 s | 121895035 | 116.2  | 411160836 |
> > > > > 392.1  |
> > > > > | VF 2 next  | 126.7 s | 111651453 | 106.5  | 134120804 |
> > > > > 127.9  |
> > > > > | RPi Zero 2 | 109.5 s |  39803044 |  40.0  |  69592789 |
> > > > > 66.4  |
> > > > > | RPi Zero 2 ²   | 103.5 s |  39804499 |  40.0  |  69592789 |
> > > > > 66.4  |
> > > > > | RPi Zero 2 next| 101.2 s |  31463352 |  30.0  |  41145762 |
> > > > > 39.2  |
> > > >
> > > > ² Updated initramfs-tools (but no compressed modules or firmware)
> > > >
> > > > The build time was averaged over five runs.
> > > >
> > > > > | improvement  | build time | size   | uncompressed size |
> > > > > |--|||---|
> > > > > | minimal  |  105.6 %   | 89.9 % |  30.2 %   |
> > > > > | full |  102.0 %   | 90.6 % |  33.1 %   |
> > > > > | nvidia   |  101.7 %   | 91.5 % |  41.5 %   |
> > > > > | VisionFive 2 |   88.7 %   | 91.6 % |  32.6 %   |
> > > > > | RPi Zero 2   |   92.4 %   | 79.0 % |  59.1 %   |
> > > >
> > > > Building the initramfs takes more CPU cycles (see tests on tmpfs),
> > > > but
> > > > saves time on disk IO. Daves Jones saw much bigger time savings on
> > > > his
> > > > Raspberry Pis but his tests were on lunar.
> > > >
> > > > Build time influence:
> > > > + add_directories plus uniq take 

Re: Amendment: Update on reducing initramfs size and speed up

[Dimitri John Ledkov]

On Thu, 27 Jul 2023 at 00:51, Michael Hudson-Doyle
 wrote:
>
>
>
> On Thu, 27 Jul 2023 at 09:21, Benjamin Drung  wrote:
>>
>> On Wed, 2023-07-26 at 17:53 +0200, Benjamin Drung wrote:
>> > Hi all,
>> >
>> > A few weeks ago, I posted an idea how to reduce the initramfs size and
>> > speed up the generation:
>> >
>> > https://lists.ubuntu.com/archives/ubuntu-devel/2023-July/042652.html
>> >
>> > This post sparked a lively discussion. The initial idea was ditched for
>> > a better solution: mkinitramfs will put all compressed files (kernel
>> > modules and firmware files) into a cpio archive that is not compressed
>> > (because compressing compressed files makes no sense). All other files
>> > will be added to a cpio archive that gets compressed. As next steps, the
>> > kernel modules and firmware files need to be shipped compressed.
>> >
>> > After several iterations for the implementation and review by Daves
>> > Jones, I just uploaded initramfs-tools 0.142ubuntu8 to mantic that puts
>> > compressed kernel modules and firmware files in an uncompressed cpio
>> > (https://launchpad.net/bugs/2028567).
>> >
>> > I created/updated the follow-up tickets and added my patches to them:
>> >
>> > Ship kernel modules Zstd compressed
>> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568
>> >
>> > compress firmware in /lib/firmware
>> > https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1942260
>> >
>> > And without further ado, here come the benchmark results:
>> >
>> > The benchmarks were done either on an AMD Ryzen 7 5700G with schroot and
>> > overlay on tmpfs or on the hardware mentioned. All tests were running
>> > the latest Ubuntu mantic development release.
>> >
>> > * minimal: schroot with linux-image-generic initramfs-tools zstd
>> > * full: minimal + busybox-initramfs cryptsetup-initramfs
>> >   isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme-spinner
>> > * nvidia: full + linux-headers-generic nvidia-driver-525
>> > * nvidia fw: nvidia + compressed /lib/firmware/nvidia/525.125.06/
>> > * VisionFive 2: VisionFive 2 RISC-V board
>> > * RPi Zero 2: Raspberry Pi Zero 2 ARM board (running armhf)
>> >
>> > "next" means using kernel/firmware/initramfs from ppa:bdrung/ppa i.e.
>> > * initramfs-tools 0.142ubuntu7bd4
>> > * linux 6.3.0-7.7bd2
>> > * linux-firmware 20230629.gitee91452d-0ubuntu1bd1
>> >
>> > > || build   | size   | uncompressed size  |
>> > > | test   | time| in bytes  | in MiB | in bytes  | in MiB |
>> > > ||-|---|||
>> > > | minimal| 4.30 s  |  66701818 |  63.6  | 224087608 | 213.7  |
>> > > | minimal next   | 4.54 s  |  59935186 |  57.2  |  67738810 |  64.6  |
>> > > | full   | 7.15 s  | 118007038 | 112.5  | 387976843 | 370.0  |
>> > > | full next  | 7.29 s  | 106937908 | 102.0  | 128610985 | 122.7  |
>> > > | nvidia | 7.04 s  | 209200523 | 199.5  | 513554279 | 489.8  |
>> > > | nvidia next| 7.21 s  | 195246287 | 186.2  | 235288174 | 224.4  |
>> > > | nvidia fw next | 7.16 s  | 191329102 | 182.5  | 213078234 | 203.2  |
>> > > | VisionFive 2   | 142.9 s | 121895035 | 116.2  | 411160836 | 392.1  |
>> > > | VF 2 next  | 126.7 s | 111651453 | 106.5  | 134120804 | 127.9  |
>> > > | RPi Zero 2 | 109.5 s |  39803044 |  40.0  |  69592789 |  66.4  |
>> > > | RPi Zero 2 ²   | 103.5 s |  39804499 |  40.0  |  69592789 |  66.4  |
>> > > | RPi Zero 2 next| 101.2 s |  31463352 |  30.0  |  41145762 |  39.2  |
>> >
>> > ² Updated initramfs-tools (but no compressed modules or firmware)
>> >
>> > The build time was averaged over five runs.
>> >
>> > > | improvement  | build time | size   | uncompressed size |
>> > > |--|||---|
>> > > | minimal  |  105.6 %   | 89.9 % |  30.2 %   |
>> > > | full |  102.0 %   | 90.6 % |  33.1 %   |
>> > > | nvidia   |  101.7 %   | 91.5 % |  41.5 %   |
>> > > | VisionFive 2 |   88.7 %   | 91.6 % |  32.6 %   |
>> > > | RPi Zero 2   |   92.4 %   | 79.0 % |  59.1 %   |
>> >
>> > Building the initramfs takes more CPU cycles (see tests on tmpfs), but
>> > saves time on disk IO. Daves Jones saw much bigger time savings on his
>> > Raspberry Pis but his tests were on lunar.
>> >
>> > Build time influence:
>> > + add_directories plus uniq take several milliseconds
>> > + depmod on compressed kernel modules take hundreds of
>> >   milliseconds longer
>> > - copying smaller kernel modules (due to compression) is faster
>> > - cpio archive that needs to be compressed is smaller
>> > - not storing intermediate cpio archives saves time
>> >
>> > Saving 10 to 20 percent on the initramfs size and only needing half or a
>> > third of the size when unpacked (i.e. needed memory during boot) is a
>> > good improvement.
>>
>> The smaller initramfs overall size (less to load into memory and unpack)
>> and the smaller compressed cpio 

Re: Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Sat, 22 Jul 2023 at 02:03, Steve Langasek  wrote:
>
> On Tue, Jul 11, 2023 at 01:17:27AM +0200, Benjamin Drung wrote:
> > If you want to test it yourself, you can find initramfs-tools
> > 0.142ubuntu7bd2 for mantic in my PPA:
> > https://launchpad.net/~bdrung/+archive/ubuntu/ppa
>
> What blocks this from being uploaded to mantic?  AFAICS the behavior should
> remain unchanged for existing kernel+firmware packages, and it's therefore
> safe to push more widely.
>

I have reviewed this, and I approve of these changes. Please note,
that we need this uploaded in mantic, such that linux-firmware can
declare breaks on versions prior to this upload and switch to .zst
compressed firmware files.

As this will be the first upload of initramfs-tools that supports .zst
compressed firmware.

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Fri, 21 Jul 2023 at 21:08, Steve Langasek  wrote:
>
> On Fri, Jul 21, 2023 at 07:33:58PM +0100, Dimitri John Ledkov wrote:
> > If it is a concern that v5.15 jammy kernel may potentially be used
> > after partial / incomplete upgrade to Mantic, we can opt into using XZ
> > compressed firmware or I can backport ZSTD firmware support to jammy
> > GA kernel - it is a trivial patch (given that ZSTD itself is otherwise
> > available and used by the jammy kernel).
>
> The scenario here is that the user has just upgraded to mantic, so they get
> the new linux firmware and the new kernel; they reboot to mantic, but there
> is a regression of some kind; they reboot back to the jammy kernel (which is
> the N-1 kernel on their system) to debug it, and that kernel has regressed
> support for their hardware because it can no longer load the needed firmware
> from the root filesystem.
>
> This is a realistic enough scenario that I think it is worth backporting the
> zstd support onto jammy's 5.15 kernel.

This portion will be tracked in
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028550 submitted
in https://lists.ubuntu.com/archives/kernel-team/2023-July/141355.html

As previously described it was fairly trivial backport of 3 files
changed, 98 insertions(+), 8 deletions(-)

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Mon, 24 Jul 2023 at 13:34, Adrien Nader  wrote:
>
> On Tue, Jul 11, 2023, Michael Hudson-Doyle wrote:
> > I was wondering if it make sense to construct a zstd dictionary for
> > compressing kernel modules but I didn't realize they need to be available
> > at decompression time, I'm not sure the kernel would support that.
>
> I've tried to use the zstd --train and I don't think it is appropriate
> here, or at least not without significant preparation of datasets. It's
> more suited for many many small and similar files. We don't have that
> many files and there are only clusters of similarities, not similarities
> shared across every file. Moreover, since our files are big (at least
> the one that matter for the overall size), they're already contributing
> a lot to the live dictionary and the pre-built one has little overall
> influence.

Also I don't think it will work for us - the train command produces
dictionary that needs to be present for both compression and
decompression, and right now the kernel doesn't have support for that
or more specifically to load one from userspace (potentially has
security implications). We could in theory generate training files,
generate dictionaries, bake it into our kernel, but then it would make
all of our compressed things non-portable or more difficult to update
dictionary, and the .zst file itself becomes non-portable as it now
depends on an external dictionary. Dictionary make sense for lots of
similar things (lots of status icons, or similar gaming images) and
then one can take all assets, train on them, and do a static build
that uses said perfect dictionary against a perfect set of fixed
assets. It sort of like the more generalised usecase of perfect-hash
functions (gperf) but for data files.

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Fri, 21 Jul 2023 at 17:59, Steve Langasek  wrote:
>
> On Tue, Jul 11, 2023 at 01:28:28AM +0200, Benjamin Drung wrote:
> > > > Okay. It works now. The not-compressed cpio archive must not be the last
> > > > one. So the order is now:
> > > >
> > > > * AMD/Intel microcode cpio archive (on amd64)
> > > > * compressed kernel modules / firmware (not compressed)
> > > > * main cpio archive (compressed)
> > > >
> > > > I'll really stop now. For a first comparison, the firmware files need to
> > > > be converted correctly. There are symlinks in /lib/firmware. So running
> > > > following was not correct/enough:
> > > >
> > > > find /lib/firmware -name '*.bin' | while read -r fw; do
> > > > sudo zstd -19 -z -o "${fw}.zst" "$fw"
> > > > sudo rm "$fw"
> > > > done
> > > >
> > > > If you want to help, hand me a correct conversion script.
>
> > > Some filenames in /lib/firmware contain spaces. There are many more
> > > files than .bin ones. A number of the files are readmes.
>
> > > Recent changes in linux-firmware.git add "install-xz" and "install-zstd"
> > > targets to make than will do what you want I guess. I haven't checked
> > > that this was actually merged; it was discussed at least on 2023-03-01
> > > on the mailing-list. It's probably the best path forward in any case.
>
> > > In case it isn't merged, you can have a look at
> > > https://lore.kernel.org/linux-firmware/20230301-fixes-and-compression-v2-0-e2b71974e...@gmail.com/T/
>
> > I agree. The conversion script was just for a quick way to test. The
> > clean approach would be to patch linux-firmware to produce two
> > additional binary packages: linux-firmware-zst and linux-firmware-xz.
>
> We need to find an upgrade path here that doesn't involve having to have
> multiple linux-firmware binary packages with different compression types.
> Ending up with three copies of the firmware in /lib on disk because of
> different dependencies is not an improvement!
>
> I understand the reason for being concerned about keeping uncompressed
> firmware available is that not all kernels have support for compressed
> firmware.  However we should work out a path that lets us switch to
> compressed firmware on releases where we know it's supported.  What kernel
> version introduced support for this?
>

XZ compressed firmware load was added in v5.3 and is turned on in the
focal GA v5.4 kernel.

ZSTD compressed firmware load was added in v5.19 and is available in
Jammy HWE kernels, kinetic and later.

initramfs-tools in Jammy has unoptimized support for xz compressed firmware.

Thus it is safe for linux-firmware to switch to ZSTD compressed
firmware straight away in mantic.

If it is a concern that v5.15 jammy kernel may potentially be used
after partial / incomplete upgrade to Mantic, we can opt into using XZ
compressed firmware or I can backport ZSTD firmware support to jammy
GA kernel - it is a trivial patch (given that ZSTD itself is otherwise
available and used by the jammy kernel).

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Thu, 13 Jul 2023, 22:37 Benjamin Drung,  wrote:

> On Thu, 2023-07-13 at 08:32 +0100, TJ wrote:
> > Is there a specific reason why the focus is on trying to shoe-horn
> > everything possible into the initrd.img and then compress rather than
> > winnow out the files an installed system will never need to find the
> > root file-system?
>
> These are two orthogonal targets. MODULES=most is the default, because
> you can change the hardware and the system will still boot. Advanced
> users can use MODULES=dep to only include the needed bits, but then they
> have to be aware of it when changing the hardware. Improving the way we
> compress the initramfs could help in both configurations.
>
> > I tackled the issue of ever-expanding host-generated initrd.img file
> > sizes when they began hitting 80MB+ back in 2018 and since then have
> > carried my own patches that reduce sizes by not including files the host
> > will not need.
> >
> > One small change that brings major benefits is only including the
> > firmware files required, not every file declared by a kernel module. The
> > GPU drivers are the main culprit there with amdgpu declaring over 500 of
> > which, for any individual GPU, only a handful are relevant, but
> > Plymouth's pulling in multiple GPU drivers doesn't help that.
> >
> > Statistics (initrd.img with kernel v6.2.4):
> > MODULES=  FIRMWARE_LOADED size  MOST DEP firmwares build-time
> > most  false   77117694   634   14.49
> > most  true60302859 -22%8   11.99
> > dep   false   42489938 -45%  606   6.84
> > dep   true25704125 -66% -40%   8   6.35
> >
> > FIMWARE_LOADED=true relies on a simple kernel patch which I've been
> > meaning to upstream that writes "Firmware loaded: " for each.
> >
> > $ journalctl --dmesg | grep 'Firmware loaded' | head -n 15
> > Jul 02 11:42:21 sunny kernel: firmware_class: Firmware loaded: reporting
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/verde_pfp.bin
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/verde_me.bin
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/verde_ce.bin
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/verde_rlc.bin
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/verde_mc.bin
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/verde_smc.bin
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/TAHITI_uvd.bin
> > Jul 02 11:42:21 sunny kernel: radeon :0a:00.0: Firmware loaded:
> > radeon/TAHITI_vce.bin
> > Jul 02 11:42:22 sunny kernel: r8152 2-8.3:1.0: Firmware loaded:
> > rtl_nic/rtl8153a-4.fw
> > Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded:
> > regulatory.db
> > Jul 02 11:42:28 sunny kernel: platform regulatory.0: Firmware loaded:
> > regulatory.db.p7s
> >
> > My experiments and patches are documented at
> >
> > https://iam.tj/projects/ubuntu/initramfs-tools/
>
> That is a really good idea. After looking at the patches, the options 1
> and 2 (sysfs or procfs interface) are the more robust ones (but need
> more work implementing them in the kernel). Option 3 (kernel logging)
> seem to be too fragile for using it programmatically. What happens if a
> user boot with quiet option and clears the kernel log buffer?
>

journald stores a copy of dmesg subject to its own retention and rotation
policy, even when dmesg is cleared. Thus it should be reliable to have
journal output for dmesg for multiple boots.

The issue however is that drivers and kernels can change on disk, and the
next kernel ABI may need different or more firmware files. examples:
1) on desktop installs hwe-22.04 rolls major versions.
2) each Nvidia SRU update needs a new firmware file name



> Is introducing FIRMWARE_LOADED really needed? Isn't it enough to only
> include the loaded firmware for MODULES=dep and all firmware otherwise?
>
> > The main take is the question:
> >
> > Is MODULES=most really the best default for installed-on-host images (it
> > obviously is for installer and portable images). If it can be nuanced
> > then =DEP plus winnowing the firmware files gives great gains with
> > minimal effort.
>
> MODULES=most is the best default since it allows changing the underlying
> hardware.
>
> --
> Benjamin Drung
> Debian & Ubuntu Developer
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Turning on phased update support in chroots in 24.10 (24.04?)

[Dimitri John Ledkov]

On Thu, 13 Jul 2023 at 11:42, Julian Andres Klode
 wrote:
>
> On Thu, Jul 13, 2023 at 11:09:02AM +0100, Dimitri John Ledkov wrote:
> > On Thu, 13 Jul 2023 at 10:27, Julian Andres Klode
> >  wrote:
> > >
> > > Hi folks,
> > >
> > > I just got reminded that when we wrote the initial phasing code
> > > we made it not apply in chroots to avoid breaking builders and
> > > things.
> > >
> > > I'd like to remove that check because it's a bit unexpected. To
> > > do that, I'll probably add an option to override the chroot check
> > > to apt soon for 23.10 and then we can drop the check in 24.10, or
> > > 24.04 even.
> > >
> > > When the initial code was written, phasing was implemented using
> > > policy and respected by the install command. Since then, phasing
> > > has moved to the upgrade calculation, using keep back, so there are
> > > significantly less concerns as installs no longer respect phasing,
> > > so image building is not affected anymore, but upgrading build chroots
> > > would be.
> >
> > If proposed is enabled, and pinned up, can the phasing be ignored on
> > the updates pocket?
> > Or is there a pinning preference we can use, to again update all our
> > chroot code to ensure unphased upgrades are done?
>
> Phasing is not affected by pinning you need to set:
>
> APT::Get::Always-Include-Phased-Updates "true";

Given this option already exists, we should just start setting it
explicitly in launchpad-buildd & mk-sbuild.

>
> I do not think encoding that enabled proposed implies no phasing
> is a good idea, because apt shouldn't know specifics about what
> suite names mean.
>
> But then again, the phasing code knows about -security right
> now, because it disables phasing if there's a security update
> between the installed and the phased version (as it cannot
> switch candidates, only keep back).
>
> Optimally one day a new solver enables us to um not need to
> know that (solver draft: https://magenta.jak-linux.org/bin/solver3.pdf
> | https://salsa.debian.org/apt-team/solver3).
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer  i speak de, en



-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Turning on phased update support in chroots in 24.10 (24.04?)

[Dimitri John Ledkov]

On Thu, 13 Jul 2023 at 10:27, Julian Andres Klode
 wrote:
>
> Hi folks,
>
> I just got reminded that when we wrote the initial phasing code
> we made it not apply in chroots to avoid breaking builders and
> things.
>
> I'd like to remove that check because it's a bit unexpected. To
> do that, I'll probably add an option to override the chroot check
> to apt soon for 23.10 and then we can drop the check in 24.10, or
> 24.04 even.
>
> When the initial code was written, phasing was implemented using
> policy and respected by the install command. Since then, phasing
> has moved to the upgrade calculation, using keep back, so there are
> significantly less concerns as installs no longer respect phasing,
> so image building is not affected anymore, but upgrading build chroots
> would be.

If proposed is enabled, and pinned up, can the phasing be ignored on
the updates pocket?
Or is there a pinning preference we can use, to again update all our
chroot code to ensure unphased upgrades are done?
And then patch mk-sbuild, lp-buildd, kteam-tools to add -proposed
pinning for lunar+.

Note launchpad builders chroots start with GA release, and depending
on the build configuration enable security/updates/proposed and do a
dist-upgrade.

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Sat, 8 Jul 2023, 02:49 Benjamin Drung,  wrote:

> On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote:
> > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung  wrote:
> > >
> > > Hi all,
> > >
> > > a year ago we changed the default compression and level for the
> > > initramfs to zstd -1. This fixed the very slow creation times on
> > > development boards (see bug #1958148), but that leads to bigger
> > > initramfs sizes that triggered other bugs (like bug #1842320).
> > > Big initramfs sizes can also fill up small sized /boot partitions
> easily
> > > (grooming the 850 initramfs-tools bugs revealed several such reports).
> > >
> > > Using xz -9 would give very good compression, but it takes very long
> > > (especially on slow development boards) and a lot of memory (good luck
> > > on Raspberry Pis with small memory like Pi Zeros).
> > >
> > > I propose following approach to address the drawback: Create cpio
> > > archives (compressed with xz -9) for the kernel modules and firmware
> > > files when building the kernel/firmware Debian package. Then ship those
> > > cpio archives in the package (or in a separate binary package). Then
> the
> > > CPU load it put on the builders. The cpio archives would contain the
> > > modules for MODULES=most.
> > >
> > > mkinitramfs will then look for those cpio archives and uses those in
> > > case they are present. Such a initramfs would look like this:
> > >
> > > * AMD/Intel microcode cpio archive (on amd64)
> > > * main cpio archive compressed with zstd -1
> > > * kernel modules from the Debian package compressed with xz -9
> > > * firmware files from the Debian package compressed with xz -9
> > >
> >
> > Majority of our instances boot without initrd, and there too they
> > don't load most of the modules.
> > Creating xz -9 compressed archive of all modules, still pays the
> > penalty to decompress most of them, and then not modprobe them.
> > I was hoping to achieve a similar in spirit approach, but didn't quite
> > have the time to implement is:
> >
> > 1) change linux-modules and linux-firmware to ship .ko.zst
> > firmware.bin.zst compressed with zstd -19 at .deb build time
> > 2) this saves install size of the packages, with only slightly
> > increased download size
> > 3) modify initramfs-tools to include compressed files into a separate
> > initrd, which is not compressed (i.e. exclude .zst files from the
> > default main compressed cpio archive, and append them in the second
> > main cpio archive that is uncompressed)
> > 4) this should achieve quick initrd creation, which will be smaller in
> > size that current status, and will boot faster as it will only
> > decompress modules/firmware it actually needs at boot
> >
> > For experimentation locally, you can recompress .ko with zstd in place
> > in /lib/modules/; and rerun depmod. To then test initramfs-tools
> > changes that skip over .zst compressed files and add them as is in an
> > uncompressed appended cpio.
>
> That is a very good idea. I created a draft for point 3 in [2]. It moves
> the compressed files into a separate directory and creates a separate
> cpio archive for that directory without compressing it:
>
> * AMD/Intel microcode cpio archive (on amd64)
> * main cpio archive (compressed)
> * compressed kernel modules / firmware (not compressed)
>
> Sadly this does not work (yet). cpio complains with "premature end of
> archive" when looking at it and the kernel fails to extract the last
> cpio part. I am heading to bed now leaving that bug for another day.
>


I had this before with lz4 and lzo compression and fixed this before in the
kernel. It is likely a kernel bug mishandling mixed compression initrds.

It will likely work if you Make compressed-files the third early
initrd. Such that we have intel, AMD microcode early initrds, followed by
compressed-files initrd, followed by compressed main initrd.


> [2]
> https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed
>
> --
> Benjamin Drung
> Debian & Ubuntu Developer
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Sat, 8 Jul 2023 at 01:19, Benjamin Drung  wrote:
>
> Hi all,
>
> a year ago we changed the default compression and level for the
> initramfs to zstd -1. This fixed the very slow creation times on
> development boards (see bug #1958148), but that leads to bigger
> initramfs sizes that triggered other bugs (like bug #1842320).
> Big initramfs sizes can also fill up small sized /boot partitions easily
> (grooming the 850 initramfs-tools bugs revealed several such reports).
>
> Using xz -9 would give very good compression, but it takes very long
> (especially on slow development boards) and a lot of memory (good luck
> on Raspberry Pis with small memory like Pi Zeros).
>
> I propose following approach to address the drawback: Create cpio
> archives (compressed with xz -9) for the kernel modules and firmware
> files when building the kernel/firmware Debian package. Then ship those
> cpio archives in the package (or in a separate binary package). Then the
> CPU load it put on the builders. The cpio archives would contain the
> modules for MODULES=most.
>
> mkinitramfs will then look for those cpio archives and uses those in
> case they are present. Such a initramfs would look like this:
>
> * AMD/Intel microcode cpio archive (on amd64)
> * main cpio archive compressed with zstd -1
> * kernel modules from the Debian package compressed with xz -9
> * firmware files from the Debian package compressed with xz -9
>

Majority of our instances boot without initrd, and there too they
don't load most of the modules.
Creating xz -9 compressed archive of all modules, still pays the
penalty to decompress most of them, and then not modprobe them.
I was hoping to achieve a similar in spirit approach, but didn't quite
have the time to implement is:

1) change linux-modules and linux-firmware to ship .ko.zst
firmware.bin.zst compressed with zstd -19 at .deb build time
2) this saves install size of the packages, with only slightly
increased download size
3) modify initramfs-tools to include compressed files into a separate
initrd, which is not compressed (i.e. exclude .zst files from the
default main compressed cpio archive, and append them in the second
main cpio archive that is uncompressed)
4) this should achieve quick initrd creation, which will be smaller in
size that current status, and will boot faster as it will only
decompress modules/firmware it actually needs at boot

For experimentation locally, you can recompress .ko with zstd in place
in /lib/modules/; and rerun depmod. To then test initramfs-tools
changes that skip over .zst compressed files and add them as is in an
uncompressed appended cpio.

> After working on initramfs-tools as part my day job, my fingers were
> itching and I had to create a quick and dirty draft in my free night
> time. You can find the result of the last two hours in [1]. This draft
> has a mkinitramfs-kernel script that creates a cpio archive containing
> the kernel modules and firmware (that needs to be split later on).
>
> The lunar test result on my AMD Ryzen 7 5700G look promising: Building
> 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds and is
> 54.9 MiB in size. Creating the initramfs speeds up from around 8.7
> seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1 MiB to
> 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but
> /lib/modules need 54.9 MiB for the cpio archive.
>
> The drawback is that building the kernel would take longer, the package
> takes more space on the archive and mirrors, and downloading them could
> take longer on slow connections.
>
> Implementing my proposal would be relative easy for initramfs-tools, but
> would mean some work for the kernel team.
>
> What do you think?
>
> [1] 
> https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/prebuilt
>
> --
> Benjamin Drung
> Debian & Ubuntu Developer
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Open-SSH server

[Dimitri John Ledkov]

On Sat, 10 Jun 2023, 19:39 Matthew Wilson, 
wrote:

> Hi there,
>
>
>
> Do you have an update as to when the repository for Ubuntu 22.04.2 package
> Open-SSH will be upgraded from 8.9 to 9.3 to patch the security issues as
> it means our server is currently non-compliant.
>

Non complaint with what exactly? 9.3 is available in the current
development release mantic. Note that 8.9 and 9.3 are major new upstream
releases of openssh with new features and bugfixes. And are not indication
of security vulnerabilities. When vulnerabilities arise, Ubuntu issues USN
and a distribution patch update addressing vulnerability alone without
upgrading to new major version. See https://ubuntu.com/security/notices

In jammy we have shipped a Ubuntu patch level update to address a poll()
issue. No other severe bugs or vulnerabilities are currently known in
openssh. If you can disclose a currently non public vulnerability affecting
openssh 8.9 through 9.3 you can do so by contacting Ubuntu Security
desclosure team at  secur...@ubuntu.com or opening an private security bug
report on launchpad against distribution Ubuntu package openssh.

Ps. Please do not use html signatures, especially those that contains ads,
when contacting public mailing lists of opensource projects as that is
considered off topic. You should be able to configure a different signature
when emailing mailing lists, or manually turn it off during compose in your
email client.

--
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Symbols files for C++ libraries for Ubuntu main

[Dimitri John Ledkov]

On Fri, 9 Jun 2023 at 20:10, Steve Langasek  wrote:
>
> Hi Seb,
>
> On Fri, Jun 09, 2023 at 02:27:02PM +0200, Sebastien Bacher wrote:
> > I would like to ask if there is any chance the MIR team would reconsider
> > their position on the topic (at least until the day we have a somewhat
> > working solution we can use)?
>
> > which also included those types of changes
>
> > - _Znam@Base 2.0~b2-0ubuntu3
> > + _Znaj@Base 2.0~b2-0ubuntu4
> > +#MISSING: 2.0~b2-0ubuntu4# _Znam@Base 2.0~b2-0ubuntu3
>
> > I personally don't understand why we have those symbols existing on armhf
> > which don't exist on amd64. Nor why _Znam@Base is becoming _Znaj@Base nor
> > how we are supposed to handle such cases
>
> Passing them through c++filt may help explain:
>
> $ echo _Znam@Base | c++filt
> operator new[](unsigned long)@Base
> $ echo _Znaj@Base | c++filt
> operator new[](unsigned int)@Base
> $
>
> There are various C++ functions whose signature will change based on the
> architecture word length.
>
> .symbols files support various kinds of globbing etc to be able to express
> this logically (e.g., you could say '_Zna[mj]@Base' instead of listing two
> different symbols as optional), but as you've found, it's an onerous,
> iterative process to identify all the ways C++ symbols vary across
> architectures and then encode this in a .symbols file.  And in this case,
> the symbol isn't part of the library's public ABI anyway, this is just a
> function from the base C++ library!
>
> > 4. doing those tweaks need to be done manually since it's not only applying
> > the diff but also adding optional keyword at places, I got one wrong and it
> > failed to build again
>
> > add one more symbol specific to risvc
> > http://launchpadlibrarian.net/647875197/libcupsfilters_2.0~b2-0ubuntu6_2.0~b2-0ubuntu7.diff.gz
>
> Yep.
>
> > I understand the motivation for wanting a symbol file but I agree with what
> > Robie, what's the benefit. In that case we spent a few hours to end up with
> > a .symbols which as over 150 '(optional)' entries, that doesn't protect us
> > much better than just not having a .symbols or using -c0 but still has a
> > high cost.
>
> I wouldn't say that it doesn't protect you.  It's a pain to set up initially
> and as you note, you can even have to do further fix-ups as a result of
> toolchain changes, as the set of template functions and other C++ sugar from
> outside of the library that gets exported as ELF symbols can change.  It
> DOES have a high cost, but in the end it provides the same level of
> protection against accidental ABI breakage as it does for C libraries.
>
> It would be nice to have better consistent tooling around ABI checking for
> C++ libraries.  I think the KDE team had some tools around automating the
> generation of symbols files - it does require two passes, first to build on
> all archs and then to merge the results.  But in principle that's better
> than whack-a-mole.
>
> We could also consider using abi-compliance-checker instead of symbols files
> for C++ libraries.  There is a dh-acc debhelper addon, but I've never used
> it.  We are currently using abi-compliance-checker for the ABI analysis of
> armhf for the move to 64-bit time_t; it's unmaintained upstream, but it does
> seem to work pretty well - the vast majority of issues we've encountered
> with it, when trying to run it over the entire Ubuntu archive, have been due
> to header files that #include headers from packages they don't depend on, or
> collections of headers that can't all be included together.  Both of these
> are issues of much less significance when it's the maintainer doing the
> work.  It would require the same sort of two-pass setup process as the KDE
> tools, though, and if it has to be done per-arch (probably), it's more
> awkward to set up because a-c-c .dump files aren't ascii that you can scrape
> from the build logs of failed builds - but it might be a more reliable tool
> over the long term than dpkg-gensymbols for C++ libraries.
>
> Downside: symbols files also let you track what version of a package a
> symbol was added in, which lets packages' versioned dependencies on
> libraries be no stricter than actually necessary.
>
>
> I don't speak for the MIR team, I have no objection to them relaxing the
> requirement of .symbols files for C++ libraries in main.  Just offering some
> suggestions on how we can do a better job of automating C++ ABI checks than
> we're doing today.


boost was in a similar position. it is too much work to maintain
symbols files for it, update them for new upstream releases, or even
just adjust for the new default compiler.
the solution there was to just bump the library soname on every
upstream change, and often bump package name (sort of soft library abi
bump) upon changes to the vendored re-exported libs and/or compiler
induced symbol changes. That works really well for c++. It may be
redundant for some updates, but it is a foolproof way, with single
pass uploads.

-- 
okurrr,

Re: NBS kernel removals: round two

[Dimitri John Ledkov]

On Thu, 1 Jun 2023 at 21:08, Steve Langasek  wrote:
>
> Hi folks,
>
> Well, we found out that removing all NBS kernel packages for stable series
> was not altogether without its problems for users.  We have modified the
> removal policy going forward in response to feedback.
>
> Meanwhile, in preparation for the move of bionic from standard support to
> ESM, the Release Team has noticed that NBS cleanup for ESM releases has so
> far been deferred until the releases go fully EOL.
>
> At this point, the *newest* kernel image in trusty-security is missing 3
> years of security updates.  I believe it's therefore reasonable to require
> that users doing deployments of trusty today should do so using a kernel
> that is *no more than* 3 years out of date, before immediately enabling
> Ubuntu Pro and upgrading to a recent kernel security release.
>
> Consequently, I will be removing all kernel packages older than linux
> 3.13.0-170.220 from trusty-{updates,security} next week.
>

I believe 3.13.0-165-generic is required to be published, as that's
the one that is used by
http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-amd64/current/images/netboot/mini.iso
 and all the related boot artifacts.

Also please ensure that 4.4.0-142 remains published too, as that one
is used by the hwe-mini.iso and related boot artifacts.

We should realistically continue to support the usage of the last
installer we built
https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu318.46

Apart from those two, all others can be cleaned up. Or set the above
versions as the ceilings for removals of generic & lts-xenial kernels.

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Choice of the openssl version for 23.10 and 24.04

[Dimitri John Ledkov]

We had similar dilemma around focal release. And I did SRU one off upgrade
from 1.1.0 to 1.1.1. it was a minor disaster. (As in like the sad
depressing songs in A minor scale).

It is best to stick to one openssl version in a release.

It is best to stick to longer supported one.

It is best not to chase minor ones that nobody will use or want long term.

Mantic is open for development, and usually optimisations are fairly
standalone. Even if upstream is not backporting performance optimisations -
we can do so, and have done that for x86, s390x, ppc64el, arm64 in the
past. If there are high priority optimisations that we want in our openssl
we should attempt backports of those onto current openssl release we ship
in mantic.

Note, we would have to then monitor for regressions & security fixes to
those optimisation backports.

On Wed, 17 May 2023, 21:35 Adrien Nader,  wrote:

> On Tue, May 16, 2023, Marc Deslauriers wrote:
> > On 2023-05-15 05:18, Adrien Nader wrote:
> > > Hello,
> > >
> > > Ubuntu currently ships openssl 3.0. Debian will release with 3.0.
> > >
> > > Debian experimental contains 3.1. Openssl 3.1 has been out for a couple
> > > months. It seemed natural to switch to 3.1 which contains a number of
> > > interesting changes including fixes for performance regressions except
> > > that...
> > >
> > > Quoting https://www.openssl.org/policies/releasestrat.html :
> > >
> > > - Version 3.1 will be supported until 2025-03-14
> > > - Version 3.0 will be supported until 2026-09-07 (LTS).
> > >
> > > The support for 3.1 spans two years while the support for 3.0 spans 5
> > > years since it's an LTS.
> > >
> > > If the openssl teams keeps the same cadence (I love extrapolating with
> > > only two data points, it's much simpler), then 3.2 could be released
> > > September 2024 and it could be just in time to be included in 24.10 but
> > > obviously not 24.04. There's also no indication at the moment that 3.2
> > > would be an LTS release. As for 3.3, it would be released March 2026
> and
> > > that would still be early enough to make it the next LTS.
> > >
> > > As I said, these dates are extrapolation based on the 3.0 to 3.1
> release
> > > and I haven't seen communication from the openssl team about their
> > > roadmap besides that they had to remove it at some point because it
> > > needed more work. It's seems unlikely that the result of "more work" is
> > > as simple as "release every 18 months".
> > >
> > > In any case, it seems unlikely that 3.2 is released in time for 24.04
> > > feature freeze AND is an LTS. I'm going to raise this topic on the
> > > openssl issue tracker but I wanted to begin the discussion here since
> > > the same issue is likely to pop again in the future.
> > >
> > > In short:
> > >
> > > In 24.04, do we want to include a release of openssl that will be
> > > supported for "at least two years", possibly starting a year before our
> > > release, or do we want to include a release that will be supported for
> > > "at least five years", possibly starting two years before our release.
> >
> > I'd rather we stick with an OpenSSL LTS release, as that is possibly what
> > others distros will be using and we will be able to collaborate on fixes
> > once the upstream support goes away.
>
> OK. I don't have strong opinions on this, especially since I'm not the
> one who will be pushing security updates.
>
> My main concern is a possible backlash, especially since 3.0's
> performance is sometimes a lot worse than 1.1's and that there won't be
> a newer version in a Ubuntu LTS before 26.04 (
> https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2009544 is one
> example ).
>

Can this be cherry picked into mantic?




> > > Bonus questions:
> > >
> > > What do we do when the support on the openssl side ends but there's
> > > three more years of support for our LTS releases?
> >
> > We do like we do for all the other packages in our archive, we backport
> > patches to the unsupported version. (And we support our LTS releases for
> a
> > minimum of 10 years now...)
>
> I had forgotten this was now 10 years; I was too set on Bionic's
> schedule.
>
> One advantage of using 3.0 is that it would be the same version as in
> Jammy. Maybe even 26.04 will use it too...
>
> > >
> > > In case we SRU newer versions of openssl, will we attempt to maintain
> > > the same behaviour? (I wanted to ask that question but the answer is
> > > probably not a yes/no: a best-effort policy might make more sense)
> > >
> >
> > We don't SRU newer versions of OpenSSL. The attempts we've made in the
> past
> > to SRU a minor point release resulted in hundreds of fixes required all
> over
> > the archive and caused regressions for users. Backporting security fixes
> and
> > specific bug fixes is the best approach.
>
> Thanks for the explanation.
>
> --
> Adrien
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> 

Re: linux-headers-5.15.0-1028-gke for Ubuntu 22.04

[Dimitri John Ledkov]

1) did you try pull-lp-debs as suggested in the first email, and is
that not sufficient to find/download header debs and install them?

2) I have to point out that linux-gke 5.15.0-1028.33 was superseded on
2023-01-06 (more than 4 months ago) by linux-gke - 5.15.0-1024.29

The kernel gke abi you are using is 4 months obsolete and likely
contains multiple publicly known security vulnerabilities. GKE
provides multiple update mechanisms (cluster/image based, or
apt/unattended-upgrades based) , which one should be using to receive
kernel security updates for your cluster. Please contact your GKE
support to investigate why your clusters are not receiving security
updates.

Given your post, and other posts from other GKE users, I am deeply
concerned that many GKE deployments are not receiving updates, which
Ubuntu is publishing for GKE. Note that Ubuntu has no callhome
capability to explain why particular Ubuntu installations are not
downloading and applying security updates.

On Sun, 14 May 2023 at 12:26, Elad Gabay  wrote:
>
> Hi,
> I don't have access to do full host bind mount therefore I can't use headers 
> from the node, I need to be able to fetch the headers from a container\other 
> machine.
> Now I see the same for linux-headers-5.15.0-1030-gke version.
> Thanks
> ________
> From: Dimitri John Ledkov 
> Sent: Saturday, May 13, 2023 03:03
> To: Elad Gabay 
> Cc: ubuntu-devel-discuss@lists.ubuntu.com 
> 
> Subject: Re: linux-headers-5.15.0-1028-gke for Ubuntu 22.04
>
> Caution - External Sender
>
>
> Please see this discussion over here
> https://lists.ubuntu.com/archives/kernel-team/2023-May/139336.html and
> the emails before/later in the thread.
>
> tl;dr Note you have access to headers on the host that you can bind
> mount in the container, you are using obsolete out-of-date kernel ABI.
> You can use `pull-pkg / pull-lp-debs / pull-lp-ddebs` as needed to
> fetch desired packages for Jammy ABI directly from launchpad.
>
> On Sat, 13 May 2023 at 00:51, Elad Gabay  wrote:
> >
> > Hello,
> > Is there a reason that "linux-headers-5.15.0-1028-gke" published only for 
> > Ubuntu 20.04 but not for 22.04?
> > https://packages.ubuntu.com/uk/focal/main/linux-headers-5.15.0-1028-gke
> > Ubuntu – Details of package linux-headers-5.15.0-1028-gke in focal
> > Linux kernel headers for version 5.15.0 on 64 bit x86 SMP
> > packages.ubuntu.com
> >
> >
> > Thanks
> > Elad
> > --
> > Ubuntu-devel-discuss mailing list
> > Ubuntu-devel-discuss@lists.ubuntu.com
> > Modify settings or unsubscribe at: 
> > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
>
>
> --
> okurrr,
>
> Dimitri



-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: linux-headers-5.15.0-1028-gke for Ubuntu 22.04

[Dimitri John Ledkov]

Please see this discussion over here
https://lists.ubuntu.com/archives/kernel-team/2023-May/139336.html and
the emails before/later in the thread.

tl;dr Note you have access to headers on the host that you can bind
mount in the container, you are using obsolete out-of-date kernel ABI.
You can use `pull-pkg / pull-lp-debs / pull-lp-ddebs` as needed to
fetch desired packages for Jammy ABI directly from launchpad.

On Sat, 13 May 2023 at 00:51, Elad Gabay  wrote:
>
> Hello,
> Is there a reason that "linux-headers-5.15.0-1028-gke" published only for 
> Ubuntu 20.04 but not for 22.04?
> https://packages.ubuntu.com/uk/focal/main/linux-headers-5.15.0-1028-gke
> Ubuntu – Details of package linux-headers-5.15.0-1028-gke in focal
> Linux kernel headers for version 5.15.0 on 64 bit x86 SMP
> packages.ubuntu.com
>
>
> Thanks
> Elad
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss



-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: NBS removals of old kernels from stable -security and -updates pockets

[Dimitri John Ledkov]

On Fri, 12 May 2023 at 16:19, Steve Langasek  wrote:
>
> On Fri, May 12, 2023 at 02:20:39PM +0200, Juerg Haefliger wrote:
> > > I am therefore intending that, for jammy and later releases, we start to
> > > prune NBS kernel packages on an ongoing basis, not just at EOL time.
>
> > We already have users complaining on IRC about missing kernel packages...
>
> What, specifically, are the complaints?
>
> > What is the official way/process for getting older packages for example for
> > crash dump analysis where one might need an older kernel+dbgsym from an
> > active series?
>
> Does the Ubuntu Kernel Team accept crash reports on out-of-date kernels?

Yes, often. Especially when a given ABI is "popular" (aka default
quick launch in clouds, point release download media, certified, and
similar).
Also Canonical Support & Livepatch mostly work with out-of-date reports too.
As generally the desire is for the kernel they are going to reboot
into, fix a specific problem, rather than rebooting to the newest one
to still discover that the issue at hand is not fixed.

> The general policy for apport is to disallow bug report submissions if the
> executable or any of the loaded libraries are from out-of-date packages.
>
> But it will still be possible to download these older packages from
> Launchpad: https://launchpad.net/ubuntu/+source/linux/+publishinghistory

As mentioned elsewhere pull-pkg (and friends pull-lp-debs /
pull-lp-ddebs ) are very useful tools to quickly & securely download
desired packages from launchpad librarian.

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Building grub2-unsigned from sources on bionic

[Dimitri John Ledkov]

Hi,

On Tue, 14 Mar 2023 at 22:52, Vishwanath Pai  wrote:
>
> Hi All,
>
> I noticed that with the latest update to grub2-unsigned, one of the build 
> dependencies is gcc-10.
> But gcc-10 is not available on bionic. We build ubuntu packages in our build 
> system from source but
> unfortunately we have no way to build the new grub2-unsigned package in our 
> bionic build
> environment.
>
> Also another dependent package: libfuse3-dev is similarly not available on 
> bionic. How is
> grub2-unsigned being built for the official ubuntu repositories?

src:grub2 provided in a given suite is currently buildable in a given
suite. It was changed to exclude building EFI platform code, which is
now built by src:grub2-unsigned, once on one series with one
toolchain, and reused in all ubuntu releases.

Due to multiple recent EFI only vulnerabilties, to reduce security
review and maintenance and debugging/compatiblity checking costs, it
was decided to split src:grub2 into src:grub2 (with all the unsigned
platforms) and src:grub2-unsigned which only contain EFI platform
bootloader without any userspace binaries or dependencies.

If you observe the builds on launchpad, you will notice that
src:grub2-unsigned is built once, and is packaged to ensure that it is
a free standing package without any hard runtime dependencies on the
host system. I.e. open pad.lv/u/grub2-unsigned, click on a desired
version and observe the build
https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.1
you will see that it was compiled in Kineitc once, and published in
all the suites identically bionic; focal; jammy; kinetic. The build
log for each architecture, clearly states that kinetic build-shcroot
was used, and that produced binaries can be published and install in
both older, same and newer series, due to unique engineering we did in
the build process of only that pacakge.

The result of that is signed by Canonical Secureboot keys, and
published from src:grub2-signed package which in turn has per-series
runtime dependencies and versions, but vendors the same identical
build of the signed EFI package.

It is on our plans to provide a gcc-10 toolchain backport, such that
each suite can rebuild the binary package again. However, this will
not reproduce the same result, and may introduce security
vulnerabilities that rely on the toolchain features provided during
build. To reproduce what we have done in the Ubuntu archive, imho it
best to redo what we did - stand up Kinetic builder, build the source
package in Kinetic chroot, sign it with your own keys, rebuild
grub2-signed from scratch pointng at your signed binaries.

Note, rebuilding grub2-unsigned alone is not very useful, as it is not
used on installed systems at all. As the binaries it produces in the
signing tarball, have to be signed (for example we use Launchpad
Signing Service provided in the PPA and Archive builders), and then
revendored back inside grub2-signed. I am assuming you have your own
secureboot signatures, and you rebuild -signed packages yourself with
your own signatures.

If you do not rebuild signed binaries, and do not resign them with
your own secureboot keys, and instead rely on industry wide CA
certificates and using Canonical signed secureboot signatures - then
you probably already skip rebuilding packages such shim / shim-signed,
grub2 / grub2-signed, linux / linux-signed' and thus should also skip
grub2-unsigned.

Please note, this is the same thing we have been doing with shim /
shim-signed packages previously. So please check what you have been
internally doing with shim / shim-signed packages.

If you do not use Secureboot at all, and the split of platform code
that is built by src:grub2 & src:grub2-unsigned of different version
is not useful to you, you could patch those to either use a stock
toolchain, or revert to making src:grub2 package to generate all
bootloader platforms. Depending on what you do, this might be a
reasonable approach, however that will not include fixes for all
publically known vulnerabilities affecting EFI builds.

Please let me know publicly or privately, if this helps, or if you
have any further questions.

-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: [ubuntu-studio-devel] Lowlatency Kernel is behind in Ubuntu Studio

[Dimitri John Ledkov]

On Mon, 13 Mar 2023 at 17:54, Erich Eickmeyer  wrote:
>
> On Monday, March 13, 2023 10:29:50 AM PDT Dimitri John Ledkov wrote:
> > Ideally all flavours would be at 6.2 already, but due to various
> > reasons they are not.
>
> This is understandable and perfectly reasonable.
>
> > This is not unique to lowlatency flavour, and applies to kvm, azure,
> > raspi, and many more kernel flavours all of which are still on v5.19
> > in Lunar.
>
> My point is that lowlatency shouldn't be grouped-in to these flavors, but
> should be given higher priority and grouped-in with generic since it's still
> used in desktop systems by default and is directly affecting the testing of an
> official flavor of Ubuntu. This was one of the reasons we had to miss testing
> week because we didn't even have kernel parity.
>

Impossible. we run out of disk space and cannot complete the
lowlatency builds with generic any more. Thus it is now treated
exactly like all other derivatives.

We do not invest additional engineering & resource time, to prioritise
lowlatency flavour, over other flavours, which have more images and
higher usage.

> > We pushed 6.1 out, and migrated, on generic only, to migrate lots of
> > packages in proposed, specifically nvidia & everything entangled with
> > it, and thus unblock autopkgtesting of all the userspace packages
> > which were otherwise failing on v5.19. There is no intention to port
> > all flavours to 6.1.
>
> Again, this is one of the reasons we had do miss testing week among another

it was a mistake to skip testing week. you should have tested ubuntu
studio during the testing week like all the other flavours did. As
there are a lot of changes in lunar, that landed and affect ubuntu
studio. For example, all cloud images which use various cloud kernel
flavours, based on v5.19 did participate.

Can you explain who made the call to skip testing week? as to me, it
seemed, it's a requirement to release a flavour. Is studio going to
skip Lunar release?

> reason (two blockers this round). To not have kernel equality here could cause
> false positives in kernel-level testing. JACK and the audio stack, in
> particular, are directly affected by the kernel, and what might work in 6.1
> might not work in 5.19 with various devices. This could cause false bug
> reports and create a lot more problems for triage.
>
> > in Lunar, no further 6.1 builds will be done for any kernel flavour
> > for the time being. And v6.2 landing, across all flavours, is in
> > progress.
>
> Understandable. I'm just trying to prevent the problem at hand in the future,
> hence requesting that the decision to split the lowlatency into a lesser 
> flavor
> be reverted and have it built and treated as if it were the generic kernel
> since it is installed by default in an official flavor of Ubuntu on desktop
> systems. It is just clear to me that it truly does not get equal treatment,
> which confirms my fears, which is why I want the decision that was made
> reverted so that proper testing can proceed as it was before this change.

It was not a choice to split it, but necessity.
We had to split lowlatency into a separate build, as generic &
lowlatency from a single builder was unable to complete anymore due to
build-time, disk usage, and upload time.
It's either split builds, or no builds at all.
Thus a revert, would just cause generic & lowlatency fail to build
from source, always.

-- 
okurrr,

Dimitri

-- 
ubuntu-studio-devel mailing list
ubuntu-studio-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-studio-devel

Re: Lowlatency Kernel is behind in Ubuntu Studio

[Dimitri John Ledkov]

On Mon, 13 Mar 2023 at 17:54, Erich Eickmeyer  wrote:
>
> On Monday, March 13, 2023 10:29:50 AM PDT Dimitri John Ledkov wrote:
> > Ideally all flavours would be at 6.2 already, but due to various
> > reasons they are not.
>
> This is understandable and perfectly reasonable.
>
> > This is not unique to lowlatency flavour, and applies to kvm, azure,
> > raspi, and many more kernel flavours all of which are still on v5.19
> > in Lunar.
>
> My point is that lowlatency shouldn't be grouped-in to these flavors, but
> should be given higher priority and grouped-in with generic since it's still
> used in desktop systems by default and is directly affecting the testing of an
> official flavor of Ubuntu. This was one of the reasons we had to miss testing
> week because we didn't even have kernel parity.
>

Impossible. we run out of disk space and cannot complete the
lowlatency builds with generic any more. Thus it is now treated
exactly like all other derivatives.

We do not invest additional engineering & resource time, to prioritise
lowlatency flavour, over other flavours, which have more images and
higher usage.

> > We pushed 6.1 out, and migrated, on generic only, to migrate lots of
> > packages in proposed, specifically nvidia & everything entangled with
> > it, and thus unblock autopkgtesting of all the userspace packages
> > which were otherwise failing on v5.19. There is no intention to port
> > all flavours to 6.1.
>
> Again, this is one of the reasons we had do miss testing week among another

it was a mistake to skip testing week. you should have tested ubuntu
studio during the testing week like all the other flavours did. As
there are a lot of changes in lunar, that landed and affect ubuntu
studio. For example, all cloud images which use various cloud kernel
flavours, based on v5.19 did participate.

Can you explain who made the call to skip testing week? as to me, it
seemed, it's a requirement to release a flavour. Is studio going to
skip Lunar release?

> reason (two blockers this round). To not have kernel equality here could cause
> false positives in kernel-level testing. JACK and the audio stack, in
> particular, are directly affected by the kernel, and what might work in 6.1
> might not work in 5.19 with various devices. This could cause false bug
> reports and create a lot more problems for triage.
>
> > in Lunar, no further 6.1 builds will be done for any kernel flavour
> > for the time being. And v6.2 landing, across all flavours, is in
> > progress.
>
> Understandable. I'm just trying to prevent the problem at hand in the future,
> hence requesting that the decision to split the lowlatency into a lesser 
> flavor
> be reverted and have it built and treated as if it were the generic kernel
> since it is installed by default in an official flavor of Ubuntu on desktop
> systems. It is just clear to me that it truly does not get equal treatment,
> which confirms my fears, which is why I want the decision that was made
> reverted so that proper testing can proceed as it was before this change.

It was not a choice to split it, but necessity.
We had to split lowlatency into a separate build, as generic &
lowlatency from a single builder was unable to complete anymore due to
build-time, disk usage, and upload time.
It's either split builds, or no builds at all.
Thus a revert, would just cause generic & lowlatency fail to build
from source, always.

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: [ubuntu-studio-devel] Lowlatency Kernel is behind in Ubuntu Studio

[Dimitri John Ledkov]

Ideally all flavours would be at 6.2 already, but due to various
reasons they are not.

This is not unique to lowlatency flavour, and applies to kvm, azure,
raspi, and many more kernel flavours all of which are still on v5.19
in Lunar.

We pushed 6.1 out, and migrated, on generic only, to migrate lots of
packages in proposed, specifically nvidia & everything entangled with
it, and thus unblock autopkgtesting of all the userspace packages
which were otherwise failing on v5.19. There is no intention to port
all flavours to 6.1.

in Lunar, no further 6.1 builds will be done for any kernel flavour
for the time being. And v6.2 landing, across all flavours, is in
progress.

On Mon, 13 Mar 2023 at 17:02, Erich Eickmeyer  wrote:
>
> Hi everyone,
>
> I'm bringing this up as a matter of concern for the Ubuntu Studio daily
> images. I believe sometime after the release of 22.04, the lowlatency kernel
> was split from the build of the generic kernel. From what I understand it was
> to make the build process easier and shorter, but I could be misremembering.
> While that would make sense were the lowlatency kernel *not* being used on
> desktop systems by default, the reality is that it is being used as such.
>
> My main concerns at the time was that the lowlatency kernel would not have
> testing or quality control parity with the generic kernel. I have been assured
> multiple times that the quality controls have not changed, so this is not a
> quality control concern, so please do not misunderstand my concern here.
>
> What I'm looking at now is a situation where the daily builds of lunar for
> Ubuntu Studio are not at kernel parity with other flavors simply because it
> builds using the lowlatency kernel. For instance, I can look at any other
> flavor and see the 6.1 generic kernel whereas Ubuntu Studio is still sitting 
> at
> 5.19 lowlatency. This means my testers aren't even testing on something
> *anywhere close* to what the final kernel will be, which, albiet, is expected
> to be 6.2. However, to be unable to test in parity with the other flavors is
> highly disappointing and is a huge setback.
>
> So, please, rather than treating the lowlatency kernel as a secondary,
> derivative kernel, I would like to kindly request that it be brought back to
> an equal kernel and built with the generic kernel since it's still being
> installed in desktop systems by default so that it can be tested on live
> images. That way, we could test it correctly and Ubuntu Studio isn't left
> behind as it is currently.
>
> Thank you in advance,
> Erich
> --
> Erich Eickmeyer
> Project Leader - Ubuntu Studio
> Technical Lead - Edubuntu--
> ubuntu-devel mailing list
> ubuntu-de...@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
okurrr,

Dimitri

-- 
ubuntu-studio-devel mailing list
ubuntu-studio-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-studio-devel

Re: Lowlatency Kernel is behind in Ubuntu Studio

[Dimitri John Ledkov]

Ideally all flavours would be at 6.2 already, but due to various
reasons they are not.

This is not unique to lowlatency flavour, and applies to kvm, azure,
raspi, and many more kernel flavours all of which are still on v5.19
in Lunar.

We pushed 6.1 out, and migrated, on generic only, to migrate lots of
packages in proposed, specifically nvidia & everything entangled with
it, and thus unblock autopkgtesting of all the userspace packages
which were otherwise failing on v5.19. There is no intention to port
all flavours to 6.1.

in Lunar, no further 6.1 builds will be done for any kernel flavour
for the time being. And v6.2 landing, across all flavours, is in
progress.

On Mon, 13 Mar 2023 at 17:02, Erich Eickmeyer  wrote:
>
> Hi everyone,
>
> I'm bringing this up as a matter of concern for the Ubuntu Studio daily
> images. I believe sometime after the release of 22.04, the lowlatency kernel
> was split from the build of the generic kernel. From what I understand it was
> to make the build process easier and shorter, but I could be misremembering.
> While that would make sense were the lowlatency kernel *not* being used on
> desktop systems by default, the reality is that it is being used as such.
>
> My main concerns at the time was that the lowlatency kernel would not have
> testing or quality control parity with the generic kernel. I have been assured
> multiple times that the quality controls have not changed, so this is not a
> quality control concern, so please do not misunderstand my concern here.
>
> What I'm looking at now is a situation where the daily builds of lunar for
> Ubuntu Studio are not at kernel parity with other flavors simply because it
> builds using the lowlatency kernel. For instance, I can look at any other
> flavor and see the 6.1 generic kernel whereas Ubuntu Studio is still sitting 
> at
> 5.19 lowlatency. This means my testers aren't even testing on something
> *anywhere close* to what the final kernel will be, which, albiet, is expected
> to be 6.2. However, to be unable to test in parity with the other flavors is
> highly disappointing and is a huge setback.
>
> So, please, rather than treating the lowlatency kernel as a secondary,
> derivative kernel, I would like to kindly request that it be brought back to
> an equal kernel and built with the generic kernel since it's still being
> installed in desktop systems by default so that it can be tested on live
> images. That way, we could test it correctly and Ubuntu Studio isn't left
> behind as it is currently.
>
> Thank you in advance,
> Erich
> --
> Erich Eickmeyer
> Project Leader - Ubuntu Studio
> Technical Lead - Edubuntu--
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Possibility of accepting a network-based installer of Ubuntu as an official flavor?

[Dimitri John Ledkov]

Secureboot allows kexec, when using the recentish kexec_file_load syscall
which performs kernel image signature verification.

All of this just works under secureboot.

On Fri, 24 Feb 2023, 20:20 Aaron Rainbolt,  wrote:

>
> On 2/24/23 11:51, Dan Bungert wrote:
> >>> On Fri, 24 Feb 2023 at 04:54, Aaron Rainbolt 
> wrote:
>  I've seen more than one person annoyed by the fact that the mini.iso
>  netinstaller is no more.
>  The "flavor" would be able to be held in a
>  very small ISO file (preferably CD sized), and it would download and
>  install all of the packages that make up the Ubuntu system at runtime.
>  This would allow a user to install Ubuntu or any desired flavor
> thereof
>  using a single installation medium, rather than having to flash an ISO
>  every time they want to make a drive install a different flavor. The
> new
>  installation would be entirely up-to-date from the get-go, and it
> would
>  enable the use of existing small storage media for those users who
> don't
>  have sufficiently sized optical discs or flash drives.
> > Hi Aaron,
> >
> > As Lukasz mentioned, I've been looking at relevant things, and expect
> that we
> > can have the first version of ubuntu-mini-iso running this cycle.  I
> missed
> > feature freeze, so I'll be filing that exception :).
> >
> > Lukasz wrote a perfect summary of the work so far, so I'll quote it here:
> >>> The ubuntu-mini-iso is a small bootable iso that can be either
> >>> downloaded and used on a CD/USB-drive or even via UEFI HTTP that
> >>> brings up a dynamic TUI menu of what Ubuntu images you want to
> >>> download/install to your target system. It uses simplestreams to
> >>> select which images, so it'll be quite customizable regarding the
> >>> selection. The difference is that it then downloads the
> >>> iso-of-interest into memory and chain-boots into it, allowing the
> >>> installation of any image as one would normally do. This has some
> >>> limitations of course, since it needs sufficiently enough RAM.
> > So I think that will address much of what you were aiming for.
> >
> > Size: the bootleg builds I'm doing of this are around 140 MiB, I expect
> the
> > official builds to produce a similar answer.  It could potentially be
> smaller,
> > the size today is dominated by use of the existing Ubuntu initrd with a
> few
> > things added on top. (compare to the size of /boot/initrd.img)
> >
> > Download at runtime: ubuntu-mini-iso achieves this by presenting a menu
> of ISOs
> > that we could download, then with the user selection, reserving some
> memory,
> > downloading that ISO, and then kexecing to it.
>
> This makes good sense to me. The concern I'm noticing here is that
> Secure Boot activates a kernel lockdown mode that prohibits kexec. One
> workaround may be to have the user choose the release of Ubuntu to
> install at a GRUB menu so that a pre-existing kernel and initrd can be
> loaded, but this would bloat the ISO and complicate its use.
>
> Another possible solution might be to use mokutil to disable Secure Boot
> verification in the shim (essentially turning Secure Boot off without
> needing to get the BIOS involved), then rebooting the system. Then
> Secure Boot can be re-enabled with mokutil and then the ISO downloaded
> and kexec'd. When the user finishes installation and reboots, Secure
> Boot will be active again. This might complicate things with third-party
> drivers though.
>
> Perhaps we just live with no Secure Boot support?
>
> > ISOs in the menu: there is a casper hook that downloads simplestream
> json data
> > and hands it to the menu application, a small ncurses app that analyzes
> the
> > json, finds what ISOs to offer, and does so.  The user chooses an entry
> from
> > the menu, that info is handed back to the casper scripts, which download
> it and
> > we chain boot.
> >
> > That menu could be extended for Flavors support, perhaps conceptually
> similar
> > to how flavors are shown today on https://releases.ubuntu.com/.  The
> relevant
> > code is at: https://github.com/canonical/mini-iso-tools
> > It's not necessary to build an ISO to start playing with the menu, if you
> > download that, get the dependencies installed, `make run`, and you can
> see what
> > the menu looks like.
> >
> > If you're interested to help, Aaron, a good starting point would be to
> add
> > entries to
> https://github.com/canonical/mini-iso-tools/blob/main/json.c#L27 to
> > teach the menu how to read the simplestreams for the flavors.
> >
> > The existing menu can fit on a single screen, so if we start adding
> flavors I
> > think it will need some nested menu support, but that's achievable.
> >
> > I have done a hacked test run of having this new mini-iso chainboot to
> lubuntu
> > 22.04.2 and it all works fine.
> Nice, sounds awesome. Thank you for the info, and I'll see if I can hack
> on this at some point!
> > -Dan
>
> --
> Aaron Rainbolt
> Lubuntu Developer
> 

Re: How to ask for introducing a new package into main?

[Dimitri John Ledkov]

On Thu, 19 Jan 2023 at 17:05,  wrote:
>
> Hello,
>
> I checked the wiki and docu but couldn't find that process described
> somewhere.
>
> Is there a ticket system where I can open a wishlist-bug or something
> else to ask for introducing a new package into the "main" repository?
>
> "universe", launchpad or privat ppa are not an option because the
> package is still there.

I've read the whole thread, and I just want to reiterate - all default
ubuntu installations enable main, restricted, universe and multiverse
components of an Ubuntu Series, including release, security and
updates pockets (e.g. jammy jammy-security jammy-updates). All of
these are considered to be part of a given Ubuntu distribution
release. A package is available to be installed by default, if it
exists in any of those places. The distinction between the various
locations are in the grand scheme of things fairly minor, and are
policy driven.

There can be specific cases and instances where things are missplaced,
or have been removed and reintroduced. If you have specific packages
and examples of things unavailable, please point them out for further
investigation. Removal and reintroduction often happens when a package
fails to be compatible with a given Ubuntu release, in the specified
timeframe due to timed releases. But these are exceptions rather than
the norm.
-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Dimitri John Ledkov]

In such cases it is usually best to bump version number, and do a fresh
upload to lunar-proposed such that it is higher than any of (kinetic,
lunar).

Might make sense to still upload no change rebuild of dbus into
lunar-proposed.

On Tue, 22 Nov 2022, 18:45 Sebastien Bacher,  wrote:

> Hey there,
>
> Le 22/11/2022 à 18:21, Paride Legovini a écrit :
> > The dbus package has now need force-migrated from lunar-proposed to
> > -release (currently pending publication). This should fix the issue.
>
> I've stated that via chat but I still disagree that was the right thing
> to do. Without tests result how are we confident that the update isn't
> bringing a regression (one other update in lunar could create issue for
> the dbus in proposed)? Why didn't we fix the autopkgtest setup to use an
> lunar image or enable proposed to allow the tests to be correctly tried
> instead?
>
> Also are we confident that they aren't other packages in the same
> situations?
>
> Cheers,
> Sebastien
>
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: any reason for CONFIG_FUSE_FS=y

[Dimitri John Ledkov]

On Tue, 9 Aug 2022 at 18:29, Bernd Schubert  wrote:
>
>
>
> On 8/9/22 18:38, Dimitri John Ledkov wrote:
>
> >> We are in the process to upstream out changes. We got disrupted by other
> >> work for our main product but will continue to send new patches soon.
> >> However, as expected getting the patches upstream takes time, which is
> >> replacing the module might be helpful.
> >>
> >
> > You can open a bug report in launchpad and attempt to submit pull
> > requests to Ubuntu directly. We do carry many features and backports
> > of features that are in progress of making it into mainline, or are
> > being pulled from maintainer trees. See for example how many SAUCE
> > patches we carry for apparmor, intel cpu features, ibm power cpu
> > features and so on. Depending on how clean and maintained your fuse
> > tree is it might be suitable for inclusion in Ubuntu kernels ahead of
> > vanilla mainline.
> >
>
> Ah nice, for me that sounds great! Maybe that solves our issue! I was
> more expecting a "please open a request through a paying Ubuntu customer".

It's more of a send patches, or pull-requests to Public
kernel-t...@lists.ubuntu.com

But with appropriate messages headers / tags / buglink reference /
sign-off / parent tree etc.

See

https://wiki.ubuntu.com/Kernel/Dev/KernelPatches
https://wiki.ubuntu.com/Kernel/Dev/StablePatchFormat

And so on.

--
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Ubuntu initramfs (Was: Re: any reason for CONFIG_FUSE_FS=y)

[Dimitri John Ledkov]

Hi,

On Tue, 9 Aug 2022 at 18:20, Aaron Rainbolt  wrote:
>
> On my system, if the initrd isn't readable by the kernel, it results
> in a kernel panic. Is that to be expected despite inird-less boot? Or
> is that an indicator that at least Lubuntu (and probably Ubuntu
> Desktop) does use an initrd?
>

This is not a choice or a configuration, on machines that we can
guarantee initrd less boot we configure and do that. On machines where
we can't we set it up for booting with initrd. It is a transparent
boot optimisation. If your system boots one way or another, it is best
experience available for you.

But your individual setups and deployments do not represent the
overall spectra of Ubuntu usage, and relative % of boots that happen
one way or the other, even if all of your machines in your particular
deployment do one particular thing.

-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Ubuntu initramfs (Was: Re: any reason for CONFIG_FUSE_FS=y)

[Dimitri John Ledkov]

Hi,

On Tue, 9 Aug 2022 at 17:53, Richard Laager  wrote:
>
> On 8/9/22 11:38, Dimitri John Ledkov wrote:
> > The fast majority of Ubuntu installations boot without initramfs at
> > all.
>
> What makes you say this? Every Ubuntu system I've ever installed has an
> initrd.img-KERNEL_VERSION in /boot. In this context, I'm talking about
> systems installed using the stock installers (primarily server, but
> desktop was that way last I installed one using the stock installer).
>

We always generate initrd.img and use it as fallback if/when
initrd-less boot fails. The vast majority of Ubuntu boots are
successful without initrd, for example almost all Ubuntu Public Cloud
images.

-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: any reason for CONFIG_FUSE_FS=y

[Dimitri John Ledkov]

On Tue, 9 Aug 2022 at 16:24, Bernd Schubert  wrote:
>
> Hi Dimitri,
>
> On 8/9/22 15:54, Dimitri John Ledkov wrote:
> > Hi,
> >
> > On Tue, 9 Aug 2022 at 14:22, Bernd Schubert  
> > wrote:
> >>
> >> Hello,
> >>
> >> I would like to ask if there is a good reason Ubuntu builds fuse as
> >> statically into the kernel instead of using a module?
> >>
> >> Reason I'm asking is that we are currently working on a couple of fuse
> >> improvements and
> >>
> >> a) A static module makes development a bit harder, I cannot take any
> >> quickly installed ubuntu system and just load an updated module, but
> >> always need to install a non custom kernel package.
> >>
> >> b) Makes it impossible to provide the updated module as preview to our
> >> customers, without extra work like renaming all exported symbols and
> >> adding another /dev/something char device to replace /dev/fuse
> >>
> >> c) Makes it hard for us to support customer issues if a fuse issue
> >> should come up (rare case).
> >>
> >
> > Historically this is due to many bugs with dynamically loaded fuse. It
> > is an autoloadable module, thus userspace requesting fuse would
> > autoload it. then later one has to mount /sys/fs/fuse/connections
> > which is all racy. And in practice fuse is often always in use as
> > well. on my system it is common to mount things in nautilus with
> > fuse; lxd/lxc uses fuse; and keybase as well. Also my windows
> > partition is mounted with fuse (not sure why it's not using kernel's
> > module here).
> >
> > you can change ACL methods to not make /dev/fuse accesible; or change
> > udev rules not not even create it; then make your loaded module
> > provide /dev/fuse possibly under a different character device number.
> > You can use linux-backports infrastructure to create a
> > non-symbol-clashing fuse module (see how we build and vendor
> > backports-iwlwifi module). Or something similar.
>
> hmm ok, I would have added "modprobe fuse" in a initramfs script and so
> only users who manually unloaded the module would get issues?
>

The fast majority of Ubuntu installations boot without initramfs at
all. And doing modprobe during early boot would be racy.
Also there becomes a non-trivial upgrade issue, where we have to roll
out a change to modprobe fuse everywhere first, before being able to
change kernel config from y to m.
For no apparent benefit to fast majority of users - just introducing
race / requirement to modprobe fuse.

I will check if static nodes for fuse are pre-created even with the
module not loaded, in which case it might be relatively safish to make
it a module. But given that it is almost always used on both desktops
and servers, I find it counterintuitive to make it a module, and thus
have a tiny module penalty (in terms of bootspeed, loading it from
disk, and having it a module vs builtin).

In general, I do wonder if there are any better kernel mechanisms for
overriding built-in modules.

> >
> > But furthermore If you have an improved fuse module, can we not
> > make it available as part of stock Ubuntu or Ubuntu Pro? If it is
> > useful, and your customers are on Ubuntu, let's make it the best out
> > of the box experience.
> >
>
> We are in the process to upstream out changes. We got disrupted by other
> work for our main product but will continue to send new patches soon.
> However, as expected getting the patches upstream takes time, which is
> replacing the module might be helpful.
>

You can open a bug report in launchpad and attempt to submit pull
requests to Ubuntu directly. We do carry many features and backports
of features that are in progress of making it into mainline, or are
being pulled from maintainer trees. See for example how many SAUCE
patches we carry for apparmor, intel cpu features, ibm power cpu
features and so on. Depending on how clean and maintained your fuse
tree is it might be suitable for inclusion in Ubuntu kernels ahead of
vanilla mainline.

-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: About Upstream version choice

[Dimitri John Ledkov]

Heya,

On Tue, 9 Aug 2022 at 15:30, Mauricio Faria de Oliveira
 wrote:
>
> Hi,
>
> I'm not directly involved with this in general, but if I understand
> the question and some processes correctly:
>
> The package versions in a given Ubuntu release are (usually) a result
> of a Time Based Release process
> and what package versions are in Ubuntu's and Debian's package
> archives; and there are exceptions.
>
> Ubuntu releases under development import packages (without Ubuntu
> customizations) from Debian
> until the Debian Import Freeze date, and there are packages that are
> Ubuntu-only or replacements
> that thus aren't (imported) from Debian.  There may be more
> details/processes I'm not aware of.
>
> If you check the release schedule wiki page of, e.g., Focal [1], you
> may find some of that information
> and other related topics that might help as well. [2,3,4].
>
> And, of course, others here may correct me / add more information.
> Hope this helps!
>
> [1] https://wiki.ubuntu.com/FocalFossa/ReleaseSchedule
> [2] https://wiki.ubuntu.com/TimeBasedReleases
> [3] https://wiki.ubuntu.com/DebianImportFreeze
> [4] https://wiki.ubuntu.com/Releases
>

All of the above is correct, and then after we release we only do
targetted bug fixes following
https://wiki.ubuntu.com/StableReleaseUpdates process.

At the time when we froze/released Focal Fossa gcc-9.3.0 was the
latest stable release of gcc, which met all of the above timelines.

And we do not upgrade major version of the default gcc toolchain post
GA release, but we do issue bugfix SRUs for it.

Other compilers are available in Focal too, for example you can
install and use gcc-10 as well.

> On Fri, Aug 5, 2022 at 3:31 PM 罗瑶明  wrote:
> >
> > Dear ubuntu-devel-discuss,
> >   I have questions about how to choice or decide which Upstream 
> > version to use for a certain software in Ubuntu Derived series.
> > is there some guiding principle or refer documents. i.e.,why choice 
> > gcc-9.3.0 in the Focal Fossa.
> > 
> > 
> > thanks.
> >
> > --
> > Ubuntu-devel-discuss mailing list
> > Ubuntu-devel-discuss@lists.ubuntu.com
> > Modify settings or unsubscribe at: 
> > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
>
>
> --
> Mauricio Faria de Oliveira
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss



-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: any reason for CONFIG_FUSE_FS=y

[Dimitri John Ledkov]

Hi,

On Tue, 9 Aug 2022 at 14:22, Bernd Schubert  wrote:
>
> Hello,
>
> I would like to ask if there is a good reason Ubuntu builds fuse as
> statically into the kernel instead of using a module?
>
> Reason I'm asking is that we are currently working on a couple of fuse
> improvements and
>
> a) A static module makes development a bit harder, I cannot take any
> quickly installed ubuntu system and just load an updated module, but
> always need to install a non custom kernel package.
>
> b) Makes it impossible to provide the updated module as preview to our
> customers, without extra work like renaming all exported symbols and
> adding another /dev/something char device to replace /dev/fuse
>
> c) Makes it hard for us to support customer issues if a fuse issue
> should come up (rare case).
>

Historically this is due to many bugs with dynamically loaded fuse. It
is an autoloadable module, thus userspace requesting fuse would
autoload it. then later one has to mount /sys/fs/fuse/connections
which is all racy. And in practice fuse is often always in use as
well. on my system it is common to mount things in nautilus with
fuse; lxd/lxc uses fuse; and keybase as well. Also my windows
partition is mounted with fuse (not sure why it's not using kernel's
module here).

you can change ACL methods to not make /dev/fuse accesible; or change
udev rules not not even create it; then make your loaded module
provide /dev/fuse possibly under a different character device number.
You can use linux-backports infrastructure to create a
non-symbol-clashing fuse module (see how we build and vendor
backports-iwlwifi module). Or something similar.

But furthermore If you have an improved fuse module, can we not
make it available as part of stock Ubuntu or Ubuntu Pro? If it is
useful, and your customers are on Ubuntu, let's make it the best out
of the box experience.

-- 
okurrr,

Dimitri

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: VT console font

[Dimitri John Ledkov]

Hi,

On Mon, 18 Jul 2022 at 13:36, Daniel van Vugt
 wrote:
>
> I find myself increasing the VT console font size on practically all modern
> machines:
>
>sudo dpkg-reconfigure console-setup
>
> Is it perhaps time that Kinetic defaulted to a larger console font?
>

Is this upgraded machine, or fresh install? i think when we enabled
auto-detected high-dpi console fonts, we couldn't really do an upgrade
case as it was not possible to check/know if configuration was default
or manual.

Or do you want font even bigger, than what we boot into by default?

You can check stock behaviour by booting live usb stick and checking
how vt console font looks there for you.

-- 
okurrr,

Dimitri

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Version string to auto-sync an Ubuntu delta (maysync1 vs ~willsync1)

[Dimitri John Ledkov]

In the past, when we did uploads into Ubuntu that we want to autosync,
we simply used `+bN` as those are bigger than last debian source
version, and get autosynced.

It is a slight misnomer, as it is Debian's binNMU version number, but
that also means none of debian's source versions may ever use that
version, and the next debian upload is guaranteed to get autosynced.

I.e. we have often uploaded snapshots of debian's packaging VCS into
ubuntu as `+bN` given strong expectations that next debian upload will
include all of those changes.

-- 
okurrr,

Dimitri

On Wed, 25 May 2022 at 10:44, Lukas Märdian  wrote:
>
> Hi all!
>
> We had an interesting discussion with @enr0n and @brian-murray recently,
> about which version string to choose if we want an Ubuntu package
> (+delta) to be auto-synced, without the need for a manual merge. The
> results might be of interest for the broader Ubuntu developer community!
>
> The context was a sponsoring request for "gnudatalanguage" [0] but the
> outcome applies to and can be useful all over the archive.
>
> Initially a version of "1.0.1-3willsync1" was suggested, as the fix was
> already committed in Debian (Salsa) but not yet uploaded and we had some
> precedence about the "willsyncX" version in the archive [1]. We wanted
> it to sync automatically, as soon as the upload happens in Debian and
> therefore avoided an "ubuntuX" version string, as this would block an
> auto-sync [2]. BUT: "-3willsync1" > "-3ubuntu1" or another potential,
> future "ubuntuX" version, so if we'd need to add another patch, which
> might not necessarily auto-sync, we cannot override the "willsync1"
> version with an "ubuntuX" version. That's a problem and could lead to
> ugly version strings.
>
>
>
> We needed a version that does not contain the word "ubuntu", so it can
> be auto-synced, once the committed patch is uploaded into Debian. But at
> the same time we needed it to be bigger than the current version
> (1.0.1-3build2) and wanted it to be smaller than a potential, future
> "1.0.1-3ubuntu1" version. We came up with the following:
>
>
> 1.0.1-3build2 < 1.0.1-3maysync1 < 1.0.1-3ubuntu1 => 1.0.1-3maysync1
>
>
> So I'd like to suggest that anybody in a similar situation should be
> using a "maysync1" revision, in favor of "willsync1".
>
> Cheers,
> Lukas
>
>
> PS: "1.0.1-3~willsync1" might have been another option, but that's in
> conflict (i.e. smaller than) the current "-3build2" version string.
>
>
> [0] https://pad.lv/1973377
> [1] $ apt search . | grep "sync[0-9]"
> [2] https://git.launchpad.net/ubuntu-archive-tools/tree/auto-sync#n566
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: isc-dhcp: should we start phasing it out?

[Dimitri John Ledkov]

On Mon, 16 May 2022 at 20:22, Dan Streetman  wrote:
>
> On Mon, May 16, 2022 at 3:06 PM Steve Langasek
>  wrote:
> >
> > Hi Andreas,
> >
> > On Mon, May 16, 2022 at 02:34:30PM -0300, Andreas Hasenack wrote:
> > > Alternatives that come to mind are:
> > > - kea, of course (from ISC). dhcp server only.
> > > - dnsmasq (for small installations?), also server
> > > - systemd-networkd itself obviously does the client part
> > > - others?
> >
> > > isc-dhcp is such a classic that it is undoubtedly referenced in many
> > > places, so phasing it out might be difficult. On the server, Kea is
> > > not a drop-in replacement.
> >
> > As I mentioned at
> > , isc-dhcp is
> > no longer used out of the box as the DHCP client in Ubuntu, on either
> > desktop or server; server uses systemd-networkd's internal dhcp client
> > implementation, and desktop uses NetworkManager's.
>
> There are still users of 'dhclient', especially in some cloud environments.
>
> I suspect there will be a demand for it until someone updates systemd
> to be able to replace its functionality, with something like:
>
> $ networkctl dhclient eth0
>
> That would require an upstream discussion to figure out the exact
> usage and implement it, of course.

$ sudo netplan set ethernets.eth0.dhcp4=true; sudo netplan apply

Maybe we want to add `--apply` flag.

A networkctl command that creates an ephemeral .network file in /run
and reconfigures the link might be ok as well, similar to what
systemd-run does. But I very much prefer netplan.

>
> > The only reason that
> > isc-dhcp is still installed by default is for the initramfs: because we
> > support nfsroot, iscsi, remote disk unlock via dropbear, etc.
> >
> > Several suggestions of path forward on support for dhcp in the initramfs, in
> > no particular order:
> >
> >  - work with klibc upstream to extend ipconfig to be a suitable DHCP client
> >for the initramfs (requires DHCP support)
> >
> >  - migrate to a systemd-based initramfs everywhere, and use systemd-networkd
> >

We already have some netplan compatibility in our initramfs. At one
point I was experimenting to add full netplan & networkd to the
initramfs, but that side project stalled.

The largest chunks of work to switch to systemd-based initramfs would
be migrating installer initramfs features (casper &
cloud-initramfs-tools).

--
okurrr,

Dimitri

> >  - Repackage isc-dhcp to provide an 'isc-dhcp-initramfs' package that no
> >longer provides integration with the main system, so that it can continue
> >to be used for initramfs without having a large ongoing support surface
> >(but, this doesn't remove the need for security support)
> >
> >
> > > We also have the udebs, but with subiquity being the installer now we
> > > probably don't need to worry about these anymore?
> >
> > We definitely don't need to worry about udebs in future Ubuntu releases;
> > they aren't even built in Launchpad in the current series.
> >
> > > rdeps at 
> > > https://people.canonical.com/~ubuntu-archive/germinate-output/ubuntu.kinetic/rdepends/isc-dhcp/
> >
> > > Removing isc-dhcp would also allow us to reduce the need of old compat
> > > src:bind9-libs package, probably even drop it.
> >
> > Ugh
> >
> > > Could we perhaps start with phasing out the client, get its rdeps to
> > > use alternatives, and then stop building it, and eventually get to the
> > > server? This could be a lot of work, as I said, isc-dhcp is a classic,
> > > but if upstream is shifting its focus elsewhere, soon we will be
> > > alone.
> >
> > I think you'll find that phasing out the client is much more work than
> > phasing out the server, given the ways the client is integrated in other
> > packages.
> >
> > > Reverse-Depends
> > > * cloud-init
> >
> > Another example of client integration that's going to require attention;
> > cloud-init uses isc-dhcp-client to be able to query specific dhcp attributes
> > in early boot used to identify the cloud metadata service.
> >
> > > * dracut-network
> > > * isc-dhcp-client-ddns [amd64 arm64 armhf ppc64el s390x]
> > > * libguestfs0 [amd64 arm64 armhf ppc64el s390x]
> > > * netscript-2.4
> > > * network-manager [amd64 arm64 armhf ppc64el s390x]
> >
> > I think network-manager's dependency on isc-dhcp-client is vestigial.  You
> > will not find any instances of isc-dhcp-client running on an Ubuntu 22.04
> > desktop system.
> >
> > Cheers,
> > --
> > Steve Langasek   Give me a lever long enough and a Free OS
> > Debian Developer   to set it on, and I can move the world.
> > Ubuntu Developer   https://www.debian.org/
> > slanga...@ubuntu.com vor...@debian.org
> > --
> > ubuntu-devel mailing list
> > ubuntu-devel@lists.ubuntu.com
> > Modify settings or unsubscribe at: 
> > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
> --
> ubuntu-devel mailing list
> 

Re: pv (a pipeline progress indicator) in main?

[Dimitri John Ledkov]

During kernel package builds compressing debs often takes a very long time
without any feedback.

I was considering to start patching dpkg-deb to fork and compress things
via pv to get progress output in the build logs.

I can't remember where, but it would be nice for pv to output info after a
delay. I.e. if operation is quick don't bother, but after like 10s start
showing progress output would be nice.

On Fri, 22 Apr 2022, 17:58 Bryce Harrington, 
wrote:

> The Ubuntu Server team is looking at several potential items to promote
> to main, including cli admin tools that might have broad usefulness.
> One of these we're on the fence about and would like broader input.
>
> pv, 'Pipe Viewer' is a command line utility that essentially copies
> stdin to stdout, and displays an animated progress bar.
>
> Standard example is compressing a large file, e.g.:
>
>   $ pv Mail/spam.assassin  | gzip > /tmp/spam.gz
>   31.3MiB 0:00:01 [31.3MiB/s] [===> ]
> 31% ETA 0:00:02
>
> pv can also be used in the middle of pipelines (although since it
> doesn't know the stream's size it can't estimate % progress):
>
>   $ mysqldump -uroot -p database1 | pv | gzip -9 > database1.sql.gz
>   53.7MiB 0:00:01 [29.7MiB/s] [ <=>
>  ]
>
> Overview: https://catonmat.net/unix-utilities-pipe-viewer
> Man page: https://linux.die.net/man/1/pv
> LP page:  https://launchpad.net/ubuntu/+source/pv
>
>
> Googling indicates that pv comes up very commonly as a general purpose
> solution to displaying progress, although there do appear to be
> main-provided solutions for at least some common situations.  For simply
> copying files, there is already rsync which has --progress and --status
> options.  For creating tarballs, tar has a --checkpoint option, though
> it's not fancy.  For copying streams, dd is in main, which has a
> status=progress option that animates the bytes copied (but not %'s or
> visual bars).
>
> That said, pv looks like it would be a relatively light addition to
> main; it's written in C, appears to have an active upstream, and looks
> pretty self-contained.  A MIR for pv looks like it would be reasonably
> straightforward to file.  With it in main, other packages could rely on
> having it available for providing progress info, and would make it more
> at hand for scripting, tutorials/howto's, tech support, etc.
>
>
> Does this look useful enough to you that it should be made available by
> default?  Are there alternatives you feel would be better to look at?
> Or other considerations that need made before deciding?
>
> Thanks,
> Bryce
>
>
>
>
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Restoring debhelper dh5 compat for the LTS?

[Dimitri John Ledkov]

On Wed, 23 Mar 2022 at 18:50, Robie Basak  wrote:
>
> On Wed, Mar 23, 2022 at 02:58:27PM +, Dimitri John Ledkov wrote:
> > Please don't. There are many bugs in older compat level that may produce
> > debs no longer compatible with our OS. Especially around generated
> > maintainer scripts.
> >
> > It's best to upgrade packaging to compat level 13. Sounds like a long
> > overdue packaging upkeep.
>
> I understand the principle here, but it seems very late in our cycle to
> be suddenly finding ourselves with the task of doing this for the entire
> archive. What changed that suddenly makes this urgent for _this_ cycle
> in particular, apart from that someone noticed?
>
> I don't have a strong view on the technicalities here, but socially it
> seems pretty harsh to suddenly impose this on development teams who are
> already busy due to what amounts to an accident of timing for general,
> tech-debt type improvements rather than something that directly impacts
> user experience. It'd be nice to see a compelling reason to push for
> more late notice work that doesn't rely on the word "may".
>

I didn't mean to enforce or request an upgrade to 13. I wanted to
ensure it is known that re-introducing support in debhelper for
compats 5 & 6 is very risky, and potentially buggy and is not
something we can or should attempt to do.

Upgrading packaging from compat 5 to 7 should be fairly minimal, even
if upgrading to 13 would be best. Especially since compat level 7 has
been available for more than 14 years now.

I am now deeply worried that we have packages in the archive that
didn't get packaging updates for 14 years now.

Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Restoring debhelper dh5 compat for the LTS?

[Dimitri John Ledkov]

On Wed, 23 Mar 2022 at 15:02, Sebastien Bacher  wrote:
>
> Le 23/03/2022 à 15:58, Dimitri John Ledkov a écrit :
> > It's best to upgrade packaging to compat level 13. Sounds like a long
> > overdue packaging upkeep.
>
> Yes it's better, but it requires work, do we have the resources to deal
> with those, especially for universe?
>
> Also you claim it produces buggy debs but do we have any report of such
> problems or any way to grep the archive to figure out special cases we
> know about that would give a non working binary?
>

See bug reports against debhelper, reverts, and ultimate removal of
support for old compat levels.
Existing binaries built with old debhelper are fine. Building binaries
using latest debhelper but old compat levels are risky and produce
buggy binaries, which are hard to track down.
Debhelper upstream tried to keep working and improving new compat
levels, whilst maintaining support for old ones, and eventually ran
into the wall of not being able to reliable maintain that.

As part of the release process we perform archive wide rebuilds, and
fix all FTBFS for the release across the full archive. We can't
release Ubuntu without fixing those. Thus yes, there is an explicit
mandate, especially for the universe, to be buildable. It can be
resolved by fixing & upgrading packaging, or via removal of those
packages from the archive. These packages are likely to be
ubuntu-specific and/or niche leaf packages, not present in Debian.

Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Restoring debhelper dh5 compat for the LTS?

[Dimitri John Ledkov]

Please don't. There are many bugs in older compat level that may produce
debs no longer compatible with our OS. Especially around generated
maintainer scripts.

It's best to upgrade packaging to compat level 13. Sounds like a long
overdue packaging upkeep.

On Wed, 23 Mar 2022, 14:34 Sebastien Bacher,  wrote:

> Hey there,
>
> Checking the desktop section of the ongoing archive rebuild we have a
> bunch of ftbfs due
>
>  > dh: error: Compatibility levels before 7 are no longer supported
> (level 5 requested)
>
> That's because the newest debhelper upload removed compatibility for
> level 5 and 6
> https://salsa.debian.org/debian/debhelper/-/commit/a80ba4f797
>
> Those issues are easy to fix but I would prefer us to focus on spending
> our effort on more user impacting problems so I'm suggesting we revert
> the change for the LTS.
>
> How do other feel like about that?
>
> Cheers,
> Sebastien Bacher
>
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Proposal: revert recent debianutils changes for Jammy

[Dimitri John Ledkov]

Longer term it would be easier if we do a merge from debian now to
preserve just the `export NO_PKG_MANGLE=1` delta (if that is still
required), otherwise a forcesync.

-- 
Regards,

Dimitri.

On Mon, Jan 24, 2022 at 1:40 PM Robie Basak  wrote:
>
> On Wed, Dec 01, 2021 at 04:05:38PM +, Robie Basak wrote:
> > This has now landed in the release pocket. Thanks to all involved for
> > your help.
>
> You may have noticed that Debian CTTE have overruled some of these
> changes in the debianutils package in Debian and reverted them in Debian
> itself.
>
> tempfile in Ubuntu is on 5.5-1ubuntu1. Debian is now on 5.7-0.1
> (through a couple of yet-to-be-acknowledged NMUs).
>
> The changes in Debian so far are below. The reversion of which
> deprecation and restoration of tempfile are amongst the changes, but
> looking at the source they're done differently to how I did it (I used
> quilt patches).
>
> I wondered if I should bring Ubuntu back into sync. But I'm tempted to
> leave things as-is until after Jammy is released. We're not far from
> feature freeze now, and with Jammy being an LTS it might be an idea to
> leave things settled. In effect this would be applying a freeze to
> debianutils in Jammy early but maybe this is appropriate for us.
>
> Opinions? Is there anything in these two uploads, or upcoming uploads,
> that we really need for Jammy?
>
> ---8<---
> debianutils (5.7-0.1) unstable; urgency=medium
>
>   * Non-maintainer upload.
>   * Remove old and now conflicting manpage.  closes: #1004167
>
>  -- Bastian Blank   Sun, 23 Jan 2022 19:05:32 +0100
>
> debianutils (5.6-0.1) unstable; urgency=medium
>
>   [ Bastian Blank ]
>   * Non-maintainer upload.
>   * Implement CTTE ruling #994275.
>   * Revert deprecation of which (no. 2).
> closes: #993582, #993700
>   * Restore tempfile (no. 4).
>   * Revert move to /usr (no. 5).
> closes: #992481, #992615, #992639, #992649
>   * Remove not longer applicable NEWS entry.
>   * Force creation of translated man-pages to fix all build.
>
>   [ Johannes Schauer Marin Rodrigues ]
>   * also distribute *.po files and *.add files in dist tarball
> (closes: #1000866)
>
>  -- Bastian Blank   Fri, 21 Jan 2022 23:12:40 +0100
> ---8<---
>
> Robie
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Revisiting default initramfs compression

[Dimitri John Ledkov]

On Wed, 8 Dec 2021, 17:13 Julian Andres Klode, 
wrote:

> Hi all,
>
> some time ago, the default compressor for initramfs was changed
> from lz4 -9 to zstd -19. This caused significant problems:
>
> - it is very slow
> - it uses a lot of memory
>
> The former is a problem for everyone, the latter means that
> zstd just crashes on a Pi Zero.
>


When zstd -19 was chosen it was for the fastest bootspeed.

Overall at all compression levels zstd decompression CPU time is faster
than bootloader IO time. Meaning any gains in compression ratio improve
boot speed, even marginal between the highest levels. (One may argue that
marginal speed up will never be recouped because one will never reboot
enough times, but so far we have been putting high value on fast cold boot
speed).

Compressed initrd that contains compressed kernel modules and compressed
firmware is larger in size, leading to slower boot speed than current
implementation of compressing cpio of uncompressed files. (Nested
compression is bad). Thus in jammy initramfs-tools was updated to
decompress firmware & modules if they are stored compressed on disk prior
to including them in the initrd.

Failure to create new initrd is obviously critical and must be avoided.
Thus if zstd -19 is going to fail, it should fallback to something else.

Are you observing "it is very slow" from the point of view of stable
release or devel release?

In devel series one may be installing updates every 6 hours, with each one
of them triggering initramfs update, which gets overridden without ever
booting the initrd that was generated. Whereas stable updates do not
usually trigger initramfs updates as often, meaning it happens upon
security & kernel updates circa once every three weeks.

Do we want to slow down stable series boot speed, due to desire to shorten
the time to apply updates?

In the past we have also discussed forking initramfs update into a systemd
unit which one can limit resources too. Especially since it may not be
critical to complete that as part of the apt transaction, or need to
complete it successfully at all if previous initrd for a given kernel abi
exists or when instance boots without initrd anyway.

The installation of updates should be routine and a background task, rather
than something we optimise at the expense of reboot & boot performance.

We'd need to look at the whole cycle again of we want to optimize for
"discover, install, and reboot into all updates", which so far has not been
the overall KPI goal.

This is an analysis of what we have in terms of time spent,
> memory spent, and file size achieved, and where we can
> go from here.
>
> # Comparison of different compression levels
>
> ## Desktop (ThinkPad T480s, jammy)
>
> levelusertime   elapsed memory fileSize
> lz4 9.65s11.09s12M  64M
> -1  5.69s 6.99s24M  57M
> -6 12.59s 8.58s99M  47M
> -1219.85s10.82s   249M  41M
> -1971.29s26.95s   519M  35M
>
> -> I believe that somewhere around -12 is a decent
>compromise between size and speed.
>
> ## Pi 4 (arm64, focal)
>
> Times have been measured for mkinitramfs only. A full
> update-initramfs call spends much more time copying
> some firmware bits to boot partition with flash-kernel
>
> levelusertime   elapsed memory fileSize
> lz421.10s64.85s21M  29M
> -1 13.73s44.55s21M  27M
> -6 26.07s49.09s91M  24M
> -1248.18s54.67s   203M  22M
> -19   130.07s92.80s   350M  20M
>
> -> 6 is essentially free if the Pi 4 is idle. Nice.
> -> -6 is still 20% of total RAM of a Pi 0
> -> There's no meaningful difference between -6 and -12
>in terms of time elapsed. -6 uses 116% CPU, -12 uses
>145% CPU.
>
> ## Adaptive compression
>
> zstd also supports adaptive compression, compressing as hard as
> it can while not impacting I/O speed. So hardware with slow I/O
> like a Pi would compress harder to avoid idling.
>
> This is somewhat suboptimal with recent update-initramfs though,
> as it first writes the cpio archive to the disk and then compresses
> it rather than doing it in a pipe where that would be more
> meaningful.
>
> Question: Does zstd --adapt adapt to memory available?
>
> # Way(s) forward
>
> To remedy the issue the proposal is to build with
>
> - zstd -1 on hardware with 512 MB or less memory
> - zstd between -1 and -19 on other hardware
> - zstd -19 during image building
>
> Finding the right level between -1 and -19 is hard. The more
> cores you have, the less penalty you pay for higher level.
>
> Going for adaptive compression would remove the guess work, but
> will result in larger images on faster machines. Maybe that's
> fine, though - they probably have more space on /boot anyway?
>
> If we want to aim for 5% of total memory, we should probably
> aim for something like:
>
> -1  on <= 512MB
> -6  on <= 2 GB (or --adapt=min=1,max=6)
> -12 on the 

Re: Add ubuntu-advantage-tools to Recommends on ubuntu-minimal

[Dimitri John Ledkov]

On Thu, Nov 25, 2021 at 1:52 PM Mattia Rizzolo  wrote:
>
> On Thu, Nov 25, 2021 at 12:56:18PM +, Robie Basak wrote:
> > On Mon, Nov 22, 2021 at 05:19:38PM -0300, Lucas Moura wrote:
> > > We want to ask for opinions of this change to other Ubuntu developers, to
> > > see if we are not missing any other aspect around the original decision to
> > > include the package into *Depends*.
> >
> > Thank you Lucas for raising this here!
> >
> > Unfortunately I think that unless someone advocate for the case for this
> > here, this discussion can't go anywhere, and the status quo will remain.
> >
> > I'm saying this for the record: if those who want the change don't
> > participate, then it's unlikely to happen.
>
> FWIW, I'd also like this to be changed.
>
> As a very practical example of why: I find very annoying that by default
> on 14.04 it keeps bothering me that ESM has however many other updates
> available that -oh how unfortunate- I can't install, and what should
> have been the trivial way to avoid that to creep into my systems (i.e.,
> removing ubuntu-advantage-tools well before that change came) couldn't
> be done.
>

This is intentional, to ensure that we make users aware that there are
vulnerabilities out there that may be affecting them. One can access
ESM for free under certain terms. Once ESM ends, like it did for
Precise, we have made all ESM updates for Precise available publically
and archived on old-releases.ubuntu.com. Thus eventually everyone does
have access to them.

It is our commitment to be transparent and not hide problems, and
fight for the users to ensure they have ways to remain secure (upgrade
to a supported release or enable ESM). Whilst some users may find this
information redundant, many others may find it eye opening.

It really is very important, especially since a lot of systems get
broken simply due to lack of installing updates or timely upgrades.

Recently newer laws are getting passed that require one to disclose if
updates are available, for how long, and notify when they cease to be
provided. Although Ubuntu is nominally so far excluded from these, it
is prudent to comply with the spirit of those laws protecting and
informing users.

I see this as no different to how we default to applying security
updates, and informing users about the number of non-security updates.

Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Proposal: revert recent debianutils changes for Jammy

[Dimitri John Ledkov]

I have been playing whack-a-mole trying to fix usage of those two
commands in all the places. It will be a painful and long process, not
only because we need to merge changes from Debian, but because we have
Ubuntu-specific deltas that use those commands all over the place as
well.

I agree that it is unnecessary transition to be done for Jammy. We can
choose to schedule this transition after Debian in a post Jammy
release.

It is in no way an expression of opinion about this transition, purely
a choice to coordinate timing of it with our release schedules.

+1 from me please go ahead, I was contemplating to propose the same myself.

-- 
Regards,

Dimitri.

On Thu, Nov 25, 2021 at 12:53 PM Robie Basak  wrote:
>
> You may be aware of a couple of recent changes in debianutils in Debian:
>
> 1. The "tempfile" command has been removed.
>
> 2. The "which" command now prints a deprecation warning on every
> invocation.
>
> These have ramifications across the archive, and also outside the
> archive, as everything that relies on these commands need adjusting.
>
> This kind of big change is being done in the right place in Debian's
> release cycle - shortly after a release. But for Ubuntu, it's the
> opposite - we're a few months away from an LTS release.
>
> Risk 1: before everything is settled, we release an LTS that is
> unpolished with regards to these changes.
>
> Risk 2: the changes may prove unpopular with users. Given that these are
> deprecations coming from Debian, it seems odd for Ubuntu users to face
> this ahead of Debian and without appearing in our interim releases
> first. Debian may end up applying mitigations for specific affected user
> stories but we would be stuck with the behaviour defined at our LTS
> release time.
>
> Proposal: we revert these two changes in an Ubuntu delta on the
> debianutils package, and reconsider syncing back with Debian _after_
> Jammy is released.
>
> Then Debian can lead the way, and we won't get additional work ensuring
> that there are no user-facing warts ahead of Debian's schedule.
>
> Any objections to an upload to debianutils in Ubuntu reverting these two
> changes?
>
> Robie
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: package docker.io is broken

[Dimitri John Ledkov]

It says right in the installation log why it is incompatible with your system:

"""
The aufs storage-driver is no longer supported.
Please ensure that none of your containers are
using the aufs storage driver, remove the directory
/var/lib/docker/aufs and try again.
"""

The package prevents breaking your containers which are still
potentially using obsolete storage driver. Make sure you don't have
any containers using obsolete storage driver, and remove the directory
as directed above (which may DESTROY data). And try again.

-- 
Regards,

Dimitri.

On Thu, Nov 18, 2021 at 3:43 PM Tony Emma  wrote:
>
> Hi,
>
> This package "docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb" seems broken.
>
> My stack trace:
>>
>>
>> # sha256sum 
>> /var/cache/apt/archives/docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb
>>
>> 27879bc3cdcc6cf6322689bbf92c1a43ab5541346f55a27b604e456893f5a4e7  
>> /var/cache/apt/archives/docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb
>
>
>
>> # dpkg --debug=1000 -i 
>> /var/cache/apt/archives/docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb
>>
>>
>>
>> (Lecture de la base de données... 344400 fichiers et répertoires déjà 
>> installés.)
>> Préparation du dépaquetage de 
>> .../docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb ...
>> The aufs storage-driver is no longer supported.
>> Please ensure that none of your containers are
>> using the aufs storage driver, remove the directory
>> /var/lib/docker/aufs and try again.
>> dpkg: erreur de traitement de l'archive 
>> /var/cache/apt/archives/docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb 
>> (--install) :
>>  new docker.io package pre-installation script subprocess returned error 
>> exit status 1
>> Des erreurs ont été rencontrées pendant l'exécution :
>>  /var/cache/apt/archives/docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb
>
>
>>  # uname -a
>>
>> Linux 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 
>> x86_64 x86_64 GNU/Linux
>
>
>
>>
>>  # lsb_release -a
>>
>> No LSB modules are available.
>> Distributor ID: Ubuntu
>> Description: Ubuntu 20.04.3 LTS
>> Release: 20.04
>> Codename: focal
>
>
>  Thanks for your help
>
> --
> Best regards,
>
> Tony EMMA
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Feature Request: Remove fwupd in future versions of Ubuntu

[Dimitri John Ledkov]

Hi,

I sympathize with your experience. The firmware updates delivered via fwupd
come directly from the OEM of your hardware. In this case Dell. The
application of dell updates via fwupd, remote management consoles, or via
manually downloaded firmware updates images from their website are
identical. It is likely that you are experiencing hardware issue and should
contact Dell for support. They do test all firmware updates across all of
their SKUs before publishing. It might be a coincidence that firmware
update triggered the fault. It is possible hardware was already defective.

I hope that Dell will be able to help you soon.

Regards.

On Tue, 16 Nov 2021, 15:22 Youran Sun,  wrote:

> Dear Developers of Ubuntu,
>
> I am a several-year Ubuntu user and I install Ubuntu on almost every
> machine I have. I also keep advertising Ubuntu to my friends. However, some
> terrible thing happened yesterday.
>
> Yesterday when I ssh to my PC, which is a Dell Optiplex 7070, the motd
> recommends I update the firmware. I checked the details using `fwupd
> get-updates` as shown in the motd and the outputs seem attractive. So I
> ran `fwupd update`. The update went smoothly and finished successfully.
> Then I reboot. And BOOM! No Logo, no bios, nothing!
>
> Now I cannot even enter bios. The blink pattern of the power light shows a
> "bios checksum error". I have tried for the whole night but no process. It
> seems I have to post my machine to Dell for repair. And the charge may be
> enough for me to buy a new motherboard for this computer.
>
> We know that updating bios is risking and the price of failure is painful.
> How can Ubuntu seduce me to update it while it cannot guarantee success?
> From the perspective of user experience, I suggest Ubuntu not install
> `fwupd` as default, or at least mask `/etc/update-motd.d/85-fwupd`. For
> users who really need an update in bios, they can either install and run
> this program or install the bios using a USB stick, as Dell recommended.
>
> For updates on my poor machine, see Cannot even enter BIOS after update
> .
>
> Sincerely,
> Youran
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: OpenSSL 3.0 transition plans

[Dimitri John Ledkov]

On Mon, Oct 11, 2021 at 2:48 PM Simon Chopin  wrote:
>
> Hi Robie,
>
> Quoting Robie Basak (2021-10-11 12:39:00)
> > I think it's worth noting what happened with nodejs in Bionic:
> >
> > https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1779863
> > https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1794589
> >
> > Summary: nodejs incorporated the version of openssl it gets built with
> > into its ABI, causing incompatibility between binary modules built in
> > different places if they mismatch, contrary to ecosystem expectations.
> > Upstream therefore considers[1] the openssl version that must be used
> > "locked" for a particular nodejs version. But if we use the version
> > upstream wants, and that differs from our "default" version, then the
> > resulting co-installability conflict between the two -dev packages
> > results in users complaining about that instead.
> >
> > It might be worth someone looking into this early in order to try to
> > avoid or mitigate a recurrence of this kind of issue.
>
> (my apologies, this mail will likely contain quite a few links)
>
> I looked a little bit into this, and as of 8 hours ago, the embedded copy
> of OpenSSL has been updated to version 3.0.0[0]. They have an open issue
> tracking the OpenSSL 3.0 support situation[1], and their technical
> committee has a document specifiying which OpenSSL release is supported
> for a given NodeJS version[2].
>
> According to this comment[3] it seems they don't plan on supporting
> OpenSSL 3.0 in the 16.x branch, but rather in the 17.x which will have
> its first release next week according to their release schedule[4].
> Sadly, the new 17.x branch isn't planned as an LTS one.
>
> Looking inwards, we currently ship a NodeJS version based on the 12.x
> branch, and Debian seems to be planning[5] a transition towards the 14.x
> branch. None of which support OpenSSL 3.0.
>
> Unless I'm missing something, I see the following options, in no
> particular order:
>
> (a) Remove NodeJS from the archive. Had to be mentioned, but I don't
>   think it's realistic ;-)
>
> (b) Keep in sync with Debian, use the 14.x branch, but keep OpenSSL 1.1.1
> in the archive via a compat package.
> (b') The same but using the embedded copy of OpenSSL (if even possible?).
>

(b') has been done in the past in Debian, and can be done again in
Debian/Ubuntu.

I'm not sure how well extensions compile when one does that and if we
will still need to package nodejs-openssl headers somewhere. Doing
(b') imho is less liability than packaging and providing 1.1.1 in the
archive, as despite all pleas people tend to assume that they don't
need to do anything and can stick with 1.1.1 for another ten years
without doing any work to migrate to 3.0.0.

--
Regards,

Dimitri.


> (c) Use NodeJS v17.x in JJ (when it's out), with OpenSSL 3.0. This would 
> entail
> doing the transition on our own, and it basically would be EOL two
> months after the JJ release.
>
> (d) Track the NodeJS master branch in JJ and update NodeJS to the official
> version 18.0 a few days after our release of 22.04.
>
> (e) Use 16.x + OpenSSL3 patches. I'm not entirely sure whether this
> would create the same issues as mentioned by Robie, as the support
> for a linked 3.0.0 is documented in [2].
>
> I feel like (b) is our safest bet. If we go this route, we'll want to
> make sure that libssl-dev and libssl1.1-dev are coinstallable, as it was
> apparently a painpoint in the previous OpenSSL transition.
>
> I welcome any other options or perspectives on the issue :)
>
> Cheers,
> Simon
>
> [0]: 
> https://github.com/nodejs/node/commit/66da32c045035cf2710a48773dc6f55f00e20c40
> [1]: https://github.com/nodejs/node/issues/29817
> [2]: https://github.com/nodejs/TSC/blob/main/OpenSSL-Strategy.md
> [3]: https://github.com/nodejs/node/issues/40106#issuecomment-937718359
> [4]: https://github.com/nodejs/Release#release-schedule
> [5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989266#10
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: gnutls plans for focal LTS?

[Dimitri John Ledkov]

On Fri, Oct 1, 2021 at 7:02 PM John Cummings  wrote:
>
> Hello, does anyone know what the plans are for gnutls in Ubuntu 20.04.03 LTS 
> (focal fossa)? It is currently at 3.6.13, and I don't see an update in 
> focal-backports. The recent expiration of a root certificate used in older 
> Let's Encrypt cert paths has triggered a problem in this version, which is 
> fixed in 3.6.14. I see that this fix was backported to gnutls 3.5 in bionic:
> https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1928648
>
> Are there (normally?) plans to add 3.6.14 to focal/focal-backports, or to 
> backport this fix into a 3.6.13 update like was done for bionic?
>
> Thank you!
>

Note that thanks to the ca-certificates package update in focal
https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1944481
/ https://ubuntu.com/security/notices/USN-5089-1 gnutls operates
correctly with letsencrypt servers with either short or long chains in
all releases of Ubuntu, including Focal's version of the package.

You are correct that focal's version of gnutls is affected and this
may trip up again, whenever the next CA expires. I've added a target
focal series on the gnutls bug report, but not it is not time critical
to fix it at the moment.

Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Packaging policy discussion: After=network-online.target

[Dimitri John Ledkov]

On Thu, May 13, 2021 at 4:12 PM Steve Langasek
 wrote:
>
> Hi there,
>
> On Wed, May 12, 2021 at 05:52:07PM +1000, Christopher James Halse Rogers 
> wrote:
> > There's an nfs-utils SRU¹ hanging around waiting for a policy decision on
> > use of the After=network-online.target systemd unit dependency. I'm not an
> > expert here, but it looks like part of my SRU rotation today is starting the
> > discussion on this so we can resolve it one way or another!
>
> > I am not an expert in this area, but as I understand it, the tradeoff here
> > is:
> > 1. Without a dependency on After=network-online.target there is no guarantee
> > that the network interface(s) will be usable at the time the nfs-utils unit
> > triggers, and nfs-utils will fail if the relevant ntwork interface is not
> > usable, or
> > 2. With a dependency on After=network-online.target nfs-utils will reliably
> > start, but if there are any interfaces which are configured but do not come
> > up this will result in the boot hanging until the timeout is hit.
>
> > In mitigation of (2), there are apparently a number of default packages
> > which already have a dependency on After=network-online.target, so boot
> > hanging if interfaces are down is the status quo?
>
> From one of the comments in the bug report, I gathered that systemd upstream
> (who, specifically?) was taking a position that distributions should not use
> After=network-online.target.  I think this is entirely unhelpful; the target
> exists for this purpose, it is not required for systemd internally to get
> the system up but exists only for other services to depend on.
>
> There are risks of services not starting on boot because the network-online
> target is not reached.  However, that is not the same thing as a "hung
> boot", because other services will still start on their own, and things like
> gdm and tty don't depend on network-online.target, *unless* you're in a
> situation where you've introduced a dependency between the filesystem and
> network-online.  This is possible when we're talking about nfs, because the
> same system might both export nfs filesystems and mount them from localhost.
> But I'm not sure it should block this specific change.
>
> > The obvious thing to do here would be to follow Debian, but as far as I can
> > tell there is not currently a Debian policy about this - the best I can find
> > is an ancient draft of a best-practises-guide² suggesting that pacakages
> > SHOULD handle networking dynamically, but if they do not MUST have a
> > dependency on After=network-online.target
>
> > As far I understand it, handling networking dynamically requires upstream
> > code changes (although maybe fairly simple code changes?).
>
> It does require upstream code changes; not always simple.  And it's not
> always *correct* to make upstream code changes instead of simply starting
> the service when the system is "online"; you can find a number of examples
> in Ubuntu of services that it only makes sense to start once your network is
> "up" - e.g. apt-daily.service, update-notifier, whoopsie, ...
>
>
> There are issues with the network-online target, to be sure.  There is not a
> clear definition of the target, and there have definitely been
> implementation bugs in what does/does not block the target.  I've had
> discussions with the Foundations Team in the past about this but it has yet
> to result in a specification.
>
> My working definition of what network-online.target SHOULD mean is:
>
>  - at least one interface is up, with routes
>  - all interfaces which are 'optional: no' (netplan sense) are up
>- including completion of ipv6 RA and ipv4 link-local if enabled on the
>  interface
>  - there is a default route for at least one configured address family
>  - attempts to discover default routes for other configured address families
>have completed (success or fail)
>  - DNS is configured
>
> Thinks that must not block the network-online target:
>  - interfaces that are marked 'optional: yes'
>  - address sources that are listed in 'optional-addresses' for an interface
>  - default route for an address family for which no interfaces have
>addresses
>
> At least historically, neither networkd nor NetworkManager has fulfilled
> this definition.  It would be nice to get there, but the first step is
> having some agreed definition such as the above so that we can treat
> deviations as bugs.
>

If netplan.io can implement that would be nice. I.e. either
synthetically (i.e. by generating a service unit on the fly that calls
systemd-networkd-wait-online with extra arguments specifying all the
non-optional interfaces) , or by creating a new binary which is
"netplan-wait-online" which will be wanted by network-online.target
and perform all of the above.

> > It seems unlikely that, whatever we decide, we'll immediately do a full
> > sweep of the archive and fix everything, so it looks like our choice is
> > between:
>
> > 1. The long-term goal is to have 

Re: General mechanism to supply "rich history" to git-ubuntu

[Dimitri John Ledkov]

My preference would be to open up some refs of git-ubuntu repositories to
uploaders to be read

Such that one could created dgit like uploads and push the commit id, as a
ref, into the git-ubuntu repository directly. Aka
refs/heads/uploads/commit-hash as a branch.

Then upon import git-ubuntu could look up the Dgit header from the dsc, and
check if it knows about refs/heads/uploads/Dgit-header-from and reuse that
as part of the archive-matching set of branches.

Separately, I also want team uploads to be open to coredevs in the
git-ubuntu repositories, i.e. such that there is no need to create
~ubuntu-core-dev/ubuntu/+source/$package repositories and instead the main
git-ubuntu repository has refs/heads/teams/ubuntu-core-dev/* namespace
available for ubuntu-core-devs to push using per-ref ACLs in Launchpad.
This would finally enable to target merge proposals to git-ubuntu
repository, merge them, and for those commits to end up as part of the
git-ubuntu locked branches history via above /uploads/ namespace.

I do not want to maintain additional repositories, neither short or long
term, outside of the git-ubuntu ones. And keep a 1-to-1 mapping with them.

The goal is to allow merging merge proposals into the git-ubuntu repository
namespaced refs. And for them to end up in the git-ubuntu managed refs, if
and when, the matching upload of those changes is accepted in the archive.
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Updating SRU docs

[Dimitri John Ledkov]

Hi,

On Thu, Feb 25, 2021 at 8:18 AM Heather Lemon
 wrote:
>
> Hi,
>
> Could we also mention that when you're doing SRU verification, [Verification 
> Done], that it needs to be a comment and not edited in the description.
>

Do you want it to be changed in the SRU policy ?
https://wiki.ubuntu.com/StableReleaseUpdates#Verification

QATeam docs?
https://wiki.ubuntu.com/StableReleaseUpdates#Verification

Or the autogenerated comment that gets added by our scripts?
https://bazaar.launchpad.net/~ubuntu-archive/ubuntu-archive-tools/trunk/view/head:/sru_workflow.py#L131

Note that the comment on the bug should normally say things like:
> 'If this package fixes the bug for you, '
> 'please add a comment to this bug, mentioning the version of the '
> 'package you tested, what testing has been performed on the '
> 'package and change the tag from '


-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: autoinstall 20.04 with grub

[Dimitri John Ledkov]

This mailing list is not appropriate for your questions.

This mailing list is for developing the next release of Ubuntu itself.
Not support or discussions around the stable series.

Please join discourse.ubuntu.com to discuss Ubuntu Server installation
options in the server topic - https://discourse.ubuntu.com/c/server/17

There you will find this thread for example
https://discourse.ubuntu.com/t/netbooting-the-live-server-installer/14510

That has step by step guide on booting Ubuntu Server installer over
the network. Specifically the initrd needs to know how to bring up
network, i.e. "ip=dhcp" kernel cmdline option, and where to find the
matching ISO for the kernel/initrd pair one is trying to launch the
installer from i.e.
"url=http://releases.ubuntu.com/focal/ubuntu-20.04.1-live-server-amd64.iso;
if one is booting amd64 with the 20.04.1 media and got the
vmlinuz/initrd pair from it.

Please continue further discussions about the Ubuntu Server installer
on https://discourse.ubuntu.com/c/server/17



On Mon, Jan 18, 2021 at 6:04 PM Jerry Geis  wrote:
>
> I have a current system (grub boot) that I want to replace (its not ubuntu 
> but centos).
> So there  is no USB and no CD.
>
> I am trying to find what to put on the grub linux kernel line to boot up and 
> install everything over teh network.
>
> I have autoinstall working from the USB - but I need it working from 
> "network" only.
>
> I tried this:
> append  initrd=/casper/initrd debian-installer/language=en fsck.mode=skip 
> splash autoinstall ds=nocloud-net;s=https://myip/autoinstall/
>
> but it did not work.  it tries to find the cd at /dev/sr0 and never does of 
> course.
>
> Is there something else I need to specify on the line ?
>
> Thanks
>
> Jerry
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss



--
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Ubuntu 20.04 kickstart

[Dimitri John Ledkov]

live supports autoinstall with simple yaml files to describe the
install which are a lot more simple than either kickstart or preseed.

Have you looked into
https://ubuntu.com/server/docs/install/autoinstall and does that at
all fit your needs?

Alternatively if you have more than 3 servers to provision, have you
tried MAAS? https://maas.io/ it's a small snap that allows
programmatically deploy, and redeploy, any Ubuntu any CentOS etc. In a
reproducible way.

On Wed, Jan 6, 2021 at 8:04 PM Jerry Geis  wrote:
>
> I had to change to legacy CD as live and desktop did not seem to support 
> kickstart.
>
> Can this be re-considered. With the CentOS 8 issues happening - more people 
> might be looking at switching - and kickstart is something they use?
>
> Seems as though kickstart was getting fazed out.
>
> Thanks
>
> Jerry
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss



-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Should one be able to install with only release + -security enabled?

[Dimitri John Ledkov]

On Wed, Nov 25, 2020 at 2:59 PM Nish Aravamudan
 wrote:
>
> Hi!
>
> I have been testing a network-isolated Ubuntu mirror inside our network and I 
> am trying to understand if what I envision should work or not.
>
> In particular, I am trying to minimize how much review is needed for package 
> updates, so I would like to just include the release and security pockets. 
> However, I am finding a few package updates (in Bionic in my case, but I 
> think Focal may also have this problem) that only have fixes in the -updates 
> pocket. This prevents installation from succeeding with preseed.
>
> So far, I have seen apt-setup, but debootstrap and base-installer both need 
> some adjustment for my test environment.
>
> Should we require -updates as well?

Actually it's the security pocket that is optional. It is a fast track
to access SRUs that happen to also contain security fixes at the
fastest speed possible, with automatic download & upgrades by default
via a direct connection to security.ubuntu.com.

When a new security update is prepared, it is based on package version
in updates; security; or release pocket in that order.

Because security update is mandatory to install, and it must not
regress any fixes that already were present in either
updates/security/release.

And then the security update is published into both updates & security
pockets on archive.ubuntu.com & mirrors, as well as onto
security.ubuntu.com host. As it must supersede everything.

When mirroring, we recommend for people to mirror release & updates
pockets. And we advise people to keep security.ubuntu.com
$suite-security archive config as is.

This way all machines can access security updates via a separate
endpoint directly. This insures that if the private mirror is lagging,
the critical security updates still get through to the end-users.

If you must mirror security.ubuntu.com $suite-security, please ensue
it is a separate mirror too. Such that resiliency remains to access
security-updates even if the stock mirror for updates is down for
maintenance.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Private home directories for hirsute onwards

[Dimitri John Ledkov]

On Thu, Nov 26, 2020 at 2:31 AM Alex Murray  wrote:
>
> setfacl -m u:libvirt-qemu:rx $HOME
>

Similar to above for qemu are there similar setfacl commands, would
something similar be also needed for:
- sshd user to access ~/.ssh/authorized_keys , or nothing needed there?
- in GNOME making ~/Public public?
- giving access to ~/public_html for the www-data user?

If yes, then what are the commands?

I like this approach of selective and explicit setfacl commands to
grant ACLs on per-usecase basis. This is inline with modern ways of
managing permissions.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: systemd PathExists triggers

[Dimitri John Ledkov]

Conditions* are unaffected at all, and are unrelated to the issue at
hand whatsoever.

On Thu, 1 Oct 2020 at 08:33, Christian Ehrhardt
 wrote:
>
> On Wed, Sep 30, 2020 at 7:43 PM Brian Murray  wrote:
> >
> > Recently there was a systemd change regarding PathExists for
> > systemd.path units that made systemd behave the way it was documented to
> > work. The effect being that if the PathExists or PathExistsGlob
> > condition is met the service is continually started which may not be the
> > behavior some packages expect as it didn't used to work that way.
> >
> > As an example apport's autoreport service uses PathExistsGlob to check
> > for a .crash file and then calls whoopsie-upload-all which used to run
> > once when the .crash file appeared. As of Groovy whoopsie-upload-all was
> > called continuously thereby causing systemd to use 100% of your CPU[1].
> >
> > While this is resolved for apport's case, the Foundation's team thought
> > it was worth mentioning as other package's services may also be
> > impacted.
>
> Thanks Brian for your warning!
> I see in your list some .path files e.g. the aforementioned
> apport-autoreport.path
> that I understand if using a PathExist* will have changed. Of those the list 
> [2]
> contains 9 to look at.
>
> I beg your pardon, but the list is vast due to including any
> "Type=oneshot" as well.
> And from the summary you gave I don't see why those are included as well.
> Looking at the first in the list that probably everyone has installed I see:
> a)
> /lib/systemd/system/alsa-restore.service there are no associated .path
> files, only
> ConditionPathExists in the unit itself - are those also affected?
> b)
> Looking slightly further /lib/systemd/system/apt-daily.service even
> has no associated
> .path nor any *Exist* statement.
>
> My question would now be - are those files of type (a) or (b) all also
> affected in
> some way or is the "critical list" actually much shorter? I was wondering if 
> you
> could help to clarify before everyone that feels responsible for a package 
> with
> a file in that list is diving into and re-understanding the case.
>
> Thanks in advance,
> Christian
>
> > An archive search[2] (thanks Seth!) was done of services which are
> > Type=oneshot or utilize a PathExists or PathExistsGlob. Briefly scanning
> > the list the following packages should probably be looked at:
> >
> > mandos-client
> > ostree
> > python-diskimage-builder
> > ubuntu-report
> >
> > [1] http://launchpad.net/bugs/1891657
> > [2] https://paste.ubuntu.com/p/k5Gj7rQ9FX/
> >
> > Cheers,
> > --
> > Brian Murray
> >
> > --
> > ubuntu-devel mailing list
> > ubuntu-devel@lists.ubuntu.com
> > Modify settings or unsubscribe at: 
> > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
>
>
> --
> Christian Ehrhardt
> Staff Engineer, Ubuntu Server
> Canonical Ltd
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Launchpad builder VMs upgraded to bionic

[Dimitri John Ledkov]

On Wed, 16 Sep 2020 at 11:35, Colin Watson  wrote:
>
> On Wed, Sep 16, 2020 at 10:22:05AM +0100, Dimitri John Ledkov wrote:
> > chroot builders do call ~= `sudo chroot` at some point.
>
> The main work is done by sbuild, which uses schroot, not "sudo chroot".
>
> (We used to use sbuild's sudo session mode, but I changed that to
> schroot in 2017.)
>
> > would changing limits.d + adding pam_limits to sudo pam config be
> > enough for launchpad-buildd?
>
> I think schroot does support PAM.  However, I don't think it's good
> design for launchpad-buildd to rely on PAM, since after all it's not
> doing any kind of interactive authentication.  I would rather not pursue
> this option any further.  There are other less complicated ways to set
> resource limits.

Are we using PAM today already? Cause on riscv64 builder I started to see

E: PAM error: Authentication token is no longer valid; new one required
Chroot setup failed
E: Error creating chroot session: skipping openssl

from time to time. Or is it just a timing issue?

-- 
Regards,

Dimitri.


buildlog_ubuntu-groovy-riscv64.openssl_1.1.1f-1ubuntu4_BUILDING.txt.gz
Description: application/gzip
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Compiling system for Ubuntu

[Dimitri John Ledkov]

Hi,

May I ask you why are you trying to rebuild systemd?

Note that CVE-2020-1712 is fixed in Ubuntu, in all series that it
affects, including 18.04 see
https://people.canonical.com/~ubuntu-security/cve/2020/CVE-2020-1712.html


On Sat, 12 Sep 2020 at 18:15, rafi Moor  wrote:
>
>
>
>
>
> Hello,
>
>
>
> I’m trying here after getting no answers in Ubuntu forums.
>
>
>
> I have hard time compiling some Ubuntu packages from source.
>
> I now try to compile systemd. On Ubunu 18.04 I've used apt source to get the 
> source that is supposed to include Ubunu patches. After compilation, I 
> replace libsystemd-shared-237.so with the one I've compiled. Programs that 
> are linked with this shared object complain about reference to undefined 
> symbol sd_bus_enqueue_for_read. using readelf I can see that the original 
> library has this symbol but the new one doesn't. I've tried to apply 
> CVE-2020-1712-2.patch but then the compilation fails on missing function 
> bus_message_ref_queued(). This function is included in systemd version 246 
> but not in 237 which is the version on Ubuntu 18.04.
>
> How can I compile systemd so that I get files identical to those of Ubuntu 
> 18.04?
>
> Thanks
> Rafi
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss



-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Launchpad builder VMs upgraded to bionic

[Dimitri John Ledkov]

On Wed, 9 Sep 2020 at 23:25, Steve Langasek  wrote:
>
> On Wed, Sep 09, 2020 at 10:33:00AM +0100, Dimitri John Ledkov wrote:
> > > Failing that, can somebody advise on whether there's an appropriate way
> > > to configure this in an image without having to maintain a fork of
> > > systemd?  The workflow here is that we consume Ubuntu cloud images and
> > > make a few small changes to them, mostly things like installing
> > > launchpad-buildd, before publishing them to Glance for use when starting
> > > new builder VMs.
>
> > I have not tried this, but i think one should be able to create a
> > snippet in /etc/security/limits.d/ with like
>
> > * soft memlock unlimited
> > * hard memlock unlimited
>
> > Or to the appropriate value of 64*1024 instead of unlimited.
>
> Which should only take effect for things which are part of PAM sessions that
> have invoked pam_limits.  I don't think this would be true for the builders?
>

chroot builders do call ~= `sudo chroot` at some point. But looking at
pam configs, sudo doesn't invoke pam_limits (which is interesting,
because su does)

would changing limits.d + adding pam_limits to sudo pam config be
enough for launchpad-buildd?

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Launchpad builder VMs upgraded to bionic

[Dimitri John Ledkov]

Hi,

On Tue, 8 Sep 2020 at 17:49, Colin Watson  wrote:
>
> On Tue, Sep 08, 2020 at 12:57:28PM -0300, Guilherme Piccoli wrote:
> > Hi Colin et.al., first of all thanks for the builder update and
> > heads-up! We've noticed a failure in building cryptsetup from source,
> > reported in LP #1891473 [0]. The reason for such failure is detailed
> > in the LP, but a summary is:
> > - cryptsetup might make use of mlock() syscall and so, restrict its
> > memory usage by the memlock limit (ulimit -l), if it succeeds the
> > mlock() call;
> > - in Xenial, the memlock limit was extremely low, so mlock() failed
> > and the cryptsetup test hereby failing (luks2-metadata validation)
> > succeeded, since cryptsetup continues with no locked memory;
> > -in Bionic, such limit was bumped to 16M [1], and cryptsetup succeeds
> > in the memory locking procedure, but...the limit is low enough in
> > order the luks2-metadata-validation-4M exceeds that and fails - worth
> > to notice that chroot/containers inherit those limits from host;
> > - in Focal such a limit was quite increased (to 64M), so the tests do
> > not fail anymore.
> >
> > In my understanding, we have 2 ways of resolving that:
> >
> > (a) We could bump the memlock limit in PPA builders to 64M (to match
> > real-life cases of Focal, specially useful for Focal packages being
> > built);
>
> The simplest and IMO best way to do this would be to SRU the systemd
> change that bumped it to 64M [1] to bionic; we'd then pick that up in
> the natural course of events by way of new cloud image builds.  Has that
> been considered?  It feels as though what's happened here is simply that
> the Launchpad build farm upgrade has surfaced a bug in bionic, so the
> most appropriate thing to do would be to fix it in bionic.
>

Upstream it was done as part of larger changes to add BPF
functionality. But it is worth-while to pick up just the limit bump
there.

> Failing that, can somebody advise on whether there's an appropriate way
> to configure this in an image without having to maintain a fork of
> systemd?  The workflow here is that we consume Ubuntu cloud images and
> make a few small changes to them, mostly things like installing
> launchpad-buildd, before publishing them to Glance for use when starting
> new builder VMs.
>

I have not tried this, but i think one should be able to create a
snippet in /etc/security/limits.d/ with like

* soft memlock unlimited
* hard memlock unlimited

Or to the appropriate value of 64*1024 instead of unlimited.


> [1] https://github.com/systemd/systemd/commit/91cfdd8d29
>

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu 18.04 and "sig_hashalgo: md4"

[Dimitri John Ledkov]

Hey,

linux kernel upstream has changed how signatures look like in
v5.2-rc1, and only kmod 27 learned how to parse them. But bionic ships
kmod 24, meaning with hwe / cloud kernels, the information printed by
e.g. modinfo is incomplete.

Normally bug reports should be opened in launchpad, i have done so now
at https://bugs.launchpad.net/ubuntu/+source/kmod/+bug/1894611

If this is time critical for you, you could install kmod from focal on
your bionic system (or use it from chroot).

On Sun, 6 Sep 2020 at 01:40, Jeffrey Walton  wrote:
>
> Hi Everyone,
>
> Ubuntu 18.04's modinfo looks like it is subject to
> https://bugzilla.redhat.com/show_bug.cgi?id=1320921 and
> https://bugzilla.redhat.com/show_bug.cgi?id=1490975.
>
> $ modinfo crypto_simd
> filename:   /lib/modules/5.3.0-66-generic/kernel/crypto/crypto_simd.ko
> license:GPL
> srcversion: 701320EC07F22E0D8427859
> depends:cryptd
> retpoline:  Y
> intree: Y
> name:   crypto_simd
> vermagic:   5.3.0-66-generic SMP mod_unload
> signat: PKCS#7
> signer:
> sig_key:
> sig_hashalgo:   md4
>
> A quick Google search did not pick up a similar Ubuntu bug report,
> like Red Hat's 1320921 and 1490975.
>
> It looks like the fix was checked in about a year and a half ago. Also
> see https://lwn.net/Articles/779249/.
>
> Would it be possible to pick up the fix?
>
> Thanks in advance.
>
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss



-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Track clarification in UbuntuSeededSnaps Policy

[Dimitri John Ledkov]

Currently https://wiki.ubuntu.com/UbuntuSeededSnaps does not specify
anything about tracks.

Currently the status quo is to seed the default-track of the snap, aka "latest".

Recently we have been approached by LXD upstream to use a different
"LTS" track of the seeded lxd snap which has longer support timeframes
and closer matches the timeframe of the Ubuntu Branch the seeded snap
tracks.

I would like to request additions to "Channel availability" section
something along the lines of

"The default track is used for seeding, which usually is called
latest. Publishers may request seeding from a different track, if it
better matches longevity of a given Ubuntu Series. Example: request to
use lts track for Ubuntu 20.04 LTS release."

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: chromium-browser epoch bump for transitional package?

[Dimitri John Ledkov]

Hi,

On Wed, 12 Aug 2020 at 13:32, Robie Basak  wrote:
>
> In doing SRU reviews today, I came across LP: #1889106 which is a
> request for a no-change rebuild to bump the version so it beats the
> versions presented in previous releases.
>
> The problem is real, but it seems suboptimal to me to keep fixing it
> this way. We'll need to keep SRUing chromium-browser to Focal (and later
> releases presumably). Every time we do that, all users (who have the
> transitional package) get yet another update to download, even if it is
> small. And every time it takes both Olivier's and the SRU team's time.
>
> Would an epoch bump be a better option here? Debian doesn't currently
> carry a package with that name, so we're already diverged enough that I
> don't forsee a problem from there.

lxd used epoch too, to ensure that deb2snap transitional package is
always higher whichever backports are uploaded as debs into previous
series.

https://launchpad.net/ubuntu/+source/lxd

So +1 from me.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance report

[Dimitri John Ledkov]

On Tue, 4 Aug 2020 at 10:54, Michael Hudson-Doyle
 wrote:
>
> Hi all,
>
> I noticed golang-gopkg-square-go-jose.v2 times out on armhf, filed 
> https://github.com/square/go-jose/issues/326.
>
> Then I looked at the icu transition.
>
> 0ad fails to build due to gcc-10, I found the fix for this upstream and 
> backported it.
>
> I retried some ucto builds which appeared to have run before a build 
> dependency had been built (maybe fallout from the build farm outage). And 
> retried "frog" builds when these completed.
>
> doxygen autopkgtests are failing because the json-c in proposed has moved its 
> Doxyfile.
>
> multipath-tools has britney complaining about impossible depends -- turns out 
> its udebs are still in main but its dependencies are now in universe. 
> Apparently a bunch of udebs turned up on component mismatches and got 
> demoted, but shouldn't have. They got repromoted again and if they appear on 
> mismatches again someone should debug it :) (or mass demote all udebs to 
> universe or stop building or at least publishing udebs or or ...)

I'm not sure.

kpartx-udeb & multipath-udeb themselves are up for demotion. Unless
the reports I'm looking at are old. So it shouldn't be a problem
Or as you say repromotion was incomplete.

Imho we should just demote all of udebs to the universe en masse.

>
> I noticed that udisks2 was failing autopkgtests, found a bug about this, 
> found the bug had a comment to a potential fix upstream so I applied the 
> patch, tested locally and uploaded it.
>
> I tricked xnox into uploading s390-tools-signed, which should unstick 
> s390-tools.
>
> frr is failing autopkgtests, and has the most unhelpful autopkgtests I've 
> seen in a while (I have no idea what the tests are expressing). The package 
> has no rdeps and is the last thing keeping json-c in proposed so maybe 
> removal is appropriate for now. I filed a bug for this: 
> https://bugs.launchpad.net/ubuntu/+source/frr/+bug/1890259
>
> libyang is failing to build on riscv64 because of symbols file differences in 
> libyang-cpp1. I don't understand how to fix this, other than by removing the 
> symbols file entirely.
>
> dee's tests are timing out on riscv64 :(
>
> libreoffice's tests seem to be very flaky on armhf.
>
> cmake-extras autopkgtest was failing, which I fixed and also submitted 
> upstream https://github.com/ubports/cmake-extras/pull/14
>
> doxygen's tests fail with the json-c in proposed: it builds the documentation 
> for json-c in an autopkgtest but json-c has moved it's Doxyfile into a 
> different directory. Easy to fix once json-c has migrated.
>
> Cheers,
> mwh
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: New coreutils release 8.32

[Dimitri John Ledkov]

Hi,

On Sat, 18 Jul 2020 at 19:09, Nicholas Guriev  wrote:
>
> Dear Ubuntu developers,
>
> GNU coreutils 8.32 have been released on March 6th, 2020, yet the
> package is not updated in groovy. The new version has enhanced support
> of file creation time in stat(1) and introduced the "--time=birth" sort
> rule for ls(1). It would be great to have these features in the coming
> Ubuntu release. Please take a look at my merge request LP#1888046.
>
> https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1888046
>
> Debian revision of the package has not migrated to testing yet due to
> build failure on ARM64. Is there a chance to get the update for AMD64 at
> least?
>
> https://buildd.debian.org/status/fetch.php?pkg=coreutils=arm64=8.32-2=1592917386=0
>

There are patches applied to coreutils to Ubuntu, thus it will need a merge.

Separate architectures are not updated separately. Packages must match
across architectures, and not regress in the support. Ubuntu supports
7 architectures, and coreutils must be build on them all.

The issue might be related to the kernel headers shipped vs expected,
and the problem may or may not reproduce on Ubuntu, as our kernels are
different. It doesn't seem like a hard thing to fix on arm64.

-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Dependencies of ubuntu-desktop

[Dimitri John Ledkov]

On Fri, 17 Jul 2020 at 21:23, Andrei Rybak  wrote:
>
> Hello,
>
> I like to use apt and aptitude for my Ubuntu installation. I was
> surprised to find out today, that the packages ubuntu-desktop and
> ubuntu-desktop-minimal do _not_ depend on the package ubuntu-minimal.
> Description of the package ubuntu-minimal claims:
>

ubuntu-minimal is a very small subset of Ubuntu Project that is shared
across all our chroots, containers, builder chroots. All packages in
ubuntu-minimal are Priority important and are always present and
installed on any of our flavours.

ubuntu-minimal by itself cannot boot. It does not ship either
bootloader, kernel, or systemd. It can only be used as a chroot
really. It does however ensure that one can download, authenticate,
and install updates.

> > Description: Minimal core of Ubuntu
> >  This package depends on all of the packages in the Ubuntu minimal system,
> >  that is a functional command-line system with the following capabilities:
> >  .
> >   - Boot
> >   - Detect hardware
> >   - Connect to a network
> >   - Install packages
> >   - Perform basic diagnostics
> >  .
> >  It is also used to help ensure proper upgrades, so it is recommended that
> >  it not be removed.
>
> Could someone please clarify why don't the packages ubuntu-desktop and
> ubuntu-desktop-minimal depend on ubuntu-minimal?  Or, in more general
> terms, what are the packages required for the "Truly Minimal"™ desktop
> installation of Ubuntu?  Would Ubuntu boot to desktop if I remove
> ubuntu-minimal, but keep ubuntu-desktop-minimal?

Removing ubuntu-minimal is not advised.

Your system will no longer be able to authenticate and install any
package updates as that's the seed that pulls in ubuntu-keyring and
apt, among other things.

-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Installing python-is-python3 should affect pip as well.

[Dimitri John Ledkov]

On Fri, 10 Jul 2020 at 00:14, Boris Verkhovskiy  wrote:
>
> When I install python-is-python3, python becomes python3 but I still
> have to type pip3. I think either python-is-python3 should make pip
> into pip3 (which might be surprising to some, since pip3 is installed
> separately) or I would like a pip-is-pip3 and pip-is-pip2 package.
>

I agree. I have opened
https://bugs.launchpad.net/ubuntu/+source/what-is-python/+bug/1887098
to track this. Please subscribe to be notified of the outcome.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: default algorithm in package zram-config 0.5

[Dimitri John Ledkov]

On Mon, 8 Jun 2020, 20:42 Mitch 74,  wrote:

> Hello,
>
> Considering that now lz4 is by default enabled in kernel, wouldn't it be
> better to use it as a compression algorithm in zram instead of lzo?
>

Can you benchmark the performance?

When changing initrd compression we have done extensive benchmarking to
insure we see improvement in the wall clock bootspeed. A data driven
decision can be made here too.

>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

+1 maintainance

[Dimitri John Ledkov]

finishing off boost1.71 transition
boost1.67 -> boost1.71 was started last cycle, but there were a few
packages still using boost1.67 in focal. Notably those that were still
using boost-python2. Uploaded: regina-normal uwsgi libpwiz ompl
freeorion mongo-cxx-driver-legacy frogatto innoextract. This completes
transition to boost1.71 and drops boost-python2, such that only
boost-python3 is now available.

Normally I look at oldest packages stuck in update-excuses. Not sure
what happened to the "days" counter, because I am sure some of them
have been stuck for multiple cycles now. Some of them are now more
urgent as they ftbfs and are holding up transition to libgcc-s1.

unanimity
stuck forever, FTBFS, removed from Debian
Opened an RM bug report to drop it from groovy-proposed. When and if,
it is fixed in Debian, it will be synced over again. Until them AA
should remove it. Also added update-excuse tag on the bug, such that
update-excuses documents this triange.
https://bugs.launchpad.net/ubuntu/+source/unanimity/+bug/1882059

ubuntu-app-launch url-dispatcher and the three indicators
Unity8 is removed from the archive. Three indicators have codepaths
for Unity8 DE. They use url-dispatcher. Which uses ubuntu-app-launch.
Which FTBFS in groovy due to glib depreceations and unity.h headers
doing weird redefines on glib macros which now fail. Drop those code
paths from indicators, to ensure we can kick
ubuntu-app-launch/url-dispatcher from the archive. If and when,
something equivalent is re-introduced in the archive, and it all
builds from source, these codepaths can be reintroduced back in. (e.g.
if there is lomiri-app-launch at one point).

singularity-container
Try to improve buildability, document next steps in a bug report
tagged ftbfs & update-excuse

fftw3
Cheat, fixup the fixup of the autopkgtest fixup to stop building fftw3
mpi based packages on i386. This should reduce the scope of packages
built on i386 significantly.

uwsgi-plugin-luajit
I think built against old uwsgi, retried the build record
uwsgi-plugin-php
forcesync to get it to build
... and uwsgi & friends should migrate now

boost1.67 transition done, it can now be removed from Ubuntu.

--
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: +1 maintenance report

[Dimitri John Ledkov]

On Tue, 26 May 2020 at 11:02, Michael Hudson-Doyle
 wrote:
>
> Hi,
>
> I've spend today working on +1 maintenance 
> (https://wiki.ubuntu.com/PlusOneMaintenanceTeam, 
> https://wiki.ubuntu.com/PlusOneMaintenanceTeam/Status).
>
> We have a few transitions ongoing (gsl, hdf5, perl, ...) which are showing 
> signs of all getting entangled together.
>
> I re-uploaded haskell-hmatrix-gsl which had been uploaded too early (before 
> gsl has been published on riscv64). It built fine.
>
> I found some more blockers for the gsl transition: 
> https://people.canonical.com/~ubuntu-archive/transitions/html/html/auto-gsl.html.
>  The cpl-* ones are semi-typical mysterious failures for a new port, probably 
> down to bugs in our emulated builders. The ea-utils one is a bit of a mess 
> though, in some ways: it should never have built in the first place! There is 
> a package (pegjs) in focal and groovy release that Provides: nodejs. This 
> means that things that transitively build-depend on nodejs don't end up in 
> depwait like they should. There is a pegjs in groovy proposed that fixes 
> this, I guess it needs to be forced to migrate and any binaries this makes 
> uninstallable removed (it's not like packages which depend on nodejs and are 
> erroneously installable are actually going to _work_).
>

That's a nice triange of nodejs / pegjs. Did you open an RM bug report
against pegjs, and subscribe ~ubuntu-archive to it, such that they can
process the binary-only removal on riscv64?

As documented at
https://wiki.ubuntu.com/UbuntuDevelopment/PackageArchive#Removing_Packages


> I fixed pbsuite's autopkgtest and sent a patch off to Debian, this has 
> migrated and been applied to the Debian repo already.
>
> I looked at the pbcopper ftbfs on armhf and ppc64el. Debian has a report 
> about the armhf failure and the packaging repo on salsa already has changes 
> to disable the build on 32 bit architectures so I guess we should follow 
> that. The ppc64el failure was strange and I spent probably too long coming to 
> the conclusion that it's a bug in the "simde" header library Debian uses to 
> compile the x86-intrinsic-using code on other architectures: 
> https://github.com/nemequ/simde/issues/325. I've uploaded a workaround and 
> sent it to Debian.
>
> I'm doing this again tomorrow, I expect I'll mostly keep on picking at 
> proposed migration.
>
> Cheers,
> mwh
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Help understanding the package set we need to maintain for partial i386

[Dimitri John Ledkov]

On Mon, 18 May 2020 at 20:25, Steve Langasek  wrote:
>
> The relevant germinate output is
> https://people.canonical.com/~ubuntu-archive/germinate-output/i386.groovy/i386+build-depends
>
> This shows the source packages that are in the set, as well as why they're
> pulled in.
>
> rdma-core is there as a dependency of openmpi, which is a dependency of
> mpi-defaults, which is a build-dependency of boost.
>

I would love to drop boost on i386, however, it seems like a very
small subset of boost is actually pulled in on i386.

Is it safe for me to reduce boost to the set of packages that are
listed in the first column? Cause then boost can potentially drop
python & mpi libraries on i386, and the openmpi build-dep.

And I guess that will make openmpi / rdma-core / mpi-defaults to drop
out of i386 seed, no? and will allow to drop those deltas?

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: deb2snap transitional packages and channel tracking

[Dimitri John Ledkov]

On Fri, 8 May 2020 at 15:00, Julian Andres Klode
 wrote:
>
> I've been reviewing some of ack's changes for the maas deb2snap
> transitional package, and in the latest merge request
>
> -
> https://code.launchpad.net/~ack/ubuntu/+source/maas/+git/maas/+merge/383411
> -
>
> a topic came up that I don't think we've discussed before.
>
> Currently, the maas deb installs maas 2.7/stable/ubuntu-$release
> (or other versions of maas instead of 2.7).

Why would deb2snap transition ever exist for stable/ubuntu-20.10 or
later? Given that it should be just a snap in 20.10 and that's it. And
in 20.10 and later, installing 20.10 should not follow the
ubuntu-xx.yy branch either.

For example, when people select to install snaps in subiquity
snapstore screen, we don't follow ubuntu-xx.yy branches either.

Upgrades from focal should be dropping /ubuntu-xx.yy branch tracking
too. Cause maas is not a seeded snap installed by default.

juju doesn't have latest/stable/ubuntu-20.04 branch either, why should
maas have it?

(lxd & desktop snaps are different because they are seeded, maas is
closer to juju => unseeded servery/devopsy software).

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Ubuntu Focal Fossa hangs while systemd is setting up on ARM64

[Dimitri John Ledkov]

On Fri, 8 May 2020 at 01:56, Suniel Mahesh  wrote:
>
> Hi All,
>
> I have a board based on Rockchip RK3399 64-bit SOC based on ARMv8A.
> The board can boot from the following devices: Micro SD, EMMC, USB, NVMe SSD.
> I have installed Ubuntu focal fossa with LXDE Display manager(built a 
> headless image
> using Buildroot, downloaded focal-base-arm64.tar.gz and installed all the 
> necessary
> packages with the help of chroot).
> I am using mainline Linux kernel 5.5.10 from kernel.org and mainline u-boot 
> 2020.04.
>
> The board boots fine with ubuntu focal fossa installed on Micro SD, emmc, 
> USB.When the
> board is booted via NVMe SSD(this NVMe SSD is connected via PCIe on to the 
> main board),
> the board loses power(powered off) automatically during the boot process. 
> This happens once
> the kernel loads and it hands over the boot process to systemd and systemd 
> trying to fully load
> ubuntu user space.
>
> In the process of identifying what is causing the problem, I found out that 
> systemd-udevd is the one.
> when loading rules from udev/rules.d, some of them are creating a problem.
>
> so, removed all the rules apart from: 50-udev-default.rules, 60-drm.rules, 
> 90-console-setup.rules,
> 60-block.rules60-serial.rules, 99-systemd.rules. (I know dy doing so, 
> initialization is not properly done)
>
> By doing the above change, I am able to get a basic command prompt. It says 
> the LXDE display manager
> is started but I couldn't get any display on my monitor connected via hdmi.
>
> Can any one please comment on this issue and suggest how to fix the problem. 
> I have attached the hang log.
>

[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.

It seems to me that either your kernel config didn't compile required
kernel modules, or you don't have them installed correctly. And given
that you are not using Ubuntu kernel it is hard to debug this.

Instead of using your self-build kernel, can you please try using
Ubuntu kernel ? i.e. v5.4 that we ship for arm on focal?

If you really want the vanilla kernel, instead of building yourself,
can you please try mainline kernel builds as provided by the Ubuntu
Kernel Team at https://kernel.ubuntu.com/~kernel-ppa/mainline/ ?
you will find mainline builds of all upstream kernel releases there
including v5.5 series. v5.5.10 is old they have v5.5.19 prebuilt
there, as well as the v5.6 & v5.7-rc builds.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Desktop installer is outdated

[Dimitri John Ledkov]

On Sun, 3 May 2020 at 07:34, Haug Bürger  wrote:
>
> Hi,
>
> I just tested the latest 20.04 release in the hope that the installer
> improved. It did not improve. The desktop installer really needs work.
>
> It prefers plain text vs encryption which is not appropriate these days
> and makes Ubuntu insecure. You have to choose extra options to get an
> encrypted setup. If yo do so, it is not possible to create a setup which
> uses multiple disks.

Which layout do you expect to be done, when trying to do both
encryption and multiple disks?

Today, we create luks and create LVM inside that. If you want, you can
add luks on additional drives, and add them as PVs to your LVM as
well. So it is possible to do this as a post-install task. I'm not
sure how to design, or explain what happens when you do that. As one
will be promoted to unlock each encrypted drive separately.

> A different issue is the plain text /boot partition required. This is
> also insecure and unnecessary. This partition reserves fixed space for
> the Kernels, causing issues if to small or wasting space if to big. The
> installer allows it to be any size and doesn't propose a size. Since
> GRUB can boot LUKS devices this is unnecessary.

Unfortunately this is not true. We default to the stronger LUKS2 which
the current grub shipped in 20.04 has no support to unlock. grub only
can unlock the significantly less secure LUKS1 which we no longer
recommend for people to use.

Instead of relying on encryption, we instead use modern firmware
features of ensuring Secureboot & Measured Boot & Lockdown. The only
bootloaders and kernels you  can boot, are those that are chained to
Canonical Master CA UEFI offline certificate, and by default only
signed kernel modules can be loaded. Thus although /boot is not
encrypted, it is impossible to boot untrusted artefacts off it. If one
has TPM one can take further attestation measures to prevent kernel
cmdline being modified too. In the context of enforced secureboot &
enforcing signed kernel modules, what security issues do you see with
unencrypted /boot ?

> The third major issue the missing support for file systems supporting
> snapshots.
>

Desktop installer offers LVM & ZFS installation options, with
snapshots integration in apt and backup software out of the box. Are
snapshots as provided by zfs or lvm not sufficient for you?

> Linux itself supports all of the mentioned short comings. It is possible
> to create encrypted multi disk setups. It is also possible to boot
> directly from the encrypted partition. It is possible to use for example
> BTRFS as root file system, gaining compression and snapshots. It is
> possible to have a swap file on a BTRFS partition. Everything is
> available and the installer should be able to glue it together.
>
> With ZFS on the doorstep it is time to renovate the installer to support
> the new features of modern file systems and bring security i to up to date.
>

Instead we integrated ZFS into our desktop installer, which does
support encryption, and is superior to btrfs in our opinion. Why use
btrfs, when zfs is offered out of the box?

> My question is. Who is in charge for the installer?
>

Ubuntu 20.04 LTS Desktop installer offers the features you deem
essential, is something was not clear in the UI for you to discover
them?

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: groovy pre-open analysis | missing dists/groovy/cnf/

[Dimitri John Ledkov]

On Mon, 27 Apr 2020, 20:19 Colin Watson,  wrote:

> On Sun, Apr 26, 2020 at 09:36:01PM +0100, Dimitri John Ledkov wrote:
> > c-n-f service probably doesn't know about groovy yet. It is deployed
> > as a service on that autopkgtest juju environment. Not sure which
> > release, or how it gets distro-info-data updated to start generating
> > newer c-n-f data. Maybe it should be somehow "copied" up by launchpad
> > upon archive opening.
>
> I think it makes more sense to just get that service updated fairly
> early (it's in https://wiki.ubuntu.com/NewReleaseCycleProcess FWIW).
> Copying those files as part of series initialisation would be
> unnecessary extra code, since it isn't absolutely necessary for them to
> be copied in order for the series to work.
>

However, we have autopkgtests that test c-n-f data. Thus it would be nice
if it was in place ahead of autosyncs & autopkgtests running for the new
series.

Regards,

Dimitri.

>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: groovy pre-open analysis | missing dists/groovy/cnf/

[Dimitri John Ledkov]

On Sun, 26 Apr 2020 at 13:18, Julian Andres Klode
 wrote:
>
> I did some work to get python-apt in with groovy added to the
> distribution metadata, so we don't end up with a lot of regressions
> like last cycle.
>
> * Various packages needed retrying with new debootstrap, as they
>   missed the groovy link:
>
>   - retried piuparts with [python-apt, debootstrap] triggers
>   - retried livecd-rootfs with [python-apt, deboostrap] triggers
>   - retried sbuild against ['courier/1.0.6-1build3', 
> 'debootstrap/1.0.123ubuntu1', 'devscripts/2.20.2ubuntu3']
>
> * Uploaded apport, switching test dependency from unavailable 
> python-twisted-core
>   to python3-twisted. Probably needs to be SRUed to focal too, as
>   twisted version seems to be the same
>
> Anyway, the reason I'm really writing this email:
>
> * The archive does not have Commands files in dists/groovy/cnf, which
>   causes command-not-found to be broken, and its autopkgtest to fail.
>
>   The tests passed initially, so I guess they were copied over from
>   focal but got deleted later on.
>

c-n-f service probably doesn't know about groovy yet. It is deployed
as a service on that autopkgtest juju environment. Not sure which
release, or how it gets distro-info-data updated to start generating
newer c-n-f data. Maybe it should be somehow "copied" up by launchpad
upon archive opening.


-- 
Regards,

Dimitri.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: nvidia-340 incapable of single user mode in 20.04

[Dimitri John Ledkov]

On Mon, 20 Apr 2020 at 20:53, Jack Howarth
 wrote:
>
>   I am finding on a 2008 MacPro with GTX680 that the installation of the 
> nvidia-340 package under Ubuntu 20.04 prevents single user mode boots from 
> working. While the nvidia-340 driver works fine from a normal boot, when 
> 'single' is added before 'quiet' in the grub kernel arguments, the boot 
> produces a black screen that never returns the expected single user mode 
> prompts.
>  A parallel test with current Ubuntu 18.04 with either the stock 
> nvidia-340 340.107-0ubuntu0.18.04.4 package or the 
> nvidia-graphics-drivers-340 340.108-0ubuntu0.18.04.1 show both of them can 
> successfully boot into single user mode.
> Jack

nvidia-340 is a very old version of nvidia.

20.04 LTS has: nvidia-driver-390, nvidia-driver-435, nvidia-driver-440.

Can you please use Super to search and launch "additional drivers"
select 440 driver and install and try that one? It is the recommended
version of nvidia on 20.04 LTS. Or whichever is highest and supports
your nvidia card.

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Proposal to update UbuntuSeededSnaps Policy

[Dimitri John Ledkov]

I would like to request the following updates to the UbuntuSeededSnaps
Policy https://wiki.ubuntu.com/UbuntuSeededSnaps

7. Bases
Seeded snaps which use a base, must use "base: core18" or higher.
Existing snaps are grandfathered, but are encouraged to update to
"base: core18" or higher.

8. Contact
Contact field must be set in the Snapstore to a URL where bug reports
and support requests can be filed. If a public tracker is not
available, email contact address may be used, but is discouraged.
These are used by apport / ubuntu-bug tools to guide users to file
issues. Examples:
https://bugs.launchpad.net/subiquity/+filebug
https://github.com/snapcore/core20/issues/new

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: UBUNTU ARM64 DEB PACKAGES on DEBIAN RELATED VERSION

[Dimitri John Ledkov]

All of Ubuntu is available for arm64 & armhf

Why bother with Debian, if all of Ubuntu is available anyway?!

On Mon, 23 Mar 2020, 19:28 Onur GURSOY,  wrote:

> Hello Everyone,
>
> First of all you're great team. you done great jobs.
> I'm lovers of arm and you're supporting arm platform.
> I'm very appreciated to you.
>
> I'm wondering one thing nowadays, I'm interested in ubuntu packaging for
> ar,64 platform. I'm inspecting your package and your .dsc file.
> *Your ubuntu arm64 packages are working on debian ?*
> I mean, your freeradius_3.0.16+dfsg-1ubuntu3.1_amd64.deb is working on
> DEBIAN buster/sid.
> In general, you are doing something for ubuntu for packaging ?
> *I can say, Most of ubuntu packages are fully **compatible with debian
> buster/sid or something like that version?*
>
> Many thanks,
> for your help,
> With best regards,
>
> --
> Onur GÜRSOY
> R Engineer in Embedded Systems
> Master Student at Gebze Institute Of Technology
> Department Of Electronic Engineering
> GSM : 0(545) 764 7653
> e-mail: onurgursoyg...@gmail.com
> --
> Ubuntu-devel-discuss mailing list
> Ubuntu-devel-discuss@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
>
-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: ICU version in Ubuntu 20.04

[Dimitri John Ledkov]

On Thu, 6 Feb 2020 at 03:23, Dimitri John Ledkov  wrote:
>
> On Fri, 10 Jan 2020 at 12:26, Shane Carr  wrote:
> >
> > Dear Ubuntu developers,
> >
> > I'm a member of the ICU TC (International Components for Unicode).  
> > Developers frequently get ICU from the apt-get package "libicu-dev".
> >
> > We have a special ICU release coming out in March/April, ICU 66.  This 
> > release is "special" because it has an emphasis on stability and 
> > compatibility with ICU 65, released in October, except that it also 
> > includes the latest Unicode 13 standard, which will be released around the 
> > same time.  Our plan with ICU 66 is so that platforms like Android can 
> > adopt this release in a relatively late stage with minimal disruption.
> >
> > I wanted to ask whether Ubuntu 20.04 would also consider picking up ICU 66 
> > as libicu-dev.  Since it is an LTS release, users would benefit from having 
> > Unicode 13 available in their server applications over the lifetime of 
> > Ubuntu 20.04.
> >
> > Shane
>
> boost1.71 is ongoing at the moment but progressing well.
> After that, I will perform icu upgrade to 65.
> April is too late, and so is March. In the future, please release in
> January to be included in the Ubuntu LTS release.
>
> Depending on how well transition to 65 goes; and how early you can
> release 66; it may or may not make it into 20.04 LTS.

As mentioned above that is still plan of record.

Since previous communication icu transition to 65 has been started on
the 10th of Feburary and, and is now fully staged in focal-proposed
which was achieved on Monday, but has not yet migrated due to all of
the dependencies not yet being ready.

There are issues with icu 65 compatibility with gnustep-base:

PREEURO used in tests, yet dropped by ICU -
https://github.com/gnustep/libs-base/issues/103

Numeric sort fails to work (i.e. order 11 to be higher than 2) -
https://github.com/gnustep/libs-base/issues/104

I think the first one needs to be resolved in gnustep test-suite,
whilst it would be appreciated some input on
https://github.com/gnustep/libs-base/issues/104 as to possibly why
NSNumericSearch test cases fail (ucol_setAttribute(coll,
UCOL_NUMERIC_COLLATION, UCOL_ON, );)

So icu upgrade to 65 will happen.

I have not yet started evaluating 66preview1 release or submitting
FeatureFreezeException.

Whilst ICU API/ABI stability will be helpful, some of the software
packages re-export icu APIs/ABIs whilst encoding the major version
number of icu in the symbols name, thus unfortunately overall it is
not quite a drop-in upgrade in the distribution as a whole.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Sugar desktop on Focal

[Dimitri John Ledkov]

On Fri, 14 Feb 2020 at 16:28, Erich Eickmeyer  wrote:
>
> Sorry to interject, and with all due respect, but at this point it's pretty 
> late in the game to expect anything to get done in Debian in time for feature 
> freeze/Debian import freeze in just under two weeks.

Apart from that I am both a Debian Developer, who can upload things
into Debian with QA and Python team hats, or NMUs, and Ubuntu Core Dev
to sync them to Ubuntu simultaneously. Thus can sponsor things from
anywhere to anywhere really.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Sugar desktop on Focal

[Dimitri John Ledkov]

Hi,

On Fri, 14 Feb 2020 at 11:31, James Cameron  wrote:
>
> G'day,
>
> Thanks for offering to be a "point of contact for upstream developers
> to reach Ubuntu developers."
>
> I'm an upstream developer for the Sugar desktop.
>
> Sugar isn't in Focal at the moment, due to Python 3 transition and
> Debian packages not ready.
>
> If any Ubuntu developers would like to fix that, I've a reprepro package
> repository with enough to get Sugar working well on Focal;
>
> http://dev.laptop.org/~quozl/.us/

Is this a request for sponsorship into Debian or into Ubuntu or both?

That repo has a lot of packages, do you have at least a list of source packages

I see for example higher versions than in your repository, shipped in
debian experimental. I.e.  sugar 2.0.1-1~exp1 is that not ready? or
what is it? Should 0.116 be packaged instead? Or like should we sync
sugar stuff from experimental?

I'd rather prefer getting everything working with python3 correctly in
either debian experimental or debian unstable and syncing from there.
Is there anything stopping from dropping python2 support in unstable?

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Auto-transition trackers

[Dimitri John Ledkov]

In debian, they have a thing called auto-transition tracker which
attempts to identify a change of library ABIs and automatically create
a ben transition tracker.

I have started to manually run that, and commit the trackers to our instance.

https://people.canonical.com/~ubuntu-archive/transitions/

At first It identified a few tiny (~10) transitions, which were
partially incomplete. And I uploaded them straight away.

I have now did a new run, and there were a few transitions dropped
(migrated) and a few new ones added.

Now the interesting bit about the auto-* transition trackers are those
that are 100% complete, yet not migrating => it means that set of
packages is most likely not-installable as a whole or has autopkgtest
regressions. Thus they are very good targets for investigation as a
whole. As one might be able to fix a few regressions and get 10+
packages migrate from -proposed to release pocket.

I hope we can drive all auto-* transition trackers to completion soon enough!

If you want help, guidance or sponsorship in completing any of the
auto-* transitions to migrate, feel free to reach out to me.

The exact way i run auto-transitions is this:
git clone https://release.debian.org/transitions/auto-transition
cd auto-transition/
mkdir -p output/finished output/ongoing output/planned
python2 ./auto-transitioner.py ../ubuntu-cdimage/ftp/dists/focal
../ubuntu-cdimage/ftp/dists/focal
../ubuntu-cdimage/ftp/dists/focal-proposed output/
(I am pretending that focal is testing & unstable, and focal-proposed
is experimental)

Then rm all the auto-linux-*, rm all the auto-* trackers in our
tracker config repo, and move all new trackers to ongoing.

Ideally I hope to run this automatically on ~ubuntu-archive reports
machine, and ideally point testing to focal, point sid to something
that has appropriately merged focal, and point
experimental at debian experimental, since this way we will see
upcomming transitions properly.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: ICU version in Ubuntu 20.04

[Dimitri John Ledkov]

On Fri, 10 Jan 2020 at 12:26, Shane Carr  wrote:
>
> Dear Ubuntu developers,
>
> I'm a member of the ICU TC (International Components for Unicode).  
> Developers frequently get ICU from the apt-get package "libicu-dev".
>
> We have a special ICU release coming out in March/April, ICU 66.  This 
> release is "special" because it has an emphasis on stability and 
> compatibility with ICU 65, released in October, except that it also includes 
> the latest Unicode 13 standard, which will be released around the same time.  
> Our plan with ICU 66 is so that platforms like Android can adopt this release 
> in a relatively late stage with minimal disruption.
>
> I wanted to ask whether Ubuntu 20.04 would also consider picking up ICU 66 as 
> libicu-dev.  Since it is an LTS release, users would benefit from having 
> Unicode 13 available in their server applications over the lifetime of Ubuntu 
> 20.04.
>
> Shane

boost1.71 is ongoing at the moment but progressing well.
After that, I will perform icu upgrade to 65.
April is too late, and so is March. In the future, please release in
January to be included in the Ubuntu LTS release.

Depending on how well transition to 65 goes; and how early you can
release 66; it may or may not make it into 20.04 LTS.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Proposed Migration done oddly

[Dimitri John Ledkov]

So cyrus-imapd. blocking lots of packages on arm64 and armhf.

Looks odd, and sometimes passes, sometimes fails with a cryptic
message that needs a library that has not been compiled

make[2]: *** No rule to make target
'/srv/dovecot.git/src/lib-lda/libdovecot-lda.la', needed by
'imaptest'.  Stop.
make[2]: *** Waiting for unfinished jobs

And the like.

And like all of foundations retrying it due to all of us being blocked
by it http://autopkgtest.ubuntu.com/packages/c/cyrus-imapd/focal/arm64

However, one can notice something odd about all of the failures, they
all fail with networking issue with the proxy issue when downloading
unicode data

make[3]: Entering directory '/srv/dovecot.git/src/lib'
test -f UnicodeData.txt || wget
http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
--2019-12-12 18:14:51--  http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
Resolving squid.internal (squid.internal)... 91.189.89.11
Connecting to squid.internal (squid.internal)|91.189.89.11|:3128... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
2019-12-12 18:15:51 ERROR 503: Service Unavailable.

Well luckily we have unicode data packaged in the archive, so I
uploaded a change to the autopackage to use UnicodeData.txt from
unicode-data package.

+# Use pacakged UnicodeData.txt as it often 503's when downloading on
+# arm64 armhf in Ubuntu's autopkgtest infrastructure
+cp /usr/share/unicode/UnicodeData.txt dovecot.git/src/lib
+cp /usr/share/unicode/auxiliary/WordBreakProperty.txt dovecot.git/src/lib-fts
+cp /usr/share/unicode/PropList.txt dovecot.git/src/lib-fts

Overall it still points at unreliable network in our autopkgtest
infra, but at least cyrus-impad should not be holding up any packages
in proposed migration. Seems like it's all green now.

--
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Supporting LZ4 as initramfs compressor

[Dimitri John Ledkov]

On Tue, 15 Oct 2019 at 11:13, Dimitri John Ledkov  wrote:
>
> On Wed, 5 Jun 2019 at 21:25, Dimitri John Ledkov  wrote:
> >
> > On Wed, 5 Jun 2019 at 19:59, Steve Langasek  
> > wrote:
> > >
> > > Hi Dimitri,
> > >
> > > One point here:
> > >
> > > On Wed, Jun 05, 2019 at 01:15:48PM +0100, Dimitri John Ledkov wrote:
> > > > - lz4 size weight over gzip is marginal (14%) but imho worth the
> > > > improved boot time & initrd creation time
> > >
> > > A 14% increase in initramfs size is NOT marginal.  Since there are (and 
> > > will
> > > always be) various scenarios which require a separate /boot partition, we
> > > have over several cycles been contending with how to ensure a clean 
> > > upgrade
> > > as kernel+initramfs sizes increase over time and push up against the space
> > > limits of the boot partitions that have historically been created.
> > >
> > > If you are making a change that increases the size of initramfses by 14%
> > > across the board, you must also:
> > >
> > >  - update the ubuntu-release-upgrader code to take this into account so
> > >users don't run out of space in the middle of the upgrade
> > >- possibly in a way that is smart enough to know if a user is already
> > >  configuring initramfs-tools to use a non-default compressor, to avoid
> > >  false-positives
> > >  - check that the minimum size requirements for /boot that are encoded in
> > >the installers are still adequate, or if they need updating (and if 
> > > they
> > >need updating, update this back to 18.04)
> > >  - document this issue in the release notes
> >
>
> ubuntu-release-upgrader is dynamically calculated, but I have now
> proposed to bump/update the fallback guestimate.
>
> did the calculation of new size requirements, and /boot sizes are
> still well enough (up to 10 kernels)
>
> release notes updated
>

The Cking recent lz4 measurements tests are documented in the
description of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1840934

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Supporting LZ4 as initramfs compressor

[Dimitri John Ledkov]

On Wed, 5 Jun 2019 at 21:25, Dimitri John Ledkov  wrote:
>
> On Wed, 5 Jun 2019 at 19:59, Steve Langasek  wrote:
> >
> > Hi Dimitri,
> >
> > One point here:
> >
> > On Wed, Jun 05, 2019 at 01:15:48PM +0100, Dimitri John Ledkov wrote:
> > > - lz4 size weight over gzip is marginal (14%) but imho worth the
> > > improved boot time & initrd creation time
> >
> > A 14% increase in initramfs size is NOT marginal.  Since there are (and will
> > always be) various scenarios which require a separate /boot partition, we
> > have over several cycles been contending with how to ensure a clean upgrade
> > as kernel+initramfs sizes increase over time and push up against the space
> > limits of the boot partitions that have historically been created.
> >
> > If you are making a change that increases the size of initramfses by 14%
> > across the board, you must also:
> >
> >  - update the ubuntu-release-upgrader code to take this into account so
> >users don't run out of space in the middle of the upgrade
> >- possibly in a way that is smart enough to know if a user is already
> >  configuring initramfs-tools to use a non-default compressor, to avoid
> >  false-positives
> >  - check that the minimum size requirements for /boot that are encoded in
> >the installers are still adequate, or if they need updating (and if they
> >need updating, update this back to 18.04)
> >  - document this issue in the release notes
>

ubuntu-release-upgrader is dynamically calculated, but I have now
proposed to bump/update the fallback guestimate.

did the calculation of new size requirements, and /boot sizes are
still well enough (up to 10 kernels)

release notes updated

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: preparing for the next archive opening

[Dimitri John Ledkov]

On Thu, 10 Oct 2019 at 14:36, Matthias Klose  wrote:
>
> We usually tend to open the archive with a few prepared changes in place, for
> the 20.04 cycle we are planning to open with
>
>   - python3.8 as a supported version, maybe already as the default.
>
>   - link time optimization enabled by default in GCC. This will
> add a few ftbfs (~400).  I am currently planning to have a
> blacklist package listing all the source which cannot be
> built with LTO.  The alternative of having to upload around
> 400 packages to disable LTO isn't appealing either.
>
>   - new boost version

well, if i have one in place in time. At least it will split python2 &
python3 portions.

What else is planned is:

- openssl, gnutls, nss with TLS v1.2 set as minimum protocol level

>
> Desktop and server teams didn't want to add anything to this list yet.
>
> When enabling the Debian syncs, we'll see some more transitions, as usual.
>
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: How to further handle Openssl 1.1.1 in Bionic?

[Dimitri John Ledkov]

On Wed, 9 Oct 2019 at 17:41, Christian Ehrhardt
 wrote:
>
> Hi,
> in recent weeks since [1][2] there were quite some bugs related to
> rebuilds or feature requests.
> Those kind of issues seemed to be partially expected quoting the bugs SRU 
> text:
>
> "OpenSSL 1.1.1 is ABI/API compatible with 1.1.0, however some software
> is sensitive to the negotiation handshake and may either need
> patches/improvements or clamp-down to maximum v1.2."
>
> Along the SRU some packages were initially identified needing patches
> to either fully support (if doable in an SRU scope) or clamp-down to
> TLSv1.2 and got such changes.
>
> But since then there is a set of bugs [3] coming up for either
> a) "could you also enable TLSv1.3 in package ..."
> b) "Openssl 1.1.1 broke package ..."
>

There is no generic guidance we can give on what to do with these, as
they need to be on a case by case basis.

I have submitted a few rebuilds were clearly the code is sensitive to
openssl used at built time and picks up libssl1.1 >= 1.1.1 shared
library dependency based on symbols used, and those got rejected by
the SRU team.

In general, the point of openssl 1.1.1 SRU was to make openssl
supportable (maintainable, certifiable) over the Bionic timeframe. It
was not to enable TLSv1.3 across the board.

We did call the risk of connectivity issues. And those need to be
handled on a case by case basis. SNI was explicitly called out for,
and yes we had to fix about a dozen packages to do that. Arguably
things should have been using SNI for years now, and it is a security
improvement to verify, enforce, and use SNI properly. Thus when issue
is w.r.t. SNI patching to support SNI is the correct course of action
and usually simple to do.

So far connectivity issues have been minor compared with when we
enabled TLSv1.2, then had to disable it by default (but keep
compiled), then enable it back in. And these days, we are luckly
enough to have openssl.cnf system-wide way to set MaxProtocol=TLSv1.2
to prevent TLSv1.3 based connectivity issues locally.



> And while some of those seem to be "just work" e.g. a rebuild or small
> patch to enable/disable TLSv1.3 others run into interesting issues as
> openssl has changed more than jsut adding TLSv1.3.
>
> A good example is a bug that was expected to just be a rebuild [4]. We
> have realized there can be subtle effects causing regressions. In the
> particular example it seems that a rebuild not only enabled TLSv1.3,
> but also bumped the minimum dh key size to 2k [5] which in turn breaks
> some older clients and therefore is a no-no from an SRU perspective.

That's not quite correct assessment of things. We will break people
and will trade connectivity for better security. That's why we have
disabled SSLv3, mitigated poodle attacks, etc. We will continue to
require entropy, and higher key sizes, and better key-exchange
algorithms as we go along. And sometimes that includes security
updates, which SRUs build on top of. The change-effect you describe is
due to a security update of openssl, which trumps SRUs. OpenSSL 1.1.0
& 1.1.1 have raised a set of minimum key size requirements, mostly
breaking all the test-suites with pre-generated tiny keys, but some
real users too.

As a local configuration, I believe one can lower OpenSSL security
requirements by setting CipherString = DEFAULT@SECLEVEL=0 which will
bring down requirements down to like pre 1.0.2, but that should only
done as a local site configuration, and clients haunted down and
upgraded/fixed.

For the HAPROXY, it would be nice to check if CipherString =
DEFAULT@SECLEVEL=0 "helps" to bring down dh sizes to less secure ones,
and if smaller dh sizes are pre-computed/available still. I would only
call this is a regression, if there really is no way to use smaller dh
sizes and if they are stopped being available at all.

IMHO the haproxy SRU should not have been accepted, should now be
tagged proposed-regression and removed from bionic-proposed. Which is
a normal SRU process, and retry later. Or like discover some CVE and
ask security team to deal with the rebuild =)

> The bug [4] currently is assigned to ubuntu-security team for guidance
> on this - do we want/need this and accept the regression it causes or
> do we want/need to "clamp-down" the dh key size as well?
>

NAK

> But this is a generic question - not only in the context of haproxy for [4].
> If formerly only "clamping down for TLSv1.2" was considered, do we
> need to revisit all packages for DH key size as well? ...
>

NAK

> Even if we don't do anything today, a security update tomorrow might
> force us to rebuild and trigger this kind of issues for 'potentially'
> all dependencies of openssl.
> We already have seen quite some of them being actual regressions in
> our LTS which is concerning for the potential estimated number of
> unreported cases.
>

That was a known risk, that's why we are choosing not to blindly
rebuild things, especially at the far tail in universe.

This 

Re: Second Eoan Ermine test rebuilds

[Dimitri John Ledkov]

On Mon, 9 Sep 2019 at 17:20, Matthias Klose  wrote:
>
> The second test rebuild of Eoan Ermine was started on September 06 2019 for
> all architectures, all components. The rebuild of the main component is
> finished, the other components (restricted, universe, multiverse) are still
> building.
>
> Unfortunately we see 1300+ build failures, and still counting ...  On the 
> other
> hand the test rebuild includes the recent GCC, glibc packages, and the
> glib2.o/gtk packages synced from experimental.
>
> Results (please also look at the superseded builds) can be found at
>
> https://people.canonical.com/~doko/ftbfs-report/test-rebuild-20190906-eoan.html
>
> The report uses some additional color coding, marking packages different which
> always failed to build, or where the build failure is no regression compared 
> to
> bionic.  The test rebuild already uses the linux-libc-dev 5.3 package found in
> the ubuntu-toolchain-r/volatile PPA.
>
> Additional build failures for packages in eoan-proposed (not yet in eoan)
> can be found at http://qa.ubuntuwire.com/ftbfs/
>
> Please help fixing the build failures.
>

Fixed btrfs-progs, submitted pull request upstream.
Fixed d-i/s390x build.


> There is also a test rebuild with link time optimization turned on by default
> (passing -flto=auto in CFLAGS/CXXFLAGS).  Other distributions turned these
> optimzations on by default, or are considering doing that.
>
> https://people.canonical.com/~doko/ftbfs-report/test-rebuild-20190906-lto-eoan.html
>
> Packages to use -flto by default (dpkg) can be found in the
> ubuntu-toolchain-r/dpkg-lto PPA. WARNNING: Make sure to install these dpkg
> packages in a throw-away environment.

Are you going to copy in fixed up packages to update the report? Or
can coredevs do that somehow?

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Some ideas about APT functionality

[Dimitri John Ledkov]

Hi,

On Sun, 11 Aug 2019 at 21:37, Mike  wrote:
>
> Today, to properly install rpm-packages on my laptops, I'm running

Which rpm packages are they? For which architectures? Is .deb
available? Or snap? If not, have you tried reaching out to the vendor
to provide snap/deb? Do you want us to reach out to them?

-- 
Regards,

Dimitri.

-- 
Ubuntu-devel-discuss mailing list
Ubuntu-devel-discuss@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss

Re: Supporting LZ4 as initramfs compressor

[Dimitri John Ledkov]

On Wed, 5 Jun 2019 at 19:59, Steve Langasek  wrote:
>
> Hi Dimitri,
>
> One point here:
>
> On Wed, Jun 05, 2019 at 01:15:48PM +0100, Dimitri John Ledkov wrote:
> > - lz4 size weight over gzip is marginal (14%) but imho worth the
> > improved boot time & initrd creation time
>
> A 14% increase in initramfs size is NOT marginal.  Since there are (and will
> always be) various scenarios which require a separate /boot partition, we
> have over several cycles been contending with how to ensure a clean upgrade
> as kernel+initramfs sizes increase over time and push up against the space
> limits of the boot partitions that have historically been created.
>
> If you are making a change that increases the size of initramfses by 14%
> across the board, you must also:
>
>  - update the ubuntu-release-upgrader code to take this into account so
>users don't run out of space in the middle of the upgrade
>- possibly in a way that is smart enough to know if a user is already
>  configuring initramfs-tools to use a non-default compressor, to avoid
>  false-positives
>  - check that the minimum size requirements for /boot that are encoded in
>the installers are still adequate, or if they need updating (and if they
>need updating, update this back to 18.04)
>  - document this issue in the release notes

agree.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Supporting LZ4 as initramfs compressor

[Dimitri John Ledkov]

On Fri, 31 May 2019 at 09:13, Seth Arnold  wrote:
>
> On Thu, May 30, 2019 at 11:46:57AM +0100, Dimitri John Ledkov wrote:
> > As if lz4 kernel & xz initrd would yield the fastest boot time? That
>
> I'm lacking some context here, but I think building the initrds is already
> too slow and I'm afraid xz on initrd rebuilds would be significantly
> worse than lz4 or zstd or even low-compression levels of zlib.
>
> Boot times are important but every now and then people do run apt upgrade
> by hand and waiting forever to rebuild an initrd that may or may not be
> used is pretty tedious. xz is great if the really slow compress times will
> be made up for by better transfers or disk savings or similar.
>

I've tried to dig up past benchmarks in private discussions, public
discussions and kernel mailing list.

Zstd patches have not made it into the upstream kernel yet.

As used by mkinitramfs:
- lz4 is faster to compress than gzip
- lz4 is blazingly fast to decompress
- lzma is dog slow to compress and decompress, but is tiny
- lz4 size weight over gzip is marginal (14%) but imho worth the
improved boot time & initrd creation time
- xz is potentially even slower and even smaller than lzma

In places where size is an absolute premium (tiny embedded iot
devices) and performance is irrelevant, xz or lzma should be used.

In all other places, our performance profile is in favor of lz4.

Imho that includes the kernel image itself, thus we should consider switching:
- initramfs tools to default to lz4
- livecd-rootfs to default to lz4
- kernels to compress kernel image with lz4
- grub to include lz4 support

I shall proceed with changing the defaults on the above to improve our
responsiveness experience on installer, cloud, core and classic
devices.
If our firstboot & subsequent boot speed degrades or disk space
becomes a concern, we can look into tweaking these changes further.

-- 
Regards,

Dimitri.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel