Re: incoming change to task handling in livecd-rootfs in mantic
On Tue, 23 May 2023 at 03:34, Steve Langasek wrote: > On Mon, May 22, 2023 at 01:04:51PM +1200, Michael Hudson-Doyle wrote: > > Thinking about this a bit more... what is germinate actually for in this > > context? :-) > > > The way packages from a seed end up in an image goes like this at the > > moment: > > > 1. it is listed in the seed > > 2. germinate runs and puts the package and all its dependencies into a > text > > file > > 3. live-build reads this text file and uses apt to install the packages > > (4. we run mimimize-manual at some point so removing a seeded package and > > running autoremove does something useful) > > > The thing about this is, *apt* obviously knows how to find the > dependencies > > of a package. Why don't we just shove the packages listed into the seed > > straight into the file live-build reads? > > > (I suppose one answer might be to do with alternatives handling? cf > > https://imgflip.com/i/7mmgcz -- does germinate do anything clever to > > satisfy alternatives with a minimal number of extra packages or anything > > like that?) > > It does not. Indeed, several of the recent seed changes made while landing > this were to fix the fact that germinate from livecd-rootfs (though not > when > run on the archive) was seeding two conflicting implementations one of > which > satisfies all the dependencies. > (Actually this was a virtual package rather > than an alternative, but AFAIK it doesn't do anything clever with > alternatives either!) > Well potato potahto. > I would expect feeding the list directly to apt rather than using germinate > would result in some further changes to the exact packages included, which > would then need to be examined to confirm they're what we want, since there > are often multiple ways to resolve a seed. > > There's also the fact that you'd be reimplementing germinate's own parsing > of the seeds and resolution of seed dependencies, so I'm not sure it's > worth > it? > Yes it's probably not worth it. But maybe we can lean this way for when we implement things in ubuntu-image (although I think that has support for running germinate already). (One thing that would be nice, in addition to dropping the time-consuming > use of germinate at runtime, The slow part IME is downloading the package lists, the computational part germinate only takes a few seconds. > would be that today germinate silently ignores > listed packages that are unavailable, whereas apt would enforce correctness > of the list...) > That does sound like it would be an improvement. > > ((Clearly we need something like germinate to generate the component > > mismatches reports. But maybe not at image build time?)) > > That's already run on ubuntu-archive-toolbox independently of the image > builds, so would continue to do so. > Yes, I think my point here was that we can't just delete germinate completely. Cheers, mwh > -- > Steve Langasek Give me a lever long enough and a Free OS > Debian Developer to set it on, and I can move the world. > Ubuntu Developer https://www.debian.org/ > slanga...@ubuntu.com vor...@debian.org > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel > -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: incoming change to task handling in livecd-rootfs in mantic
On Mon, May 22, 2023 at 01:04:51PM +1200, Michael Hudson-Doyle wrote: > Thinking about this a bit more... what is germinate actually for in this > context? :-) > The way packages from a seed end up in an image goes like this at the > moment: > 1. it is listed in the seed > 2. germinate runs and puts the package and all its dependencies into a text > file > 3. live-build reads this text file and uses apt to install the packages > (4. we run mimimize-manual at some point so removing a seeded package and > running autoremove does something useful) > The thing about this is, *apt* obviously knows how to find the dependencies > of a package. Why don't we just shove the packages listed into the seed > straight into the file live-build reads? > (I suppose one answer might be to do with alternatives handling? cf > https://imgflip.com/i/7mmgcz -- does germinate do anything clever to > satisfy alternatives with a minimal number of extra packages or anything > like that?) It does not. Indeed, several of the recent seed changes made while landing this were to fix the fact that germinate from livecd-rootfs (though not when run on the archive) was seeding two conflicting implementations one of which satisfies all the dependencies. (Actually this was a virtual package rather than an alternative, but AFAIK it doesn't do anything clever with alternatives either!) I would expect feeding the list directly to apt rather than using germinate would result in some further changes to the exact packages included, which would then need to be examined to confirm they're what we want, since there are often multiple ways to resolve a seed. There's also the fact that you'd be reimplementing germinate's own parsing of the seeds and resolution of seed dependencies, so I'm not sure it's worth it? (One thing that would be nice, in addition to dropping the time-consuming use of germinate at runtime, would be that today germinate silently ignores listed packages that are unavailable, whereas apt would enforce correctness of the list...) > ((Clearly we need something like germinate to generate the component > mismatches reports. But maybe not at image build time?)) That's already run on ubuntu-archive-toolbox independently of the image builds, so would continue to do so. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: incoming change to task handling in livecd-rootfs in mantic
On Fri, 19 May 2023 at 22:38, Iain Lane wrote: > On Fri, May 19, 2023 at 09:05:28PM +1200, Michael Hudson-Doyle wrote: > > After a few rounds of fixups, this change passed all tests and has now > > migrated to release, so the next round of mantic image builds will be > built > > with it. Let me know if you see anything strange in them! > > Really nice, great work. This has been long overdue, glad it finally got > sorted out. > Thanks! I think it makes things a bit saner. > Although the backslash line was one of my favourite lines of code in the > archive and I'll be sad to see it go. :-) > Heh yes. At least version control will preserve it for future generations... I was reminded of > > https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1921862 > > When reading this thread. tl;dr is that it's not (was not at the time) > possible to seed new packages post-release because germinate isn't being > run for -updates in the publisher so the newly-seeded packages don't get > Task headers. Would that be any more possible to achieve now? (For > seeding only: *un*seeding is more gnarly, it involves knowing how to > have -updates override release.) > Modulo what Steve said I think this mostly fixed now, to the extent that additions and removals to seeds will just be reflected in the next image build. The gap we have is that if a seeded package has different dependencies in different pockets, that might not get picked up. But this seems a touch edge casey. Thinking about this a bit more... what is germinate actually for in this context? :-) The way packages from a seed end up in an image goes like this at the moment: 1. it is listed in the seed 2. germinate runs and puts the package and all its dependencies into a text file 3. live-build reads this text file and uses apt to install the packages (4. we run mimimize-manual at some point so removing a seeded package and running autoremove does something useful) The thing about this is, *apt* obviously knows how to find the dependencies of a package. Why don't we just shove the packages listed into the seed straight into the file live-build reads? (I suppose one answer might be to do with alternatives handling? cf https://imgflip.com/i/7mmgcz -- does germinate do anything clever to satisfy alternatives with a minimal number of extra packages or anything like that?) ((Clearly we need something like germinate to generate the component mismatches reports. But maybe not at image build time?)) Cheers, mwh Cheers! > > -- > Iain Lane [ i...@orangesquash.org.uk ] > Debian Developer [ la...@debian.org ] > Ubuntu Developer [ la...@ubuntu.com ] > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel > -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: incoming change to task handling in livecd-rootfs in mantic
On Fri, May 19, 2023 at 11:38:22AM +0100, Iain Lane wrote:
> On Fri, May 19, 2023 at 09:05:28PM +1200, Michael Hudson-Doyle wrote:
> > After a few rounds of fixups, this change passed all tests and has now
> > migrated to release, so the next round of mantic image builds will be built
> > with it. Let me know if you see anything strange in them!
> Really nice, great work. This has been long overdue, glad it finally got
> sorted out.
> Although the backslash line was one of my favourite lines of code in the
> archive and I'll be sad to see it go. :-)
> I was reminded of
> https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1921862
> When reading this thread. tl;dr is that it's not (was not at the time)
> possible to seed new packages post-release because germinate isn't being
> run for -updates in the publisher so the newly-seeded packages don't get
> Task headers. Would that be any more possible to achieve now? (For
> seeding only: *un*seeding is more gnarly, it involves knowing how to
> have -updates override release.)
The effect of this change is that we're not dependent any longer on the
Task: headers in the archive, so I *think* both addition and removal of
packages from the seeds will now be honored.
However, we're currently invoking germinate with "-d $SUITE" and I think we
need to change this to "-d ${SUITE},${SUITE}-updates" (with an added
${SUITE}-proposed when building with PROPOSED=1) to get germinate to look at
all the right pockets.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: PGP signature
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: incoming change to task handling in livecd-rootfs in mantic
On Fri, May 19, 2023 at 09:05:28PM +1200, Michael Hudson-Doyle wrote: > After a few rounds of fixups, this change passed all tests and has now > migrated to release, so the next round of mantic image builds will be built > with it. Let me know if you see anything strange in them! Really nice, great work. This has been long overdue, glad it finally got sorted out. Although the backslash line was one of my favourite lines of code in the archive and I'll be sad to see it go. :-) I was reminded of https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1921862 When reading this thread. tl;dr is that it's not (was not at the time) possible to seed new packages post-release because germinate isn't being run for -updates in the publisher so the newly-seeded packages don't get Task headers. Would that be any more possible to achieve now? (For seeding only: *un*seeding is more gnarly, it involves knowing how to have -updates override release.) Cheers! -- Iain Lane [ i...@orangesquash.org.uk ] Debian Developer [ la...@debian.org ] Ubuntu Developer [ la...@ubuntu.com ] signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: incoming change to task handling in livecd-rootfs in mantic
On Mon, 15 May 2023, 09:47 Michael Hudson-Doyle, < michael.hud...@canonical.com> wrote: > Hi all, > > I've just uploaded a change[1] to livecd-rootfs that changes how the > "add_task" function works. > > It switches away from using the "Task" headers in the archive's package > lists to find the packages (and snaps) that make up a task to reading the > information directly from the output of germinate. > > I originally wrote this because I wanted to make an image from a rebuild > archive (which don't have Task headers) but it also makes the handling of > tasks that much more direct and hopefully a touch less confusing -- for > example, changes to seeds will be reflected in image builds without having > to wait for an archive publication cycle (it also removes two sequence of > six consecutive backslashes from the source, which has to be a win). > > Now this _should_ not result in any changes to the contents of images -- > I've tested a few builds before landing and it all looks fine -- but to be > sure we will hold this new version of livecd-rootfs in proposed until we > have done a bunch more test images. > Hi again, After a few rounds of fixups, this change passed all tests and has now migrated to release, so the next round of mantic image builds will be built with it. Let me know if you see anything strange in them! Cheers, mwh Cheers, > mwh > > [1] > https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/425047 > -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
incoming change to task handling in livecd-rootfs in mantic
Hi all, I've just uploaded a change[1] to livecd-rootfs that changes how the "add_task" function works. It switches away from using the "Task" headers in the archive's package lists to find the packages (and snaps) that make up a task to reading the information directly from the output of germinate. I originally wrote this because I wanted to make an image from a rebuild archive (which don't have Task headers) but it also makes the handling of tasks that much more direct and hopefully a touch less confusing -- for example, changes to seeds will be reflected in image builds without having to wait for an archive publication cycle (it also removes two sequence of six consecutive backslashes from the source, which has to be a win). Now this _should_ not result in any changes to the contents of images -- I've tested a few builds before landing and it all looks fine -- but to be sure we will hold this new version of livecd-rootfs in proposed until we have done a bunch more test images. Cheers, mwh [1] https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/425047 -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
