Just to close the loop on this (the d-i build failure due to the regression in kmod), this has been reported on LP#1889297, and the fix is already in bionic-proposed.
The debootstrap changes (original work from SF case) too, and verified (thanks Matthew!) So, all good! On Mon, Jul 27, 2020 at 1:01 PM Mauricio Oliveira <[email protected]> wrote: > > Hi Matthew, > > Nice tricks you found out. > > So, I checked w/ Łukasz today, and he's more aware of the d-i problem; > and welcomed help to fix it. > > He would have to rebuild d-i anyway to bump the kernels, so fixing it > is on their critical path too. > (Thus we don't have to worry about having our fix for d-i/kmod/systemd > being neglected :-) > > As for timing for release, now that I have more energy to think about it, > we don't need the d-i rebuilt to actually pick debootstrap in the ISO > (which is the timing sensitive part), > just the new/fixed package available in the archive. > Because debootstrap does not go into the initrd (which is what d-i > builds), but in the ISO "archive" only (I guess it's debian-cd which > does it.) > [see below] > > And similarly, it seems that the network installer pulls debootstrap > as well -- it's not in initrd either. > So, fixing debootstrap in the archive is the only thing needed to get > it on both ISO and netinst. :-) > > Today I've done some more testing w/ debootstrap, for the SRU team to > be a bit happier, > and uploaded to B/F. Łucasz has ACKed. :) > > I've got to go for the day, but should be able to check MM in a few hours. > (And later today or tomorrow get into the fun of > > [ISO checks for debootstrap] > > $ sudo mount -o loop,ro ubuntu-18.04.4-server-amd64.iso mnt > > $ find mnt/ -name '*debootstrap*' > mnt/pool/main/d/debootstrap > mnt/pool/main/d/debootstrap/debootstrap-udeb_1.0.95ubuntu0.5_all.udeb > > $ mkdir initrd > $ cd initrd > $ gzip -dc ../mnt/install/initrd.gz | cpio -H newc -mid > $ find . -name '*debootstrap*' > $ > > [netinst checks for debootstrap] > > $ wget > http://archive.ubuntu.com/ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz > $ gzip -dc initrd.gz | cpio -H newc -mid > $ find . -name '*debootstrap*' > $ > > On Sun, Jul 26, 2020 at 11:02 PM Matthew Ruffell > <[email protected]> wrote: > > > > Hi Mauricio, > > > > Thanks for your awesome analysis! > > > > > At this point I think that *if* the case/matter is only about the > > > network installer, > > > not the ISO (which is indeed critical with 18.04.5), then it can be > > > fixed via SRU, > > > without any concerns. > > > > > Would you please evaluate this w/ customer/Lukasƶ on Monday? > > > > The customer was not interested in the mini.iso image, they use the network > > installer, with vmlinuz and initrd.gz. So, that's good to hear that it can > > be > > fixed via SRU without any concerns. > > > > I suppose there is no need to rush now. > > > > > You may notice that even before the --add-udeb flag removal, > > > libkmod2-udeb *did not* > > > ship that shared library! > > > > > > I wondered how the installer worked. It DOES have that library in > > > /lib/libkmod.so.2. > > > And I have no idea where it pulled it from. > > > > > I checked MANIFEST files, d-i build log, even download all udebs in > > > bionic/bionic-updates, > > > extracted, but nothing shipped it. > > > > > This might be some d-i build magic, which I didn't look/find yet. > > > > You and me both! I hit the problem when I was making the test mini.iso and > > test vmlinuz / initrd.gz files. What I did to workaround it was do a grep > > over > > the local repository cache d-i keeps when building images, and then do a > > cheeky > > sed -i s/libkmod2/libkmod2-udeb/g, commands below: > > > > "sed -i "s/Depends: libblkid1-udeb (>= 2.31), libc6-udeb (>= 2.27), > > libkmod2, util-linux-udeb/Depends: libblkid1-udeb (>= 2.31), > > libc6-udeb (>= 2.27), libkmod2-udeb, util-linux-udeb/g" > > ./apt.udeb/state/lists/nz.archive.ubuntu.com_ubuntu_dists_bionic-updates_main_debian-installer_binary-amd64_Packages" > > > > "sed -i "s/Depends: libblkid1-udeb (>= 2.31), libc6-udeb (>= 2.27), > > libkmod2, util-linux-udeb/Depends: libblkid1-udeb (>= 2.31), > > libc6-udeb (>= 2.27), libkmod2-udeb, util-linux-udeb/g" > > ./apt.udeb/state/lists/security.ubuntu.com_ubuntu_dists_bionic-updates_main_debian-installer_binary-amd64_Packages" > > > > I then re-ran "fakeroot make build_monolithic". Apt seemed to be perfectly > > happy > > with the changes, even though libkmod2-udeb was missing from the archive, > > and > > produced the test install media correctly. > > > > I suppose that hack worked since libkmod2-udeb didn't actually ship > > any libraries, > > and /lib/libkmod.so.2 turns up through another magical mechanism. > > > > As for that magical mechanism, I found this in the README file in > > debian-installer/build/README: > > > > * Apt is used to download the required udebs. This does *not* include > > libraries; libraries used by udebs must be installed on the build system, > > and so are build-depended on. > > > > Sounds to me that if we include libkmod2, it copies the library from the > > build > > host into the install image. That's cheating =p > > > > As for the fix, well, I think the path of least resistance is to simply > > revert > > the change to libkmod2, and get it to start building libkmod2-udeb > > again (in bionic). > > > > Sure, it will incorrectly advertise itself as containing something useful > > when > > it really doesn't, but everything was fine for the two years it existed in > > the > > archive. I wouldn't revert it on anything beyond Bionic though. > > > > I'll send the above to Lukasz in case he has seen the dependency problem and > > had been looking into it. > > > > Thanks, > > Matthew > > > > On Sat, Jul 25, 2020 at 12:49 PM Mauricio Oliveira > > <[email protected]> wrote: > > > > > > Hi Matthew, > > > > > > Long story short, the debian-installer rebuild on PPA fails on Bionic. > > > > > > The fix for this should apparently involve changes to kmod and systemd, > > > at least systemd, and I think this may get push back from release team > > > at this time before the point release (however small the change is.) > > > > > > If the fix is needed only in the network-installer, that can be an SRU > > > later. > > > > > > Let me explain, and hopefully you can start addressing it with Lukasƶ > > > on your Monday, before I join. > > > > > > The d-i build on PPA fail because 'udev-udeb' cannot pull 'libkmod2'. > > > > > > Building dependency tree... > > > Installing libslang2-udeb as Depends of cdebconf-newt-udeb > > > udev-udeb:amd64 Depends on libkmod2:amd64 < none @un H > can't > > > be satisfied! > > > > > > ... > > > > > > The following packages have unmet dependencies: > > > udev-udeb : Depends: libkmod2 but it is not installable > > > E: Unable to correct problems, you have held broken packages. > > > Makefile:620: recipe for target > > > 'stamps/get_udebs-cdrom_isolinux-stamp' failed > > > make[7]: *** [stamps/get_udebs-cdrom_isolinux-stamp] Error 100 > > > > > > This is correct, because the d-i build does configure sources only for > > > udeb > > > packages, and libkmod2 is a regular deb. > > > > > > Previously it worked because udev-udeb pulled libkmod2-udeb (note suffix.) > > > > > > This is a regression in the dependencies of udev-udeb in 237-3ubuntu10.40: > > > > > > https://launchpad.net/ubuntu/bionic/amd64/udev-udeb/237-3ubuntu10.39 > > > Depends on: libkmod2-udeb > > > > > > https://launchpad.net/ubuntu/bionic/amd64/udev-udeb/237-3ubuntu10.40 > > > Depends on: libkmod2 > > > > > > This happened because the build dependency kmod (which you're familiar > > > with :) > > > changed versions in between, 24-1ubuntu3.2 to 3.3: > > > > > > On u10.39: Get:45 http://ftpmaster.internal/ubuntu > > > bionic-updates/main amd64 libkmod2 amd64 24-1ubuntu3.2 [40.0 kB] > > > On u10.40: Get:45 http://ftpmaster.internal/ubuntu > > > bionic-updates/main amd64 libkmod2 amd64 24-1ubuntu3.3 [40.1 kB] > > > > > > And this change caused this: > > > > > > kmod (24-1ubuntu3.3) bionic; urgency=medium > > > ... > > > * Removed --add-udeb from dh_makeshlibs, since libkmod2-udeb does > > > not > > > actually contain a library. (Closes: #939779) > > > ... > > > > > > This still builds/ships the libkmod2-udeb package, but... > > > > > > The lack of dh_makeshlibs with --add-udeb causes the libkmod2 > > > (non-udeb)'s "shlibs" > > > file to no longer indicate that udeb packages which depend on that > > > library package > > > should depend on the -udeb package instead (exactly what happens here.) > > > > > > From dh_makeshlibs(1): > > > > > > --add-udeb=udeb > > > Create an additional line for udebs in the shlibs file and > > > use udeb as the package name for udebs to depend on instead of the > > > regular library package. > > > > > > This can be confirmed with: > > > > > > $ pull-lp-debs -a amd64 libkmod2 bionic 24-1ubuntu3.2 > > > $ pull-lp-debs -a amd64 libkmod2 bionic 24-1ubuntu3.3 > > > > > > $ dpkg-deb --control libkmod2_24-1ubuntu3.2_amd64.deb > > > libkmod2-32-control > > > $ dpkg-deb --control libkmod2_24-1ubuntu3.3_amd64.deb > > > libkmod2-33-control > > > > > > $ cat libkmod2-32-control/shlibs > > > libkmod 2 libkmod2 > > > udeb: libkmod 2 libkmod2-udeb > > > > > > $ cat libkmod2-33-control/shlibs > > > libkmod 2 libkmod2 > > > > > > And according to systemd/debian/control file's entry for udev-udeb: > > > > > > Package: udev-udeb > > > ... > > > Depends: ${shlibs:Depends}, > > > ${misc:Depends}, > > > util-linux-udeb > > > > > > That dependency comes from ${shlibs:Depends}, handled by dh_shlibdeps(1): > > > > > > https://www.debian.org/doc/manuals/maint-guide/dreq.en.html > > > """ > > > dh_shlibdeps(1) calculates shared library dependencies for binary > > > packages. > > > It generates a list of ELF executables and shared libraries it has > > > found for each binary package. > > > This list is used for substituting ${shlibs:Depends}. > > > """ > > > > > > So, > > > > > > Since d-i needs both udev-udeb and libkmod2-udeb (for udev, > > > load/remove modules, etc) > > > fixing the udev-udeb dependency would require *at least* a > > > change/rebuild to systemd > > > to update the Depends: field of udev-udeb. > > > > > > Say, even if we came up with a hack/solution that doesn't need a > > > change/fix to libkmod2. > > > > > > This is where I think the SRU/release team might prefer to hold things > > > for release. > > > (systemd.) > > > > > > At this point I think that *if* the case/matter is only about the > > > network installer, > > > not the ISO (which is indeed critical with 18.04.5), then it can be > > > fixed via SRU, > > > without any concerns. > > > > > > Would you please evaluate this w/ customer/Lukasƶ on Monday? > > > > > > > > > A bit on libkmod2 change: > > > > > > The change/regression of --add-udeb to libkmod2 is not incorrect on > > > its own, since > > > libkmod2-udeb doesn't ship a shared library. > > > > > > I discussed this a bit with Rafael Tinoco (which introduced the > > > change), and perhaps > > > the change can be reverted (I'd have to check with a rebuild of kmod > > > w/ it reverted.) > > > > > > (then rebuild kmod, then systemd, then d-i -- after debootstrap w/ > > > your fix, of course. > > > See it's a chain that looks big for this time before the release. :) > > > > > > > > > Some tech details: > > > > > > udev-udeb depends on libkmod2 shared library because of udevadm, for > > > the kmod built-in. > > > > > > You may notice that even before the --add-udeb flag removal, > > > libkmod2-udeb *did not* > > > ship that shared library! > > > > > > I wondered how the installer worked. It DOES have that library in > > > /lib/libkmod.so.2. > > > And I have no idea where it pulled it from. > > > > > > I checked MANIFEST files, d-i build log, even download all udebs in > > > bionic/bionic-updates, > > > extracted, but nothing shipped it. > > > > > > This might be some d-i build magic, which I didn't look/find yet. > > > > > > > > > Finally, I'm a bit tired to think now, so I'll give you the ball for now > > > to play > > > a bit on considerations of 1) should this really be fixed before > > > release for ISO, > > > or just network installer is OK? and more fun in 2) how to properly > > > fix this :) > > > > > > Thanks and have a great weekend! > > > > > > P.S.: I'm attaching my hacky notes about this today, in case it might > > > help. > > > > > > On Fri, Jul 24, 2020 at 7:27 PM Mauricio Oliveira > > > <[email protected]> wrote: > > > > > > > > Matthew, > > > > > > > > By the way, per the SF case, the customer uses the network install > > > > method. > > > > This can be fixed post-release, as SRUs to debian-installer will > > > > produce new netboot images. > > > > > > > > Hopefully they (and other people) don't plan to try to use HTTPS on > > > > the _ISO_ installer? > > > > The ISO image is what is more strictly tied to the release date. > > > > I'm not sure how much that scenario makes sense. > > > > (there may be the case of trying to use HTTPS for the ISO installer to > > > > download updates though?) > > > > > > > > I'm saying this because this is now (of course!) heading into systemd > > > > :-) > > > > specifically the dependencies of the udev-udeb package. And that may > > > > be just an adventure > > > > to try to fix / SRU before release date. :) > > > > > > > > I'll give more details later. > > > > > > > > cheers, > > > > > > > > > > > > On Fri, Jul 24, 2020 at 6:30 PM Mauricio Oliveira > > > > <[email protected]> wrote: > > > > > > > > > > Definitely looking at it right now. :) > > > > > > > > > > The debootstrap portion is OK. However, > > > > > > > > > > there's a PPA build error with the required rebuild of d-i in Bionic > > > > > (not on Focal -- there's always fun on the releases we actually need!) > > > > > due to udev-udeb changing dependencies from libkmod2-udeb to libkmod2 > > > > > (non-udeb), which I just tracked down to kmod 24-1ubuntu3.3. > > > > > > > > > > I'll provide a write up on the LP bug (and mention internally in SF > > > > > case) and hopefully it should be relatively simple to fix. > > > > > > > > > > As mentioned, debootstrap looks good; so I'll move that one soon. > > > > > > > > > > cheers, > > > > > > > > > > On Fri, Jul 24, 2020 at 6:25 PM Dan Streetman > > > > > <[email protected]> wrote: > > > > > > > > > > > > Hi Matthew, > > > > > > > > > > > > As @mfo has done work in this (installer image) area before (LP: > > > > > > #1807023) I asked him to take a look; I'm cc'ing our sts sponsors > > > > > > email list (which includes Mauricio) also. > > > > > > > > > > > > I think he's handling the sponsoring, so hopefully it'll make it for > > > > > > the point release! > > > > > > > > > > > > Thanks! > > > > > > > > > > > > On Fri, Jul 24, 2020 at 1:14 AM Matthew Ruffell > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > Hi Dan and Eric, > > > > > > > > > > > > > > [URGENT] Please review and sponsor LP1772556. This needs to make > > > > > > > 18.04.5 LTS. > > > > > > > > > > > > > > https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/1772556 > > > > > > > > > > > > > > I have written to Lukasz Zemczak and made him aware of this bug, > > > > > > > and I have also > > > > > > > asked him to review and sponsor the debdiffs, since he was the > > > > > > > last to modify > > > > > > > debootstrap in Bionic, and is also in charge of producing 18.04.5 > > > > > > > LTS images. > > > > > > > > > > > > > > I haven't heard back from Lukasz though. I emailed him on my > > > > > > > Wednesday, and > > > > > > > on my Thursday. > > > > > > > > > > > > > > Could you please maybe ping Lukasz during your day to make sure > > > > > > > he got my email? > > > > > > > > > > > > > > The debdiffs in the LP bug are kinda weird. debootstrap uses "3.0 > > > > > > > (native)" > > > > > > > instead of "3.0 (quilt)", and the debdiff seems to take diffs of > > > > > > > all the ubuntu > > > > > > > releases, even though they are all symlinks to a single file that > > > > > > > got changed, > > > > > > > which is "gutsy". > > > > > > > > > > > > > > Anyway, please nudge Lukasz, or sponsor my debdiffs. Konica > > > > > > > Minolta filed a > > > > > > > L2 about this in SF289200, and it needs to make 18.04.5 LTS or it > > > > > > > will never be > > > > > > > fixed. > > > > > > > > > > > > > > Thanks! > > > > > > > Matthew > > > > > > > > > > > > -- > > > > > > Mailing list: https://launchpad.net/~sts-sponsors > > > > > > Post to : [email protected] > > > > > > Unsubscribe : https://launchpad.net/~sts-sponsors > > > > > > More help : https://help.launchpad.net/ListHelp > > > > > > > > > > > > > > > > > > > > -- > > > > > Mauricio Faria de Oliveira > > > > > > > > > > > > > > > > -- > > > > Mauricio Faria de Oliveira > > > > > > > > > > > > -- > > > Mauricio Faria de Oliveira > > > > -- > Mauricio Faria de Oliveira -- Mauricio Faria de Oliveira -- Mailing list: https://launchpad.net/~sts-sponsors Post to : [email protected] Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
