Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Christian Kastner]

On 2022-04-02 07:58, Johannes Schauer Marin Rodrigues wrote:
> Quoting Christian Kastner (2022-03-23 21:36:59)
>> I initially wondered whether foreign architecture guestfish/supermin
>> binaries couldn't be run through qemu-user-static, but that solution
>> would be just far too hacky I think.
> 
> I think how "hacky" it is depends on how you look at it. The advantage is, 
> that
> then you only have to maintain the "native" codepath which you run under
> emulation if you want a foreign arch binary. I just tried it out and it seems
> to work as expected.

Well, that's cool to hear!

> Big downside: running qemu (from guestfish) inside qemu (from
> qemu-user-static) is *ralllyy sloow*. XD

How slow? On my end, IIRC sbuild-qemu-create takes 2mins to build a
native image, and 12-13mins to build a foreign one.

However, that's wall-time, for a process that can run unattended. Having
a one-time, slow, non-interactive process might not be that bad,
compared to needing root.

> But I think it's a good idea to have for the sake of symmetry. We'd then have:
> 
>  - guestfish
>  - fastest method when creating a native image
>  - slowest method when creating a foreign image
>  - initramfs
>  - medium speed for both native and foreign images
> 
> Then the new tool could have a --method switch with possible values:
> 
>  - "auto" choose guestfish if installed for native images and initramfs
>otherwise
>  - "guestfish" forces guestfish for native and foreign
>  - "initramfs" forces initramfs for native and foreign

Sounds good.

>> Since these are QEMU options, I think autopkgtest-virt-qemu's existing
>> --qemu-options could be used to supply kernel/initrd/rootfs?
> 
> Yes, that might work. Though from reading the autopkgtest-virt-qemu man page I
> think this will not support paths with spaces. On the other hand, the man page
> also says that additional arguments can be supplied via a file that contains
> one argument per line.

Indeed, I just checked and autopkgtest-virt-qemu just calls .split() on
the string. The file workaround should be fine, though.

> I'm not so sure whether it's a fantastic idea. First I want to evaluate other
> existing tools like debos and see if they can do the job or not. I want to
> resist the NIH syndrome.

Full ACK. Hence the focus on extending vmdb2 if possible, or using
guestfish or debos, but I myself don't favor any particular tool. My
first choice would simply be the tool that accomplishes it.

> Another question that I'm still pondering is how much configuration this new
> tool should allow. I'm tempted to make it just fit for the use-case of
> autopkgtest backend or porterbox or sbuild-qemu backend just to keep it 
> simple.

Indeed, I have asked myself the same question a number of times.

It's tempting to make this as flexible as possible, but in the end, it's
main use cases are build and test environments. From experience, this
means that one frequently needs to configure hardware (e.g. number of
CPUs), or modify the build environment (configure external repos,
pre-install packages, copy files e.g. SSH keys).

So it can be helpful to facilitate a few common uses, but it's probably
overkill to allow for all, and it's probably also not needed: after all,
one can always boot the image, and modify interactively.

> And lastly, the name is also still an open question.

The hardest part :D

> Alas, I'll probably not get to working on this much in the next two weeks.

Best of luck! I wish I could do more to help, but still swamped at the
moment, with little time for Debian.

Best,
Christian

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Christian Kastner (2022-03-23 21:36:59)
> On 2022-03-23 12:43, Johannes Schauer Marin Rodrigues wrote:
> > Quoting Francesco Poli (2022-03-23 00:09:21)
> > Full: the problem with the current version of 
> > mmdebstrap-autopkgtest-build-qemu
> > is, that it can only build qemu images for the native architecture. This is
> > because it relies on guestfish. Guestfish will never be able to operate on
> > foreign architecture qemu guests. This is because:
> > 
> >  - guestfish sets architecture specific options at compile time. This means
> >that every guestfish binary can only be used for qemu guests of the same
> >architecture as that binary and this cannot be changed at runtime
> > 
> >  - guestfish relies on another program called supermin. Essentially, 
> > supermin
> >assembles a minimal chroot which is then loaded as qemu boots and then
> >carries out the guestfish operations. Since supermin just copies binaries
> >from the host, it cannot create foreign chroots either.
> 
> This is unfortunate, as otherwise guestfish would be universal, but it's
> understandable.
> 
> I initially wondered whether foreign architecture guestfish/supermin
> binaries couldn't be run through qemu-user-static, but that solution
> would be just far too hacky I think.

I think how "hacky" it is depends on how you look at it. The advantage is, that
then you only have to maintain the "native" codepath which you run under
emulation if you want a foreign arch binary. I just tried it out and it seems
to work as expected. Big downside: running qemu (from guestfish) inside qemu
(from qemu-user-static) is *ralllyy sloow*. XD

But I think it's a good idea to have for the sake of symmetry. We'd then have:

 - guestfish
 - fastest method when creating a native image
 - slowest method when creating a foreign image
 - initramfs
 - medium speed for both native and foreign images

Then the new tool could have a --method switch with possible values:

 - "auto" choose guestfish if installed for native images and initramfs
   otherwise
 - "guestfish" forces guestfish for native and foreign
 - "initramfs" forces initramfs for native and foreign

> > A disadvantage is, that this only works for architectures for which qemu 
> > knows
> > how to boot them without kernel and initrd from the outside. This means that
> > mips* and s390x are not supported and neither are most of the Debian ports
> > architectures. I don't know how to solve this other than by teaching
> > autopkgtest-virt-qemu that now it needs three input files: the kernel, the
> > initrd and the rootfs.
> 
> Since these are QEMU options, I think autopkgtest-virt-qemu's existing
> --qemu-options could be used to supply kernel/initrd/rootfs?

Yes, that might work. Though from reading the autopkgtest-virt-qemu man page I
think this will not support paths with spaces. On the other hand, the man page
also says that additional arguments can be supplied via a file that contains
one argument per line.

> > Another disadvantage is, that the output can never be bit-by-bit
> > reproducible because grub-install is unreproducible.
> > 
> > I've worked on this in context with replacing vmdb2 in 
> > autopkgtest-build-qemu
> > so that sbuild-qemu-create (maintained by Christian Kastner) can be run 
> > without
> > superuser privileges.  We've tried to approach the vmdb2 author but Lars is
> > reluctant to include such drastic changes:
> > https://gitlab.com/larswirzenius/vmdb2/-/issues/62
> > 
> > So I'll probably release yet another 
> > https://wiki.debian.org/SystemBuildTools
> > because the existing solutions don't do what I want. The closest is probably
> > debos which can be run without being root because everything is done inside
> > qemu. But it also doesn't solve the hard problem of installing a bootloader,
> > leaving it to the user: https://github.com/go-debos/debos/issues/137
> 
> I think this is a fantastic idea, not just because sbuild-qemu would be
> a major beneficiary but because there is a general gap here: there
> currently is no tool that can easily create images (1) without root and
> (2) for arbitrary architectures.
> 
> Having such a tool would enable many upstreams to easily test their
> software on other architectures, e.g. via GitHub Actions. PowerPC and
> RISC-V are interesting contenders but only the fewest projects test
> them. However, our buildds and debci discover issues all the time.
> 
> Additionally,  it's really useful to have an on-demand, throwaway
> porterbox when you need one. I probably use sbuild-qemu-boot more than
> sbuild-qemu, and the former is just a QEMU launcher. I use it even for
> native architectures because it's basically a container-like workspace
> where I can do Bad Stuff without fear of polluting or trashing my system.
> 
> I've run out of ideas how to easily solve this, other than what
> Francesco suggested (e.g. create a base with FAI or preseed.cfg, then
> finalize by running 

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Christian Kastner]

Johannes,

thank you for this enlightening update!

You're far ahead of any ideas I've been having (some of which I
attribute to Franscesco), but I'd nevertheless like to contribute a few
thoughts just in case.

On 2022-03-23 12:43, Johannes Schauer Marin Rodrigues wrote:
> Quoting Francesco Poli (2022-03-23 00:09:21)
> Full: the problem with the current version of 
> mmdebstrap-autopkgtest-build-qemu
> is, that it can only build qemu images for the native architecture. This is
> because it relies on guestfish. Guestfish will never be able to operate on
> foreign architecture qemu guests. This is because:
> 
>  - guestfish sets architecture specific options at compile time. This means
>that every guestfish binary can only be used for qemu guests of the same
>architecture as that binary and this cannot be changed at runtime
> 
>  - guestfish relies on another program called supermin. Essentially, supermin
>assembles a minimal chroot which is then loaded as qemu boots and then
>carries out the guestfish operations. Since supermin just copies binaries
>from the host, it cannot create foreign chroots either.

This is unfortunate, as otherwise guestfish would be universal, but it's
understandable.

I initially wondered whether foreign architecture guestfish/supermin
binaries couldn't be run through qemu-user-static, but that solution
would be just far too hacky I think.

> This means we have to replace guestfish by something else. I'm not aware that
> this already exists so I wrote a proof-of-concepts that does what we need. It
> works by first building a kernel and initramfs and then booting qemu with 
> both.
> The initramfs contains scripts that partition the disk, copy the rootfs and
> install the bootloader:
> 
> main script: http://paste.debian.net/1235312/
> initramfs-hook: http://paste.debian.net/1235313/
> initramfs-script: http://paste.debian.net/1235314/

This is really neat!

> This works for both, foreign and native architectures. But since running
> mmdebstrap to build the initramfs is far slower than supermin, I'd like to
> fallback to using guestfish in the native case. So I have to combine above
> script with mmdebstrap-autopkgtest-build-qemu.
> 
> I'm yet unsure whether I want to make these more general so that they can be
> used for other purposes or whether the script should specifically build
> autopkgtest qemu images.
> 
> A disadvantage is, that this only works for architectures for which qemu knows
> how to boot them without kernel and initrd from the outside. This means that
> mips* and s390x are not supported and neither are most of the Debian ports
> architectures. I don't know how to solve this other than by teaching
> autopkgtest-virt-qemu that now it needs three input files: the kernel, the
> initrd and the rootfs.

Since these are QEMU options, I think autopkgtest-virt-qemu's existing
--qemu-options could be used to supply kernel/initrd/rootfs?

> Another disadvantage is, that the output can never be bit-by-bit reproducible
> because grub-install is unreproducible.
> 
> I've worked on this in context with replacing vmdb2 in autopkgtest-build-qemu
> so that sbuild-qemu-create (maintained by Christian Kastner) can be run 
> without
> superuser privileges.  We've tried to approach the vmdb2 author but Lars is
> reluctant to include such drastic changes:
> https://gitlab.com/larswirzenius/vmdb2/-/issues/62
> 
> So I'll probably release yet another https://wiki.debian.org/SystemBuildTools
> because the existing solutions don't do what I want. The closest is probably
> debos which can be run without being root because everything is done inside
> qemu. But it also doesn't solve the hard problem of installing a bootloader,
> leaving it to the user: https://github.com/go-debos/debos/issues/137

I think this is a fantastic idea, not just because sbuild-qemu would be
a major beneficiary but because there is a general gap here: there
currently is no tool that can easily create images (1) without root and
(2) for arbitrary architectures.

Having such a tool would enable many upstreams to easily test their
software on other architectures, e.g. via GitHub Actions. PowerPC and
RISC-V are interesting contenders but only the fewest projects test
them. However, our buildds and debci discover issues all the time.

Additionally,  it's really useful to have an on-demand, throwaway
porterbox when you need one. I probably use sbuild-qemu-boot more than
sbuild-qemu, and the former is just a QEMU launcher. I use it even for
native architectures because it's basically a container-like workspace
where I can do Bad Stuff without fear of polluting or trashing my system.

I've run out of ideas how to easily solve this, other than what
Francesco suggested (e.g. create a base with FAI or preseed.cfg, then
finalize by running commands in the VM just as sbuild-qemu-update does
it). Really looking forward to what you are working on.

If you need a test user, I'll happily volunteer!

Best,

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Hi Francesco,

Quoting Francesco Poli (2022-03-23 00:09:21)
> Do I understand correctly that mmdebstrap-autopkgtest-build-qemu is currently
> [included] in the Debian source package, but not shipped in the Debian binary
> package?
> 
>   $ dpkg -L mmdebstrap | grep qemu
> 
> gives no output.
> 
> [included]: 
> 
> 
> Are you going to ship the script in the next version of the binary package?

short answer: no.

A bit longer: The perfect is again the enemy of the good.

Full: the problem with the current version of mmdebstrap-autopkgtest-build-qemu
is, that it can only build qemu images for the native architecture. This is
because it relies on guestfish. Guestfish will never be able to operate on
foreign architecture qemu guests. This is because:

 - guestfish sets architecture specific options at compile time. This means
   that every guestfish binary can only be used for qemu guests of the same
   architecture as that binary and this cannot be changed at runtime

 - guestfish relies on another program called supermin. Essentially, supermin
   assembles a minimal chroot which is then loaded as qemu boots and then
   carries out the guestfish operations. Since supermin just copies binaries
   from the host, it cannot create foreign chroots either.

This means we have to replace guestfish by something else. I'm not aware that
this already exists so I wrote a proof-of-concepts that does what we need. It
works by first building a kernel and initramfs and then booting qemu with both.
The initramfs contains scripts that partition the disk, copy the rootfs and
install the bootloader:

main script: http://paste.debian.net/1235312/
initramfs-hook: http://paste.debian.net/1235313/
initramfs-script: http://paste.debian.net/1235314/

This works for both, foreign and native architectures. But since running
mmdebstrap to build the initramfs is far slower than supermin, I'd like to
fallback to using guestfish in the native case. So I have to combine above
script with mmdebstrap-autopkgtest-build-qemu.

I'm yet unsure whether I want to make these more general so that they can be
used for other purposes or whether the script should specifically build
autopkgtest qemu images.

A disadvantage is, that this only works for architectures for which qemu knows
how to boot them without kernel and initrd from the outside. This means that
mips* and s390x are not supported and neither are most of the Debian ports
architectures. I don't know how to solve this other than by teaching
autopkgtest-virt-qemu that now it needs three input files: the kernel, the
initrd and the rootfs.

Another disadvantage is, that the output can never be bit-by-bit reproducible
because grub-install is unreproducible.

I've worked on this in context with replacing vmdb2 in autopkgtest-build-qemu
so that sbuild-qemu-create (maintained by Christian Kastner) can be run without
superuser privileges.  We've tried to approach the vmdb2 author but Lars is
reluctant to include such drastic changes:
https://gitlab.com/larswirzenius/vmdb2/-/issues/62

So I'll probably release yet another https://wiki.debian.org/SystemBuildTools
because the existing solutions don't do what I want. The closest is probably
debos which can be run without being root because everything is done inside
qemu. But it also doesn't solve the hard problem of installing a bootloader,
leaving it to the user: https://github.com/go-debos/debos/issues/137

I also still need a name for this tool.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2022-01-13 23:52:02)
> On Mon, 10 Jan 2022 23:18:36 +0100 Johannes Schauer Marin Rodrigues wrote:
> > You could set TMPDIR to a location that has enough space.
> 
> I had tried, if you recall.
> But you told me that the temporary directory must be world-writable
> (and preferably with the sticky bit set) and all the directories in the
> path must be world-readable.
> 
> This rules out anything within my home directory (setting my home
> directory as world-readable is out of the question).

I just remembered this message of yours and it occurred to me that maybe the
following bit might be interesting to you. If you have a directory like this:

/home/username/tmp

And if /home/username is set to 750, then an unshared process (or any process
other than one from your user or group) will not be able to read
/home/username/tmp either, even if you chmod /home/username/tmp to 1777.

But to fix this, there is another way than making /home/username
world-readable. You can also chmod /home/username to 751. That way, the
executable bit is set on your home directory but the readable bit is not. This
means that other processes are able to access directories below /home/username
if they know their path but they can *not* get a directory listing of
/home/username.

Maybe you already know this in which case, sorry for the noise. :)

In any case I think that this detail about the executable bit versus the
readable bit on directories is not widely known and I should probably add some
documentation to the mmdebstrap man page about this to inform users that the
path to their TMPDIR does *not* need to be world-readable, just
world-executable.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-03-04 19:22:11)
> > I've put this into the examples directory for now. I'm not sure whether I
> > want to make this into a script that can be put into PATH and which would
> > then also need options to set the image size and distribution.
> 
> I will comment on your version of the script later on.
> 
> For the time being, I tried to test it.
> I cannot understand why, but it fails for me:
> 
>   I: automatically chosen mode: unshare
>   I: chroot architecture amd64 is equal to the host's architecture
>   I: automatically chosen format: tar
>   I: using /dev/shm/mmdebstrap.oJ0dezfu6T as tempdir
>   [...]
>   I: installing essential packages...
>   done
>   Selecting previously unselected package base-files.
>   dpkg: regarding ...//base-files_12.2_amd64.deb containing base-files, 
> pre-dependency problem:
>base-files pre-depends on awk
> awk is not installed.
>   
>   dpkg: warning: ignoring pre-dependency problem!
>   (Reading database ... 0 files and directories currently installed.)
>   Preparing to unpack ...//base-files_12.2_amd64.deb ...
>   [...]
>   Processing triggers for libc-bin (2.33-7) ...
>   dpkg (subprocess): unable to execute installed libc-bin package 
> post-installation script (/var/lib/dpkg/info/libc-bin.postinst): No such file 
> or directory
>   dpkg: error processing package libc-bin (--install):
>installed libc-bin package post-installation script subprocess returned 
> error exit status 2
>   Errors were encountered while processing:
>dash
>libgssapi-krb5-2:amd64
>login
>libpam-modules:amd64
>libpam-runtime
>libc-bin
>   E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR 
> /usr/sbin/chroot /dev/shm/mmdebstrap.oJ0dezfu6T dpkg --install 
> --force-depends --status-fd=<$fd> 
> /var/cache/apt/archives//base-files_12.2_amd64.deb [...] 
> /var/cache/apt/archives//zlib1g_1%3a1.2.11.dfsg-2_amd64.deb failed
>   W: listening on child socket failed: 
>   I: removing tempdir /dev/shm/mmdebstrap.oJ0dezfu6T...
>   E: mmdebstrap failed to run
> 
> 
> The most surprising thing is that I basically get the same errors with
> my own version of the script.
> The same script that had worked yesterday!
> Well, actually, what I reported above is taken from the output of the my own 
> version of the script...   :-/
> 
> Is there any major uninstallability going on in Debian unstable?
> 
> My (host) system has had an upgrade that changed the running kernel and
> a few other packages:
> 
>   [INSTALL, DEPENDENCIES] linux-image-5.16.0-3-amd64:amd64 5.16.11-1
>   [UPGRADE] libbpf0:amd64 1:0.5.0-1 -> 1:0.7.0-2
>   [UPGRADE] libhivex0:amd64 1.3.21-1+b5 -> 1.3.21-1.1
>   [UPGRADE] libssh2-1:amd64 1.10.0-2 -> 1.10.0-3
>   [UPGRADE] libwin-hivex-perl:amd64 1.3.21-1+b5 -> 1.3.21-1.1
>   [UPGRADE] linux-image-amd64:amd64 5.16.7-2 -> 5.16.11-1
>   [UPGRADE] linux-libc-dev:amd64 5.16.7-2 -> 5.16.11-1
> 
> Do you think that any of these package upgrade could have caused the
> regression I am seeing?
> 
> Otherwise, what could be the root cause of the issue?

The problem is dash. I've already fixed it here:

https://salsa.debian.org/debian/dash/-/merge_requests/12

So the problem should fix itself once your mirror has the new dash version.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-03-01 23:53:27)
> I performed the first test: I seem to be able to use the resulting image as a
> testbed for autopkgtest!

awesome! Feel free to close this bug then.

> > Can you share the final version of your script? Maybe I should include it
> > somewhere in the mmdebstrap packaging.
> 
> Well, I would be happy to see it included in the package.
> A very preliminary version (basically, the first one that produced a
> usable image, after all the failed attempts since the beginning of this
> bug log!) is attached.
> 
> As you can see from the various FIXME and TODO comments, there's a lot
> of room for improvement.
> Of course, if you feel like suggesting modifications (especially those
> that implement any of the imagined developments described in the FIXME
> or TODO comments), please go ahead!
> 
> Looking forward to receiving your feedback. Bye and thanks for your kind
> assistance!

I used your FIXME and TODO comments to put the instructions from the man page
into a finished script:

https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/examples/mmdebstrap-autopkgtest-qemu

Mainly there are now no temporary files anymore and everything works in a
single shell pipeline.

I'm afraid that this will not work on architectures other than amd64 and i386
for the reasons given in the comments at the top of the script.

I've put this into the examples directory for now. I'm not sure whether I want
to make this into a script that can be put into PATH and which would then also
need options to set the image size and distribution.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-03-01 00:49:50)
> > > However, it leaves a lot of "production scrap" within /dev/shm :
> > >
> > >  $ du --si -s /dev/shm/.guestfs-1000/
> > >  755M/dev/shm/.guestfs-1000/
> > >
> > > More than 750 MB of "production scrap"!
> > >
> > > Why does guestfish do this?
> > 
> > I don't know and I don't have these files in my own $TMPDIR.
> 
> I added a
> 
>   rm -Rf ${TMPDIR}/.guestfs-*/
> 
> at the end of the script...

Interesting. I don't see a directory like that anywhere on my system. Not even
while guestfish is running.

> > In any case, if it works for you now, maybe you can close this bug?
> 
> Well, I first need to check that it actually works as a testbed for
> autopkgtest (this was the initial goal of this incredible journey, if
> you recall!).

Yes. :)

> Moreover, I tried to modify the script to more closely follow the
> [streamlined suggestions].
> 
> [streamlined suggestions]: 
> 
> 
> And, after doing so, the image was no longer able to boot!
> It hanged (seemingly forever) while booting from hard disk.
> 
> I performed various tests (basically bisecting between the working
> script and the failing one).
> I found out that the issue was in the suggested 'extlinux.conf':
> 
> $ diff -u bin/BAD_mmdebstrap-autopkgtest-qemu bin/mmdebstrap-autopkgtest-qemu 
> --- bin/BAD_mmdebstrap-autopkgtest-qemu 2022-03-01 00:05:16.418739282 +0100
> +++ bin/mmdebstrap-autopkgtest-qemu 2022-03-01 00:27:14.874469123 +0100
> @@ -134,7 +134,7 @@
>  
>  label linux
>  kernel /vmlinuz
> -append initrd=/initrd.img root=/dev/vda1 console=ttyS0
> +append initrd=/initrd.img root=/dev/vda1 rw net.ifnames=0 console=ttyS0
>  EOF
>  
>  $GUESTFISH --new debian-unstable.img=disk:"$SIZE" -- \
> 
> 
> I don't understand why, but this is how I fixed the script.
> Should the documentation be fixed?

Yes, thank you! I can confirm that the "rw" kernel cmdline option is indeed
essential (I don't know why) so thanks for finding this before I make a release
with wrong instructions.

Can you share the final version of your script? Maybe I should include it
somewhere in the mmdebstrap packaging.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Hi Francesco,

Quoting Francesco Poli (2022-02-26 00:35:01)
> At least, it completes without any error.
> And it generates a 'debian-unstable.qcow2' image that can boot, if I
> test it with:
> 
>   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> unix:/tmp/ttyS0,server,nowait -drive  
> "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
> 
> I can even login as root (password-less!) in the graphical window that
> QEMU shows!
> 
> Now, I remember that you said I should test the serial console with
> minicom on the /tmp/ttyS0 socket.
> I typically use picocom for serial console connections.
> Is there a way to make picocom connect to a UNIX socket?
> 
> Maybe I should try after using socat, as suggested in this [howto].
> 
> [howto]: 
> 
> Or is there a better way?

I don't know about picocom. Either of the following should work:

$ minicom -D unix#/tmp/ttyS0
$ nc -U /tmp/ttyS0
$ socat - UNIX-CONNECT:/tmp/ttyS0

> However, it leaves a lot of "production scrap" within /dev/shm :
>
>  $ du --si -s /dev/shm/.guestfs-1000/
>  755M/dev/shm/.guestfs-1000/
>
> More than 750 MB of "production scrap"!
>
> Why does guestfish do this?

I don't know and I don't have these files in my own $TMPDIR.

In any case, if it works for you now, maybe you can close this bug?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-02-24 23:14:43)
> On Wed, 23 Feb 2022 00:03:01 +0100 Francesco Poli wrote:
> 
> [...]
> > I will (hopefully soon) try with TMPDIR=/dev/shm
> > and let you know.
> 
> Unfortunately, it does not seem to work this way, either.
> 
>   I: automatically chosen mode: unshare
>   I: chroot architecture amd64 is equal to the host's architecture
>   I: automatically chosen format: tar
>   I: using /dev/shm/mmdebstrap.kgUCDAC2QZ as tempdir
>   [...]
>   I: cleaning package lists and apt cache...
>   done
>   done
>   I: creating tarball...
>   I: done
>   I: removing tempdir /dev/shm/mmdebstrap.kgUCDAC2QZ...
>   I: success in 82.0568 seconds
>   libguestfs: error: open: usr/lib/SYSLINUX/mbr.bin: No such file or directory
> 
> and the resulting 'debian-unstable.img' image (size: 8.0 GiB == 8.6 GB)
> fails to boot with ("Booting from Hard Disk..." which never completes),
> if I test it with the following command:
> 
>   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> unix:/tmp/ttyS0,server,nowait -drive 
> "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

Yes, it fails because you don't have /usr/lib/SYSLINUX/mbr.bin on your system.
Did you install the syslinux package? Sylinux/extlinux is the bootloader, so it
makes sense that the result doesn't boot after this error message.

Also notice, that the instructions of how to run all of this were recently
streamlined by another user, so you might want to double-check that your
scripts do the same thing:

https://gitlab.mister-muffin.de/josch/mmdebstrap/src/branch/main/mmdebstrap#L6844

If you don't have any luck with extlinux, then I recently figured out a way to
boot with grub2:

$ mmdebstrap --include=linux-image-amd64,grub2,systemd-sysv unstable 
rootfs.tar
$ guestfish -N debian-unstable.img=disk:2G -- \
part-disk /dev/sda mbr : \
part-set-bootable /dev/sda 1 true : \
mkfs ext2 /dev/sda1 : \
set-label /dev/sda1 rootfs : \
mount /dev/sda1 / : \
tar-in rootfs.tar / xattrs:true : \
command "grub-install /dev/sda" : \
command update-grub : \
sync : umount / : shutdown
$ qemu-system-x86_64 -m 512M -enable-kvm debian-unstable.img

This example is missing all the autopkgtest related bits but it's bootable.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-01-13 23:52:02)
> On Mon, 10 Jan 2022 23:18:36 +0100 Johannes Schauer Marin Rodrigues
> wrote:
> 
> [...]
> > You could set TMPDIR to a location that has enough space.
> 
> I had tried, if you recall.
> But you told me that the temporary directory must be world-writable
> (and preferably with the sticky bit set) and all the directories in the
> path must be world-readable.
> 
> This rules out anything within my home directory (setting my home
> directory as world-readable is out of the question).
> 
> We have already seen that my /tmp lives in a partition which is too
> small.
> 
> I would rather not create some non-FHS directory for this specific
> purpose: it would effectively be a second non-standard /tmp directory
> somewhere...
> 
> So, what's left?!?
> 
> Is /dev/shm a possible choice?
> 
>   $ df --si /dev/shm
>   Filesystem  Size  Used Avail Use% Mounted on
>   tmpfs   8.3G  108M  8.2G   2% /dev/shm
> 
> It's half the size of my main memory and it's within the main memory.
> Should I try to set TMPDIR=/dev/shm ?

yes, I think that's worth a try.

Other than that, you could run mmdebstrap with --mode=fakechroot which, in
comparison to unshare mode, doesn't require the world-readable path for the
TMPDIR.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2022-01-10 23:12:34)
> On Sat, 08 Jan 2022 14:36:35 +0100 Johannes Schauer Marin Rodrigues wrote:
> 
> [...]
> > Please try again once 0.8.3-1 hits your mirror.
> 
> I have just tried:
> 
>   $ apt policy mmdebstrap 
>   mmdebstrap:
> Installed: 0.8.3-1
> Candidate: 0.8.3-1
> Version table:
>*** 0.8.3-1 800
>   800 http://deb.debian.org/debian testing/main amd64 Packages
>   500 http://deb.debian.org/debian unstable/main amd64 Packages
>   100 /var/lib/dpkg/status
>   $ mmdebstrap-autopkgtest-qemu 8GiB
>   I: automatically chosen mode: unshare
>   I: chroot architecture amd64 is equal to the host's architecture
>   I: automatically chosen format: tar
>   I: using /tmp/mmdebstrap.5tbbK6Pj4J as tempdir
>   [...]
>   Unpacking linux-image-5.15.0-2-amd64 (5.15.5-2) ...
>   dpkg: error processing archive 
> /tmp/apt-dpkg-install-GmBlUJ/69-linux-image-5.15.0-2-amd64_5.15.5-2_amd64.deb 
> (--unpack):
>cannot copy extracted data for 
> './lib/modules/5.15.0-2-amd64/kernel/sound/usb/snd-usb-audio.ko' to 
> '/lib/modules/5.15.0-2-amd64/kernel/sound/usb/snd-usb-audio.ko.dpkg-new': 
> failed to write (No space left on device)
>   dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
>   [...]
>   E: Sub-process env returned an error code (1)
>   E: run_chroot failed: E: apt-get -o Dir::Bin::dpkg=env -o 
> DPkg::Options::=--unset=TMPDIR -o DPkg::Options::=dpkg -o 
> DPkg::Chroot-Directory=/tmp/mmdebstrap.5tbbK6Pj4J --yes install 
> -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false 
> ?narrow(?or(?archive(^sid$),?codename(^sid$)),?architecture(amd64),?or(?priority(required),?priority(important)))
>  linux-image-amd64 failed
>   W: listening on child socket failed: 
>   I: removing tempdir /tmp/mmdebstrap.5tbbK6Pj4J...
>   E: mmdebstrap failed to run
> 
> 
> So I am back to the previous situation.
> My /tmp is not big enough for the needs of mmdebstrap:
> 
>   $ df --si /tmp
>   Filesystem  Size  Used Avail Use% Mounted on
>   /dev/md3869M  930k  806M   1% /tmp
> 
> Have you had a chance to investigate why you need less than 200 MB,
> while I need more than 869 MB?
> 
> Any idea how to solve this issue?

You could set TMPDIR to a location that has enough space.

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2022-01-06 19:22:28)
> On Wed, 31 Mar 2021 22:45:02 +0200 Francesco Poli wrote:
> 
> [...]
> > I cannot understand why you needed less than 200 MB ...
> 
> Hello again Johannes,
> I decided to try again (with mmdebstrap/0.8.2-1 and the attached
> work-in-progress script), but I obtained the following error, which is
> completely new to me:
> 
>   $ cd ~/Downloads/
>   $ mmdebstrap-autopkgtest-qemu 8GiB
>   I: automatically chosen mode: unshare
>   I: chroot architecture amd64 is equal to the host's architecture
>   I: automatically chosen format: tar
>   I: using /tmp/mmdebstrap.YCxe0VQ5LQ as tempdir
>   dpkg: warning: failed to open configuration file '${HOME}/.dpkg.cfg' for 
> reading: Permission denied
>   I: running apt-get update...
>   done
>   I: downloading packages with apt...
>   done
>   E: nothing got downloaded
>   W: listening on child socket failed: Undefined subroutine 
> ::Cover::get_coverage called at /usr/bin/mmdebstrap line 105.
> 
>   I: removing tempdir /tmp/mmdebstrap.YCxe0VQ5LQ...
> 
> And the resulting tar archive is empty:
> 
>   $ ls -altrF --si
>   total 17k
>   drwxrwx---   2 $USER $USER 4.1k Jan  6 19:10 ./
>   -rw-r--r--   1 $USER $USER0 Jan  6 19:10 debian-unstable.tar
>   drwxr-x--- 113 $USER $USER  13k Jan  6 19:12 ../
> 
> 
> What's going on?

that was #1003191. Please try again once 0.8.3-1 hits your mirror.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Tue, 30 Mar 2021 02:04:03 +0200 Johannes Schauer Marin Rodrigues wrote:

[...]
> That depends on what you want to do inside that qemu environment. And it's
> totally possible to create a 10 GB disk image in your $HOME while only using
> 200 MB in your /tmp for the chroot.

Well... I tried dropping the TMPDIR setting and exporting in the script.
But:

  $ cd ~/Downloads/
  $ mmdebstrap-autopkgtest-qemu 8GiB
  I: automatically chosen mode: unshare
  I: chroot architecture amd64 is equal to the host's architecture
  I: automatically chosen format: tar
  I: using /tmp/mmdebstrap.h6DOq1LzM2 as tempdir
  [...]
  Setting up linux-image-5.10.0-5-amd64 (5.10.26-1) ...
  [...]
  cp: error writing 
'/var/tmp/mkinitramfs_chjGze//usr/lib/modules/5.10.0-5-amd64/kernel/drivers/input/mouse/psmouse.ko':
 No space left on device
  E: Sub-process /usr/bin/dpkg returned an error code (1)
  E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR 
/usr/sbin/chroot /tmp/mmdebstrap.h6DOq1LzM2 apt-get --yes install 
-oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false dmidecode linux-image-amd64 
logrotate sensible-utils fdisk rsyslog systemd libpam-modules-bin iproute2 
vim-common passwd iputils-ping init e2fsprogs kmod adduser tzdata ifupdown udev 
vim-tiny libreadline8 gcc-10-base mawk apt apt-utils gpgv gcc-9-base mount 
libpam-runtime whiptail debian-archive-keyring procps debconf-i18n tasksel-data 
readline-common nano netbase isc-dhcp-client systemd-sysv nftables cpio debconf 
less isc-dhcp-common libpam-modules cron failed
  W: listening on child socket failed: 
  I: removing tempdir /tmp/mmdebstrap.h6DOq1LzM2...


The same happens with:

  $ mmdebstrap-autopkgtest-qemu 1GiB


I cannot understand why you needed less than 200 MB ...


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpm3w8jtm6fC.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2021-03-29 23:26:27)
> On Sun, 28 Mar 2021 21:04:24 +0200 Johannes Schauer Marin Rodrigues wrote:
> 
> > Quoting Francesco Poli (2021-03-28 20:24:12)
> > > Well, the problem is probably my total ignorance about unshare, user
> > > namespaces, and so forth...
> > > 
> > > First of all, I do not know what exactly subuid and subgid are.
> > 
> > man subuid? ;)
> 
> OK, I have read subuid(5) man page.
> Now I know which subordinate UIDs my user can use within a namespace.
> 
> But do I understand correctly that I cannot predict which one will be
> used next time I try to start mmdebstrap in unshare mode?

Theoretically your user can indeed use any uid from that range for its purposes
but in practice I never saw a program choose the uid mapping arbitrarily.
Instead (and that's also what mmdebstrap does) it will choose the first subuid
that you can use for the root user (uid 0) and then all the remaining uids from
then on. So my /etc/subuid says:

josch:10:65536

So my root user in unshare mode will always get the uid 10 as seen from the
outside and a user with uid 1000 inside the unshare environment will have uid
101000 as seen from the outside.

This is what happens when you use tools like lxc-usernsexec without the -m
option which allows you to use a custom range. Other programs require you to
know your subuid offset from /etc/subuid and you have to provide it manually
(see systemd-nspawn in the mmdebstrap man page).

But I don't think that this kind of information belongs into the mmdebstrap man
page.

> > > I suppose they have something to do with the uid and gid seen from within
> > > the namespace (I noticed that the temporary directory was created with
> > > high values of uid and gid, not corresponding to any existing user or
> > > group).
> > 
> > Correct. So if some directory on the path of your $TMPDIR is not accessible 
> > by
> > that user with the very high uid/gid, then it obviously cannot create the
> > chroot directory.
> 
> This seems to imply that I cannot use anything within my home directory
> as $TMPDIR ...
> Is this the case?

Not necessarily. If your $HOME is readable by "everybody" and if you create a
directory inside your $HOME which allows "everybody" to create stuff in it then
you can use that as your $TMPDIR. Best even if you chmod that directory with
the sticky bit in the same way as it's done for /tmp. If the directory is not
directly inside your $HOME, then all directories in the path must be world
readable. Only the last component must be world writable.

I think the "world writable" assumption is a fair one for a temporary
directory.

> If this is indeed the case, and /tmp is too tightly sized (SSD partition of
> size about 870 MB), where can I put my $TMPDIR ?

How many chroots do you want to put into your /tmp?

$ mmdebstrap unstable | wc -c
...
196625920

That's 188 MB.

> /dev/shm seems to be risky (its size is about 8.3 GB, less than half the
> system memory).
> 
> Suggestions?
> 
> I am assuming that creating a QEMU/KVM image of size, say, 200 MiB is
> not enough as an autopkgtest-virt-qemu testbed.
> I was aiming at 4 GiB or 8 GiB, but maybe I am off-track here.
> Please advise!

That depends on what you want to do inside that qemu environment. And it's
totally possible to create a 10 GB disk image in your $HOME while only using
200 MB in your /tmp for the chroot.

> > > But what is not clear to me is what I am supposed to do with this
> > > problem.
> > > 
> > > What would you suggest?
> > 
> > Your whole problem started because you decided to deviate from the default 
> > and
> > chose a different TMPDIR. If you use custom options, then you should know 
> > what
> > you are doing. If you don't set TMPDIR, then it will work. If you do set
> > TMPDIR, then you have to make sure that whatever path TMPDIR points to is 
> > setup
> > in a way that the unshared user is able to access it and write stuff into 
> > it.
> 
> As said above: I chose a different TMPDIR, since /tmp turned out to not
> be big enough.
> I am open to suggestions on how to work around this issue.
> 
> And by the way, I still do not understand how the --unshare-helper can
> help me... as long as it can help me at all!
> Could you please clarify?

It cannot help you. The option is undocumented because its interface is
unstable. What the --unshare-helper option does is to provide you similar
functionality to lxc-usernsexec -- lxc-unshare -s 'MOUNT|PID|UTSNAME|IPC'
without installing the lxc package.

> Thanks for your time and great patience.   :-)

You are welcome. I would love to hear things I could add to the mmdebstrap man
page. I fear that an introduction to Linux user namespaces would not be
fitting.

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 28 Mar 2021 21:04:24 +0200 Johannes Schauer Marin Rodrigues wrote:

> Quoting Francesco Poli (2021-03-28 20:24:12)
> > Well, the problem is probably my total ignorance about unshare, user
> > namespaces, and so forth...
> > 
> > First of all, I do not know what exactly subuid and subgid are.
> 
> man subuid? ;)

OK, I have read subuid(5) man page.
Now I know which subordinate UIDs my user can use within a namespace.

But do I understand correctly that I cannot predict which one will be
used next time I try to start mmdebstrap in unshare mode?

> 
> > I suppose they have something to do with the uid and gid seen from
> > within the namespace (I noticed that the temporary directory was
> > created with high values of uid and gid, not corresponding to any
> > existing user or group).
> 
> Correct. So if some directory on the path of your $TMPDIR is not accessible by
> that user with the very high uid/gid, then it obviously cannot create the
> chroot directory.

This seems to imply that I cannot use anything within my home directory
as $TMPDIR ...
Is this the case?

If this is indeed the case, and /tmp is too tightly sized (SSD
partition of size about 870 MB), where can I put my $TMPDIR ?

/dev/shm seems to be risky (its size is about 8.3 GB, less than half
the system memory).

Suggestions?

I am assuming that creating a QEMU/KVM image of size, say, 200 MiB is
not enough as an autopkgtest-virt-qemu testbed.
I was aiming at 4 GiB or 8 GiB, but maybe I am off-track here.
Please advise! 

> 
> > I tried to take a look at some man pages (unshare(1),
> > user_namespaces(7), ...) in order to understand something more, but
> > there seems to be much more information than the basics, and I failed
> > to pinpoint where I can read the bare minimum I need to know...
> > My bad, of course!
> > But anyway, if you know a good reference to a short explanation of the
> > topic (ideally a man page), I would suggest to cite it in the
> > mmdebstrap man page.
> 
> Unfortunately, I do not know of such a reference. All I learned to add unshare
> support to mmdebstrap was from reading source code of other tools. :(

I see, the good old "Use the Force, read the Source!".   ;-)

> 
> > After that, I can guess that the problem is that, inside the unshared
> > namespace, there's no permission to create/write files in my
> > ~/Downloads directory.
> 
> Or one of its parent. You also need more than write access. You also need read
> access. And directories need to be executable.
> 
> > But what is not clear to me is what I am supposed to do with this
> > problem.
> > 
> > What would you suggest?
> 
> Your whole problem started because you decided to deviate from the default and
> chose a different TMPDIR. If you use custom options, then you should know what
> you are doing. If you don't set TMPDIR, then it will work. If you do set
> TMPDIR, then you have to make sure that whatever path TMPDIR points to is 
> setup
> in a way that the unshared user is able to access it and write stuff into it.

As said above: I chose a different TMPDIR, since /tmp turned out to not
be big enough.
I am open to suggestions on how to work around this issue.

And by the way, I still do not understand how the --unshare-helper can
help me... as long as it can help me at all!
Could you please clarify?

Thanks for your time and great patience.   :-)


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpTQZCOh3x3a.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2021-03-28 20:24:12)
> Well, the problem is probably my total ignorance about unshare, user
> namespaces, and so forth...
> 
> First of all, I do not know what exactly subuid and subgid are.

man subuid? ;)

> I suppose they have something to do with the uid and gid seen from
> within the namespace (I noticed that the temporary directory was
> created with high values of uid and gid, not corresponding to any
> existing user or group).

Correct. So if some directory on the path of your $TMPDIR is not accessible by
that user with the very high uid/gid, then it obviously cannot create the
chroot directory.

> I tried to take a look at some man pages (unshare(1),
> user_namespaces(7), ...) in order to understand something more, but
> there seems to be much more information than the basics, and I failed
> to pinpoint where I can read the bare minimum I need to know...
> My bad, of course!
> But anyway, if you know a good reference to a short explanation of the
> topic (ideally a man page), I would suggest to cite it in the
> mmdebstrap man page.

Unfortunately, I do not know of such a reference. All I learned to add unshare
support to mmdebstrap was from reading source code of other tools. :(

> After that, I can guess that the problem is that, inside the unshared
> namespace, there's no permission to create/write files in my
> ~/Downloads directory.

Or one of its parent. You also need more than write access. You also need read
access. And directories need to be executable.

> But what is not clear to me is what I am supposed to do with this
> problem.
> 
> What would you suggest?

Your whole problem started because you decided to deviate from the default and
chose a different TMPDIR. If you use custom options, then you should know what
you are doing. If you don't set TMPDIR, then it will work. If you do set
TMPDIR, then you have to make sure that whatever path TMPDIR points to is setup
in a way that the unshared user is able to access it and write stuff into it.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sat, 27 Mar 2021 20:15:29 +0100 Johannes Schauer Marin Rodrigues
wrote:

> Quoting Francesco Poli (2021-03-27 16:18:26)
[...]
> > I tried to read the relevant section in the mmdebstrap(1) man page, and
> > there's some recommendation about an --unshare-helper option.
> > 
> > But to be honest, I failed to understand what I am supposed to do with
> > this --unshare-helper option.
> > Could you please clarify?
> 
> Quoting from the man page:
> 
> "For correct ownership information, the directory must be accessed from a
> user namespace with the right subuid/subgid offset"
> 
> Which part is unclear about that?

Thanks for your prompt reply!

Well, the problem is probably my total ignorance about unshare, user
namespaces, and so forth...

First of all, I do not know what exactly subuid and subgid are.

I suppose they have something to do with the uid and gid seen from
within the namespace (I noticed that the temporary directory was
created with high values of uid and gid, not corresponding to any
existing user or group).
I tried to take a look at some man pages (unshare(1),
user_namespaces(7), ...) in order to understand something more, but
there seems to be much more information than the basics, and I failed
to pinpoint where I can read the bare minimum I need to know...
My bad, of course!
But anyway, if you know a good reference to a short explanation of the
topic (ideally a man page), I would suggest to cite it in the
mmdebstrap man page.

After that, I can guess that the problem is that, inside the unshared
namespace, there's no permission to create/write files in my
~/Downloads directory.
But what is not clear to me is what I am supposed to do with this
problem.

What would you suggest?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpEOQrhSgC3a.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer Marin Rodrigues]

Quoting Francesco Poli (2021-03-27 16:18:26)
> As a consequence, I feel like giving the "unshare" mode of mmdebstrap a
> try.
> 
> I tried again with my work-in-progress script:
> 
>   $ cd ~/Downloads/
>   $ mmdebstrap-autopkgtest-qemu 8GiB
>   I: automatically chosen mode: unshare
>   I: chroot architecture amd64 is equal to the host's architecture
>   I: automatically chosen format: tar
>   I: using ${HOME}/Downloads/mmdebstrap.43LR3zoGLq as tempdir
>   E: cannot create ${HOME}/Downloads: Permission denied; cannot create 
> ${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied; cannot create 
> ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc: Permission denied; cannot 
> create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt: Permission denied; 
> cannot create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt/apt.conf.d: 
> Permission denied
>   W: listening on child socket failed: 
>   I: removing tempdir ${HOME}/Downloads/mmdebstrap.43LR3zoGLq...
>   E: unable to chdir() to parent directory of 
> ${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied
>   E: remove_tree failed
> 
> 
> OK, this does not work at all.
> I tried to read the relevant section in the mmdebstrap(1) man page, and
> there's some recommendation about an --unshare-helper option.
> 
> But to be honest, I failed to understand what I am supposed to do with
> this --unshare-helper option.
> Could you please clarify?

Quoting from the man page:

"For correct ownership information, the directory must be accessed from a
user namespace with the right subuid/subgid offset"

Which part is unclear about that?

> Please remember that my script exports TMPDIR='.', in order to avoid
> using a tightly sized /tmp partition, where several GB of data would
> definitely not fit.

I also don't understand which part of the error message above is unclear. It
tells you that you don't have permission to create the directories which means
that whatever your '${HOME}/Downloads' is (or one of its parent directories),
has permissions set such that the directory cannot be created and/or accessed.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Mon, 16 Nov 2020 00:28:57 +0100 Johannes Schauer wrote:

[...]
> Do you have the time to investigate further on this issue?

As you can see, not much time, unfortunately!:-(
And I am sad about this, believe me.

> This does not seem
> to be a problem of initramfs-tools or a problem of fakechroot not being 
> enabled
> but a weird problem with fakechroot.

I don't know whether I can get around to investigating the fakechroot
failure, but, in the meanwhile I noticed that #898446 has been closed
and a decision has been made.
It was apparently concluded that enabling
"kernel.unprivileged_userns_clone=1" by default is better than
disabling it by default.

As I have [previously said], I am not brave enough (and knowledgeable
enough on this topic) to diverge from the default Debian settings.

[previously said]: 

However, now the Debian default has changed:

  $ /sbin/sysctl kernel.unprivileged_userns_clone
  kernel.unprivileged_userns_clone = 1

As a consequence, I feel like giving the "unshare" mode of mmdebstrap a
try.

I tried again with my work-in-progress script:

  $ cd ~/Downloads/
  $ mmdebstrap-autopkgtest-qemu 8GiB
  I: automatically chosen mode: unshare
  I: chroot architecture amd64 is equal to the host's architecture
  I: automatically chosen format: tar
  I: using ${HOME}/Downloads/mmdebstrap.43LR3zoGLq as tempdir
  E: cannot create ${HOME}/Downloads: Permission denied; cannot create 
${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied; cannot create 
${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc: Permission denied; cannot create 
${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt: Permission denied; cannot 
create ${HOME}/Downloads/mmdebstrap.43LR3zoGLq//etc/apt/apt.conf.d: Permission 
denied
  W: listening on child socket failed: 
  I: removing tempdir ${HOME}/Downloads/mmdebstrap.43LR3zoGLq...
  E: unable to chdir() to parent directory of 
${HOME}/Downloads/mmdebstrap.43LR3zoGLq: Permission denied
  E: remove_tree failed


OK, this does not work at all.
I tried to read the relevant section in the mmdebstrap(1) man page, and
there's some recommendation about an --unshare-helper option.

But to be honest, I failed to understand what I am supposed to do with
this --unshare-helper option.
Could you please clarify?

Please remember that my script exports TMPDIR='.', in order to avoid
using a tightly sized /tmp partition, where several GB of data would
definitely not fit.

Thanks for your time and patience.


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpEsD_Y8fIdz.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-11-16 00:05:08)
> On Sun, 15 Nov 2020 23:09:27 +0100 Johannes Schauer wrote:
> > Quoting Francesco Poli (2020-11-15 22:57:38)
> As I have previously said, I am worried by security implications of
> setting "kernel.unprivileged_userns_clone=1" with sysctl.
> Bug #898446 is still being discussed...

I agree. Similarly I am worried about the security implications of running the
whole thing as root. It would be great if fakechroot would work. In the
meantime, I managed to track down the problem a bit. I put this shell snippet:

case "`FAKECHROOT_DETECT=1 /bin/echo`" in fakechroot*) echo LOADED;;*) echo NOT 
LOADED;;esac

Into various places like /etc/kernel/postinst.d/initramfs-tools,
/etc/kernel/postinst.d/initramfs-tools, /usr/sbin/update-initramfs,
/usr/sbin/mkinitramfs and /usr/share/initramfs-tools/hooks/klibc-utils and
found out that fakechroot still remains active all the way down to the deepest
level where the error is then produced by this line:

cp -pnL /usr/lib/klibc/bin/* "${DESTDIR}/bin"

The files in question *do* exist, what seems to be the problem are the
wildcards. For example this works:

mmdebstrap --mode=fakechroot --variant=apt --customize-hook='chroot "$1" sh 
-c "ls *"' unstable /dev/null

and so does this:

mmdebstrap --mode=fakechroot --variant=apt --customize-hook='chroot "$1" sh 
-c "ls ./*"' unstable /dev/null

But this fails:

mmdebstrap --mode=fakechroot --variant=apt --customize-hook='chroot "$1" sh 
-c "ls /*"' unstable /dev/null

So as soon as the wildcard is part of an absolute path, things start breaking.

Do you have the time to investigate further on this issue? This does not seem
to be a problem of initramfs-tools or a problem of fakechroot not being enabled
but a weird problem with fakechroot.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 15 Nov 2020 23:09:27 +0100 Johannes Schauer wrote:

[...]
> Quoting Francesco Poli (2020-11-15 22:57:38)
[...]
> >   $ mmdebstrap-autopkgtest-qemu 8GiB
> >   I: automatically chosen mode: fakechroot
> >   [...]
> 
> that's the culprit. Try using --mode=unshare. With fakechroot I see the errors
> that you see. It works with unshare mode.

As I have previously said, I am worried by security implications of
setting "kernel.unprivileged_userns_clone=1" with sysctl.
Bug #898446 is still being discussed...

[...]
> One suggestion would be to throw in a "set -e" at the beginning

Done, thanks for the suggestion!
It definitely makes sense for such a script.



-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgp7PzbTY8TiY.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-11-15 22:57:38)
> On Sun, 15 Nov 2020 21:05:08 +0100 Johannes Schauer wrote:
> > Which mode was mmdebstrap run in?
> 
>   $ mmdebstrap-autopkgtest-qemu 8GiB
>   I: automatically chosen mode: fakechroot
>   [...]

that's the culprit. Try using --mode=unshare. With fakechroot I see the errors
that you see. It works with unshare mode.

> > Maybe this is something similar to this initramfs-tools bug that popped up
> > with fakechroot:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944929
> 
> Isn't that fixed in unstable and in testing?

Yes, but the reason might be a similar one. Fakechroot works by preloading
libfakechroot.so and if any maintainer script messes with the environment
variables, then fakechroot gets disabled for that process.

> > Can you share the whole script that you used for these results?
> 
> Sure, the modified script is attached.
> 
> As I said before, the script is what I am trying to develop, based on
> your suggestions.
> It is still experimental, needs to be improved in many aspects, and has
> not yet produced any working QEMU image (!). I wanted to send it to you
> as a contribution to mmdebstrap, once it was mature enough.
> I am sending it to you as a preview (not to be included in the official
> package, yet!).

One suggestion would be to throw in a "set -e" at the beginning because
otherwise you'd have to check the exit status of each program execution
manually.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 15 Nov 2020 21:05:08 +0100 Johannes Schauer wrote:

[...]
> Which mode was mmdebstrap run in?

  $ mmdebstrap-autopkgtest-qemu 8GiB
  I: automatically chosen mode: fakechroot
  [...]

> Maybe this is something similar to this
> initramfs-tools bug that popped up with fakechroot:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944929

Isn't that fixed in unstable and in testing?

> 
> Can you share the whole script that you used for these results?

Sure, the modified script is attached.

As I said before, the script is what I am trying to develop, based on
your suggestions.
It is still experimental, needs to be improved in many aspects, and has
not yet produced any working QEMU image (!). I wanted to send it to you
as a contribution to mmdebstrap, once it was mature enough.
I am sending it to you as a preview (not to be included in the
official package, yet!).

Please note that I am running it on an amd64 box. Hence:

  $ dpkg --print-architecture
  amd64


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


mmdebstrap-autopkgtest-qemu
Description: Binary data


pgp8NMDVxsfYc.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-11-15 20:43:49)
> I re-tried with the modified guestfish command, the modified
> extlinux.conf file and requesting an 8 GiB size.
> 
> Unfortunately, the new test results in the following errors:
> 
> 
> [...]
> Setting up initramfs-tools-core (0.139) ...
> Setting up initramfs-tools (0.139) ...
> update-initramfs: deferring update (trigger activated)
> Setting up linux-image-5.9.0-2-amd64 (5.9.6-1) ...
> I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.9.0-2-amd64
> I: /initrd.img.old is now a symlink to boot/initrd.img-5.9.0-2-amd64
> I: /vmlinuz is now a symlink to boot/vmlinuz-5.9.0-2-amd64
> I: /initrd.img is now a symlink to boot/initrd.img-5.9.0-2-amd64
> /etc/kernel/postinst.d/initramfs-tools:
> update-initramfs: Generating /boot/initrd.img-5.9.0-2-amd64
> W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125b-2.fw for module 
> r8169
> [...]
> find: ‘/var/tmp/mkinitramfs_KOh7yF/lib/modules/5.9.0-2-amd64/kernel’: No such 
> file or directory
> cp: cannot stat '/usr/lib/klibc/bin/*': No such file or directory
> cp: cannot stat '/lib/klibc-*.so': No such file or directory
> E: /usr/share/initramfs-tools/hooks/klibc-utils failed with return 1.
> update-initramfs: failed for /boot/initrd.img-5.9.0-2-amd64 with 1.
> run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
> dpkg: error processing package linux-image-5.9.0-2-amd64 (--configure):
>  installed linux-image-5.9.0-2-amd64 package post-installation script 
> subprocess returned error exit status 1
> Setting up iptables (1.8.6-1) ...
> [...]
> dpkg: dependency problems prevent configuration of linux-image-amd64:
>  linux-image-amd64 depends on linux-image-5.9.0-2-amd64 (= 5.9.6-1); however:
>   Package linux-image-5.9.0-2-amd64 is not configured yet.
> 
> dpkg: error processing package linux-image-amd64 (--configure):
>  dependency problems - leaving unconfigured
> [...]
> Processing triggers for initramfs-tools (0.139) ...
> Errors were encountered while processing:
>  linux-image-5.9.0-2-amd64
>  linux-image-amd64
> E: Sub-process /usr/bin/dpkg returned an error code (1)
> E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR 
> /usr/sbin/chroot ${HOME}/Downloads/TEMP/mmdebstrap.HHRQ7rNuyL env 
> --unset=APT_CONFIG --unset=TMPDIR apt-get --yes install 
> -oAPT::Status-Fd=<$fd> -oDpkg::Use-Pty=false linux-image-amd64 adduser apt 
> apt-utils bsdmainutils cpio cron debconf debconf-i18n debian-archive-keyring 
> dmidecode e2fsprogs gcc-10-base gcc-9-base gpgv ifupdown init iproute2 
> iptables iputils-ping isc-dhcp-client isc-dhcp-common kmod less logrotate 
> mawk nano netbase whiptail libpam-modules libpam-modules-bin libpam-runtime 
> procps libreadline8 readline-common rsyslog sensible-utils passwd systemd 
> systemd-sysv udev tasksel-data tzdata fdisk mount vim-common vim-tiny failed
> W: listening on child socket failed: 
> I: removing tempdir ${HOME}/Downloads/TEMP/mmdebstrap.HHRQ7rNuyL...
> libguestfs: error: /usr/bin/supermin exited with error status 1.
> To see full error messages you may need to enable debugging.
> Do:
>   export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
> and run the command again.  For further information, read:
>   http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
> You can also run 'libguestfs-test-tool' and post the *complete* output
> into a bug report or message to the libguestfs mailing list.
> 
> 
> 
> And the directory now includes the following files:
> 
> $ ls -altrF --si
> total 218k
> drwxrwx--- 3 $USER $USER 4.1k Nov 15 20:28 ../
> -rw-r--r-- 1 $USER $USER0 Nov 15 20:30 debian-unstable.tar
> -rw-rw 1 $USER $USER  125 Nov 15 20:30 extlinux.conf
> drwxr-xr-x 2 $USER $USER 4.1k Nov 15 20:30 .guestfs-1000/
> -rw-rw 1 $USER $USER 8.6G Nov 15 20:30 debian-unstable.img
> drwxrwx--- 3 $USER $USER 4.1k Nov 15 20:30 ./
> -rw-r- 1 $USER $USER 197k Nov 15 20:30 debian-unstable.qcow2
> 
> 
> Any idea why initramfs-tools hooks are failing?

Which mode was mmdebstrap run in? Maybe this is something similar to this
initramfs-tools bug that popped up with fakechroot:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944929

Can you share the whole script that you used for these results?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Wed, 11 Nov 2020 23:27:21 +0100 Johannes Schauer wrote:

> Hi Francesco,

Hello Johannes, thanks for the updated followup!

[...]
>So a better guestfish command is this one:
> 
> guestfish -N debian-unstable.img=disk:8G -- \
[...]
> Just to make sure, here is my
> extlinux.conf:
[...]
> Also, you reported that the guestfish call fails
[...]
> maybe your
> disk size is not large enough for your tarball and that's why it fails?

I re-tried with the modified guestfish command, the modified
extlinux.conf file and requesting an 8 GiB size.

Unfortunately, the new test results in the following errors:


[...]
Setting up initramfs-tools-core (0.139) ...
Setting up initramfs-tools (0.139) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-image-5.9.0-2-amd64 (5.9.6-1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.9.0-2-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-5.9.0-2-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-5.9.0-2-amd64
I: /initrd.img is now a symlink to boot/initrd.img-5.9.0-2-amd64
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.9.0-2-amd64
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125b-2.fw for module 
r8169
[...]
find: ‘/var/tmp/mkinitramfs_KOh7yF/lib/modules/5.9.0-2-amd64/kernel’: No such 
file or directory
cp: cannot stat '/usr/lib/klibc/bin/*': No such file or directory
cp: cannot stat '/lib/klibc-*.so': No such file or directory
E: /usr/share/initramfs-tools/hooks/klibc-utils failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.9.0-2-amd64 with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.9.0-2-amd64 (--configure):
 installed linux-image-5.9.0-2-amd64 package post-installation script 
subprocess returned error exit status 1
Setting up iptables (1.8.6-1) ...
[...]
dpkg: dependency problems prevent configuration of linux-image-amd64:
 linux-image-amd64 depends on linux-image-5.9.0-2-amd64 (= 5.9.6-1); however:
  Package linux-image-5.9.0-2-amd64 is not configured yet.

dpkg: error processing package linux-image-amd64 (--configure):
 dependency problems - leaving unconfigured
[...]
Processing triggers for initramfs-tools (0.139) ...
Errors were encountered while processing:
 linux-image-5.9.0-2-amd64
 linux-image-amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR /usr/sbin/chroot 
${HOME}/Downloads/TEMP/mmdebstrap.HHRQ7rNuyL env --unset=APT_CONFIG 
--unset=TMPDIR apt-get --yes install -oAPT::Status-Fd=<$fd> 
-oDpkg::Use-Pty=false linux-image-amd64 adduser apt apt-utils bsdmainutils cpio 
cron debconf debconf-i18n debian-archive-keyring dmidecode e2fsprogs 
gcc-10-base gcc-9-base gpgv ifupdown init iproute2 iptables iputils-ping 
isc-dhcp-client isc-dhcp-common kmod less logrotate mawk nano netbase whiptail 
libpam-modules libpam-modules-bin libpam-runtime procps libreadline8 
readline-common rsyslog sensible-utils passwd systemd systemd-sysv udev 
tasksel-data tzdata fdisk mount vim-common vim-tiny failed
W: listening on child socket failed: 
I: removing tempdir ${HOME}/Downloads/TEMP/mmdebstrap.HHRQ7rNuyL...
libguestfs: error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.



And the directory now includes the following files:

$ ls -altrF --si
total 218k
drwxrwx--- 3 $USER $USER 4.1k Nov 15 20:28 ../
-rw-r--r-- 1 $USER $USER0 Nov 15 20:30 debian-unstable.tar
-rw-rw 1 $USER $USER  125 Nov 15 20:30 extlinux.conf
drwxr-xr-x 2 $USER $USER 4.1k Nov 15 20:30 .guestfs-1000/
-rw-rw 1 $USER $USER 8.6G Nov 15 20:30 debian-unstable.img
drwxrwx--- 3 $USER $USER 4.1k Nov 15 20:30 ./
-rw-r- 1 $USER $USER 197k Nov 15 20:30 debian-unstable.qcow2


Any idea why initramfs-tools hooks are failing?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgp1jXakrZd_d.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi Francesco,

Quoting Francesco Poli (2020-09-04 18:18:32)
> > I am using the attached script to build my own autopkgtest qemu images and 
> > it
> > works fine for me. Maybe you want to try again?
> 
> I have just retried with my script (which seems to do the same things as 
> yours, except that it does set the unshare mode, since I have 
> kernel.unprivileged_userns_clone = 0).
> 
> Unfortunately, I still see the same guestfish error:
> 
>   I: automatically chosen mode: fakechroot
>   I: chroot architecture amd64 is equal to the host's architecture
>   I: automatically chosen format: tar
>   I: using ${HOME}/Downloads/TEST/mmdebstrap.Cd9NV53RA7 as tempdir
>   [...]
>   I: creating tarball...
>   I: done
>   I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.Cd9NV53RA7...
>   I: success in 107.2860 seconds
>   libguestfs: error: /usr/bin/supermin exited with error status 1.
>   To see full error messages you may need to enable debugging.
>   Do:
> export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
>   and run the command again.  For further information, read:
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
>   You can also run 'libguestfs-test-tool' and post the *complete* output
>   into a bug report or message to the libguestfs mailing list.
> 
> Once again, the .qcow2 image is tiny and the .img does not boot with 
> 
>   $ qemu-system-x86_64 -enable-kvm -m 512 \
> -serial unix:/tmp/ttyS0,server,nowait -drive \
> "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"
> 
> After attempting all possible boot devices, including a network boot,
> it bails out with "No bootable device" error message.

I have some more insights. After having gotten the mmdebstrap/guestfish thing
running on salsaci, debci and jenkins I also happened to see stuff that looked
very similar to the errors you are seeing. Going over my last emails to this
thread, the guestfish command I told you about was indeed missing something:
the installation of mbr.bin. So a better guestfish command is this one:

guestfish -N debian-unstable.img=disk:8G -- \
part-disk /dev/sda mbr : \
mkfs ext2 /dev/sda1 : \
mount /dev/sda1 / : \
tar-in debian-unstable.tar / : \
copy-in "extlinux.conf" / : \
upload /usr/lib/SYSLINUX/mbr.bin /mbr.bin : \
copy-file-to-device /mbr.bin /dev/sda size:440 : \
rm /mbr.bin : \
extlinux / : \
sync : \
umount / : \
part-set-bootable /dev/sda 1 true : \
shutdown

I have no idea why, but on my system, the disk boots even without writing
mbr.bin to the first 440 byte of the disk. Just to make sure, here is my
extlinux.conf:

default linux
timeout 0

label linux
kernel /vmlinuz
append initrd=/initrd.img root=/dev/vda1 rw net.ifnames=0 console=ttyS0

Also, you reported that the guestfish call fails for you and that if you set
LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 then you get lots of lines like:

guestfsd: receive_file: reading length word
guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 
0x560e1b866690

These lines come from the tar-in command. If it fails at that stage, maybe your
disk size is not large enough for your tarball and that's why it fails?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Thu, 03 Sep 2020 22:53:13 +0200 Johannes Schauer wrote:

> Hi Francesco,

Hello Johannes, thanks for following up my bug report!

> 
> Quoting Francesco Poli (2020-05-06 19:40:37)
> > > did you get any further with this problem?
> > 
> > Unfortunately, I failed to progress any further.
> 
> I uploaded a new mmdebstrap version.
> 
> I am using the attached script to build my own autopkgtest qemu images and it
> works fine for me. Maybe you want to try again?

I have just retried with my script (which seems to do the same things as yours, 
except that it does set the unshare mode, since I have 
kernel.unprivileged_userns_clone = 0).

Unfortunately, I still see the same guestfish error:

  I: automatically chosen mode: fakechroot
  I: chroot architecture amd64 is equal to the host's architecture
  I: automatically chosen format: tar
  I: using ${HOME}/Downloads/TEST/mmdebstrap.Cd9NV53RA7 as tempdir
  [...]
  I: creating tarball...
  I: done
  I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.Cd9NV53RA7...
  I: success in 107.2860 seconds
  libguestfs: error: /usr/bin/supermin exited with error status 1.
  To see full error messages you may need to enable debugging.
  Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.

Once again, the .qcow2 image is tiny and the .img does not boot with 

  $ qemu-system-x86_64 -enable-kvm -m 512 \
-serial unix:/tmp/ttyS0,server,nowait -drive \
"file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

After attempting all possible boot devices, including a network boot,
it bails out with "No bootable device" error message.

:-(


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpIdlH9TPXII.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi Francesco,

Quoting Francesco Poli (2020-05-06 19:40:37)
> > did you get any further with this problem?
> 
> Unfortunately, I failed to progress any further.

I uploaded a new mmdebstrap version.

I am using the attached script to build my own autopkgtest qemu images and it
works fine for me. Maybe you want to try again?

Thanks!

cheers, josch#!/bin/sh

set -exu

#sudo vmdebootstrap --verbose --serial-console --distribution=sid --customize=/usr/share/autopkgtest/setup-commands/setup-testbed --user=test/test --size=20 --grub --mirror=http://127.0.0.1:3142/httpredir.debian.org/debian/ --configure-apt --apt-mirror=http://10.0.2.2:3142/httpredir.debian.org/debian/ --image=autopkgtest-sid.raw
#qemu-img convert -O qcow2 autopkgtest-sid.raw autopkgtest-sid.img
#qemu-system-x86_64 -m 500M -enable-kvm /srv/qemu/unstable-amd64-autopkgtest.qcow2


mmdebstrap --mode=unshare --variant=important \
	--include=linux-image-amd64 \
	--customize-hook='chroot "$1" passwd --delete root' \
	--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
	--customize-hook='chroot "$1" passwd --delete user' \
	--customize-hook='echo host > "$1/etc/hostname"' \
	--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
	--customize-hook=/usr/share/autopkgtest/setup-commands/setup-testbed \
	unstable debian-unstable.tar
cat << END > extlinux.conf
default linux
timeout 0

label linux
kernel /vmlinuz
append initrd=/initrd.img root=/dev/vda1 rw console=ttyS0
END
guestfish -N debian-unstable.img=disk:8G -- \
	part-disk /dev/sda mbr : \
	part-set-bootable /dev/sda 1 true :  \
	mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
	tar-in debian-unstable.tar / \
	:  extlinux / : copy-in extlinux.conf / : \
	sync : umount / : shutdown
qemu-img convert -O qcow2 debian-unstable.img debian-unstable.qcow2
sudo mv debian-unstable.qcow2 /srv/qemu/unstable-amd64-autopkgtest.qcow2
rm debian-unstable.img debian-unstable.tar


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Tue, 05 May 2020 19:34:31 +0200 Johannes Schauer wrote:

> Hi,

Hello Johannes, glad to hear from you.

> 
> did you get any further with this problem?

Unfortunately, I failed to progress any further.

I have just retried (who knows, after many system upgrades?!?).
Bad luck! Once again, I am stuck at the following error:

  [...]
  I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.9R3P2pDSAZ...
  libguestfs: error: /usr/bin/supermin exited with error status 1.
  To see full error messages you may need to enable debugging.
  Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.

and I really do not know how to debug this.

That's unfortunate, since mmdebstrap looks very promising, but I seem
to be unable to make it work correctly...:-(


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpJETvONd9zb.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

did you get any further with this problem?

cheers, josch

Quoting Francesco Poli (2020-02-24 23:51:52)
> On Sat, 22 Feb 2020 19:03:48 +0100 Johannes Schauer wrote:
> 
> [...]
> > for what it's worth I'm attaching the script I am using to re-generate my
> > autopkgtest qemu image. I uploaded a few packages today and just 
> > successfully
> > used that script so I can confirm it working.
> 
> I cannot spot any significant difference between your script and mine
> (apart from the fact that I have recently added "sync : umount / :
> shutdown" to the guestfish command-line, as you suggested)...
> 
> I really cannot understand what's going on! :-(
> 
> -- 
>  http://www.inventati.org/frx/
>  There's not a second to spare! To the laboratory!
> . Francesco Poli .
>  GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sat, 22 Feb 2020 19:03:48 +0100 Johannes Schauer wrote:

[...]
> for what it's worth I'm attaching the script I am using to re-generate my
> autopkgtest qemu image. I uploaded a few packages today and just successfully
> used that script so I can confirm it working.

I cannot spot any significant difference between your script and mine
(apart from the fact that I have recently added "sync : umount / :
shutdown" to the guestfish command-line, as you suggested)...

I really cannot understand what's going on! :-(

-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpkXfzkZehUw.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Johannes Schauer (2020-02-22 08:14:55)
> Quoting Francesco Poli (2020-02-21 23:14:12)
> > I modified my script (see the current version attached to this message) and
> > tried again.
> > 
> >   $ env | grep SOURCE
> >   SOURCE_DATE_EPOCH=1582320746
> >   $ mmdebstrap-autopkgtest-qemu
> >   [...]
> >   I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.vt5NxWQb_6...
> >   libguestfs: error: /usr/bin/supermin exited with error status 1.
> >   To see full error messages you may need to enable debugging.
> >   Do:
> > export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
> >   and run the command again.  For further information, read:
> > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
> >   You can also run 'libguestfs-test-tool' and post the *complete* output
> >   into a bug report or message to the libguestfs mailing list.
> 
> did you try running guestfish with these env variables to figure out what's
> wrong?
> 
> > The qcow2 image is suspiciously small:
> > 
> >   $ ls --si -l debian-unstable.*
> >   -rw-rw 1 $USER $GROUP 2.2G Feb 21 22:34 debian-unstable.img
> >   -rw-r- 1 $USER $GROUP 197k Feb 21 22:34 debian-unstable.qcow2
> >   -rw-r--r-- 1 $USER $GROUP 629M Feb 21 22:34 debian-unstable.tar
> > 
> > and actually fails to boot (but for different reasons, with respect to
> > the previous attempt), see the attached screenshot.
> > I tried both the qcow2 image and the raw image:
> > 
> >   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> > unix:/tmp/ttyS0,server,nowait -drive  
> > "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
> >   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> > unix:/tmp/ttyS0,server,nowait -drive 
> > "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"
> > 
> > with identical results: after attempting all possible boot devices
> > (including a network boot!), it bails out with "No bootable device"
> > error message.
> > 
> > I also tried:
> > 
> >   $ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
> >   $ SIZE="2GiB"
> >   $ guestfish --new debian-unstable_DEBUG.img=disk:"$SIZE" -- \
> > part-disk /dev/sda mbr : \
> > part-set-bootable /dev/sda 1 true : \
> > mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
> > tar-in debian-unstable.tar / : \
> > extlinux / : \
> > copy-in extlinux.conf / : \
> > sync : umount / : shutdown
> > 
> > which produced an unmanageable quantity of output strings, with an
> > apparently endless stream line pairs identical to:
> > 
> >   guestfsd: receive_file: reading length word
> >   guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 
> > 0x560e1b866690
> > 
> > until I hit [Ctrl+C]...
> > 
> > Does it make any sense?
> 
> No, it doesn't I have never seen the effects you see.
> 
> I have more ideas but I think you first somehow have to fix the guestfish
> problems that you are experiencing. What version of Debian and guestfish are
> you running?

for what it's worth I'm attaching the script I am using to re-generate my
autopkgtest qemu image. I uploaded a few packages today and just successfully
used that script so I can confirm it working.

Thanks!

cheers, josch#!/bin/sh

set -exu

#sudo vmdebootstrap --verbose --serial-console --distribution=sid --customize=/usr/share/autopkgtest/setup-commands/setup-testbed --user=test/test --size=20 --grub --mirror=http://127.0.0.1:3142/httpredir.debian.org/debian/ --configure-apt --apt-mirror=http://10.0.2.2:3142/httpredir.debian.org/debian/ --image=autopkgtest-sid.raw
#qemu-img convert -O qcow2 autopkgtest-sid.raw autopkgtest-sid.img
#qemu-system-x86_64 -m 500M -enable-kvm /srv/qemu/unstable-amd64-autopkgtest.qcow2


./mmdebstrap --mode=unshare --variant=important --include=linux-image-amd64 --customize-hook='chroot "$1" passwd --delete root' --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' --customize-hook='chroot "$1" passwd --delete user' --customize-hook='echo host > "$1/etc/hostname"' --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' --customize-hook=/usr/share/autopkgtest/setup-commands/setup-testbed unstable debian-unstable.tar
cat << END > extlinux.conf
default linux
timeout 0

label linux
kernel /vmlinuz
append initrd=/initrd.img root=/dev/vda1 rw console=ttyS0
END
guestfish -N debian-unstable.img=disk:2G -- part-disk /dev/sda mbr : part-set-bootable /dev/sda 1 true :  mkfs ext2 /dev/sda1 : mount /dev/sda1 / : tar-in debian-unstable.tar / :  extlinux / : copy-in extlinux.conf /
qemu-img convert -O qcow2 debian-unstable.img debian-unstable.qcow2
sudo mv debian-unstable.qcow2 /srv/qemu/unstable-amd64-autopkgtest.qcow2
rm debian-unstable.img debian-unstable.tar


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sat, 22 Feb 2020 08:14:55 +0100 Johannes Schauer wrote:

[...]
> Quoting Francesco Poli (2020-02-21 23:14:12)
[...]
> > Does it make any sense?
> 
> No, it doesn't I have never seen the effects you see.

As I have already said, I am apparently very unlucky with this effort
of mine!

> 
> I have more ideas but I think you first somehow have to fix the guestfish
> problems that you are experiencing. What version of Debian and guestfish are
> you running?

I am running Debian testing (bullseye) and I was using
libguestfs-tools/1:1.40.2-7 (currently in testing).
When you asked, I noticed that there is a version rebuilt from the same
source in unstable, and I upgraded to it.

I have just retested my script with libguestfs-tools/1:1.40.2-7+b1:

  $ apt policy libguestfs-tools 
  libguestfs-tools:
Installed: 1:1.40.2-7+b1
Candidate: 1:1.40.2-7+b1
Version table:
   *** 1:1.40.2-7+b1 500
  500 http://deb.debian.org/debian unstable/main amd64 Packages
  100 /var/lib/dpkg/status
   1:1.40.2-7 800
  800 http://deb.debian.org/debian testing/main amd64 Packages

but the result was the same.
At the end of the script guestfish fails with the same error and
qemu-img produces a really tiny .qcow2 image, which does not seem to
contain any bootable partition.

The mystery deepens, as soon as I try to issue the guestfish command
manually (after the script exited):

  $ guestfish --new debian-unstable_DEBUG.img=disk:"$SIZE" -- part-disk 
/dev/sda mbr : part-set-bootable /dev/sda 1 true : mkfs ext2 /dev/sda1 : mount 
/dev/sda1 / : tar-in debian-unstable.tar / : extlinux / : copy-in extlinux.conf 
/ : sync : umount / : shutdown

which exits successfully and produces a .img file which qemu-img is
able to convert into a reasonably sized .qcow2 file:

  $ ls --si -ltrF debian-unstable*
  -rw-r--r-- 1 $USER $GROUP 629M Feb 22 18:30 debian-unstable.tar
  -rw-rw 1 $USER $GROUP 2.2G Feb 22 18:30 debian-unstable.img
  -rw-r- 1 $USER $GROUP 197k Feb 22 18:30 debian-unstable.qcow2
  -rw-rw 1 $USER $GROUP 2.2G Feb 22 18:31 debian-unstable_DEBUG.img
  -rw-r- 1 $USER $GROUP 707M Feb 22 18:31 debian-unstable_DEBUG.qcow2

Nevertheless, debian-unstable_DEBUG.qcow2 does not boot with:

  $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait 
-drive  "file=./debian-unstable_DEBUG.qcow2,cache=unsafe,if=virtio,index=0"

It hangs exactly like the first image I obtained weeks ago (that is to
say, at the "Booting from Hard Disk..." message.

I cannot understand how (what really appears to be) the same guestfish
command behaves differently inside the script and outside (after the
end of the script)!
This looks like black magic to me...   :-|


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgptPo8t4kNIu.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-21 23:14:12)
> I modified my script (see the current version attached to this message) and
> tried again.
> 
>   $ env | grep SOURCE
>   SOURCE_DATE_EPOCH=1582320746
>   $ mmdebstrap-autopkgtest-qemu
>   [...]
>   I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.vt5NxWQb_6...
>   libguestfs: error: /usr/bin/supermin exited with error status 1.
>   To see full error messages you may need to enable debugging.
>   Do:
> export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
>   and run the command again.  For further information, read:
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
>   You can also run 'libguestfs-test-tool' and post the *complete* output
>   into a bug report or message to the libguestfs mailing list.

did you try running guestfish with these env variables to figure out what's
wrong?

> The qcow2 image is suspiciously small:
> 
>   $ ls --si -l debian-unstable.*
>   -rw-rw 1 $USER $GROUP 2.2G Feb 21 22:34 debian-unstable.img
>   -rw-r- 1 $USER $GROUP 197k Feb 21 22:34 debian-unstable.qcow2
>   -rw-r--r-- 1 $USER $GROUP 629M Feb 21 22:34 debian-unstable.tar
> 
> and actually fails to boot (but for different reasons, with respect to
> the previous attempt), see the attached screenshot.
> I tried both the qcow2 image and the raw image:
> 
>   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> unix:/tmp/ttyS0,server,nowait -drive  
> "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
>   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> unix:/tmp/ttyS0,server,nowait -drive 
> "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"
> 
> with identical results: after attempting all possible boot devices
> (including a network boot!), it bails out with "No bootable device"
> error message.
> 
> I also tried:
> 
>   $ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
>   $ SIZE="2GiB"
>   $ guestfish --new debian-unstable_DEBUG.img=disk:"$SIZE" -- \
> part-disk /dev/sda mbr : \
> part-set-bootable /dev/sda 1 true : \
> mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
> tar-in debian-unstable.tar / : \
> extlinux / : \
> copy-in extlinux.conf / : \
> sync : umount / : shutdown
> 
> which produced an unmanageable quantity of output strings, with an
> apparently endless stream line pairs identical to:
> 
>   guestfsd: receive_file: reading length word
>   guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 
> 0x560e1b866690
> 
> until I hit [Ctrl+C]...
> 
> Does it make any sense?

No, it doesn't I have never seen the effects you see.

I have more ideas but I think you first somehow have to fix the guestfish
problems that you are experiencing. What version of Debian and guestfish are
you running?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Thu, 20 Feb 2020 06:34:44 +0100 Johannes Schauer wrote:

> I have an idea.
[...]
> I added the following to the end of the guestfish invocation:
> 
> : sync : umount / : shutdown
> 
[...]
> Try it out and tell me if it helped!

I am apparently very unlucky.   :-(

I modified my script (see the current version attached to this message)
and tried again.

  $ env | grep SOURCE
  SOURCE_DATE_EPOCH=1582320746
  $ mmdebstrap-autopkgtest-qemu
  [...]
  I: removing tempdir ${HOME}/Downloads/TEST/mmdebstrap.vt5NxWQb_6...
  libguestfs: error: /usr/bin/supermin exited with error status 1.
  To see full error messages you may need to enable debugging.
  Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  and run the command again.  For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
  You can also run 'libguestfs-test-tool' and post the *complete* output
  into a bug report or message to the libguestfs mailing list.

The qcow2 image is suspiciously small:

  $ ls --si -l debian-unstable.*
  -rw-rw 1 $USER $GROUP 2.2G Feb 21 22:34 debian-unstable.img
  -rw-r- 1 $USER $GROUP 197k Feb 21 22:34 debian-unstable.qcow2
  -rw-r--r-- 1 $USER $GROUP 629M Feb 21 22:34 debian-unstable.tar

and actually fails to boot (but for different reasons, with respect to
the previous attempt), see the attached screenshot.
I tried both the qcow2 image and the raw image:

  $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait 
-drive  "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
  $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait 
-drive "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

with identical results: after attempting all possible boot devices
(including a network boot!), it bails out with "No bootable device"
error message.

I also tried:

  $ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
  $ SIZE="2GiB"
  $ guestfish --new debian-unstable_DEBUG.img=disk:"$SIZE" -- \
part-disk /dev/sda mbr : \
part-set-bootable /dev/sda 1 true : \
mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
tar-in debian-unstable.tar / : \
extlinux / : \
copy-in extlinux.conf / : \
sync : umount / : shutdown

which produced an unmanageable quantity of output strings, with an
apparently endless stream line pairs identical to:

  guestfsd: receive_file: reading length word
  guestfsd: receive_file: got chunk: cancel = 0x0, len = 8192, buf = 
0x560e1b866690

until I hit [Ctrl+C]...

Does it make any sense?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


mmdebstrap-autopkgtest-qemu
Description: Binary data


pgp2vRRWVBHCf.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-19 23:21:55)
> On Tue, 18 Feb 2020 00:03:06 +0100 Johannes Schauer wrote:
> > Can you show me how you converted the tarball mmdebstrap
> > produced into your debian-unstable.qcow2 image?
> 
> Sure, the command was:
> 
>   $ mmdebstrap-autopkgtest-qemu
> 
> which is a small script I am trying to develop, based on your suggested
> command.
> It is still experimental, needs to be improved in many aspects, and has
> not yet produced any working QEMU image (!).
> I wanted to send it to you as a contribution to mmdebstrap, once it was
> mature enough.
> I am sending it to you now, as a preview (not to be included in the
> official package, yet!). Any suggestion for improvement is more than
> welcome...
> 
> Please note that I am running it on an amd64 box. Hence:
> 
>   $ dpkg --print-architecture
>   amd64
> 
> 
> I tried the image again, after adding my regular user to the 'kvm'
> group (I hadn't done so, out of ignorance about QEMU/KVM), but the
> result was the same:
> 
>   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> unix:/tmp/ttyS0,server,nowait -drive  
> "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
> 
> or
> 
>   $ qemu-system-x86_64 -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive 
> "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
> 
> or
> 
>   $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> unix:/tmp/ttyS0,server,nowait -drive 
> "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"
> 
> always starts a virtual machine that hangs (seemingly) forever, at the
> "Booting from Hard Disk..." stage.
> 
> 
> I really cannot understand what's going on.

I have an idea. When I tried running guestfish on salsa CI it also would not
boot at all. The image that was created by guestfish just wasn't bootable for
some reason. After much tinkering, I added the following to the end of the
guestfish invocation:

: sync : umount / : shutdown

And that worked. Since there seem to be some machines where these additions to
guestfish are needed, I also now put these into all guestfish invocations and
examples in mmdebstrap but that version isn't released yet and you would have
had to look into the current git master.

Try it out and tell me if it helped!

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Tue, 18 Feb 2020 00:03:06 +0100 Johannes Schauer wrote:

[...]
> Can you show me how you converted the tarball mmdebstrap
> produced into your debian-unstable.qcow2 image?

Sure, the command was:

  $ mmdebstrap-autopkgtest-qemu

which is a small script I am trying to develop, based on your suggested
command.
It is still experimental, needs to be improved in many aspects, and has
not yet produced any working QEMU image (!).
I wanted to send it to you as a contribution to mmdebstrap, once it was
mature enough.
I am sending it to you now, as a preview (not to be included in the
official package, yet!). Any suggestion for improvement is more than
welcome...

Please note that I am running it on an amd64 box. Hence:

  $ dpkg --print-architecture
  amd64


I tried the image again, after adding my regular user to the 'kvm'
group (I hadn't done so, out of ignorance about QEMU/KVM), but the
result was the same:

  $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait 
-drive  "file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"

or

  $ qemu-system-x86_64 -m 512 -serial unix:/tmp/ttyS0,server,nowait -drive 
"file=./debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"

or

  $ qemu-system-x86_64 -enable-kvm -m 512 -serial unix:/tmp/ttyS0,server,nowait 
-drive "file=./debian-unstable.img,format=raw,cache=unsafe,if=virtio,index=0"

always starts a virtual machine that hangs (seemingly) forever, at the
"Booting from Hard Disk..." stage.


I really cannot understand what's going on.


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


mmdebstrap-autopkgtest-qemu
Description: Binary data


pgpQiQn35glig.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-17 23:10:57)
> On Sun, 02 Feb 2020 20:29:04 +0100 Johannes Schauer wrote:
> 
> > Quoting Johannes Schauer (2020-02-02 20:22:49)
> [...]
> > > 1. you can try the qcow2 image you built without autopkgtest by just 
> > > running it
> > >inside qemu like so:
> > > 
> > > $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> > > unix:/tmp/ttyS0,server,nowait -drive 
> > > "file=$HOME/Downloads/TEST/debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
> > 
> > One non-obvious thing is missing here. After running the above command you 
> > have
> > a new unix socket under /tmp/ttyS0 that you should be able to connect to 
> > using
> > a serial terminal emulator. This is how autopkgtest connects to qemu. So 
> > after
> > running above command, check that everything works by running something 
> > like:
> > 
> > $ minicom -D 'unix#/tmp/ttyS0'
> 
> Hello and sorry for the long delay (I had been busy in too many things...).
> 
> I tried the test you suggested and found out that the virtual machine
> fails to boot!
> The attached screenshot is what is shown in the graphical window...
> seemingly forever.
> 
> I haven't even tried to connect to /tmp/ttyS0, as I guess a virtual
> machine that fails to boot will never bring the serial terminal
> emulator up and running...
> 
> I am really ignorant about qemu: do you happen to have any idea on what
> went wrong?

here is a hunch:

Can you show me how you converted the tarball mmdebstrap produced into your
debian-unstable.qcow2 image?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 02 Feb 2020 20:29:04 +0100 Johannes Schauer wrote:

> Quoting Johannes Schauer (2020-02-02 20:22:49)
[...]
> > 1. you can try the qcow2 image you built without autopkgtest by just 
> > running it
> >inside qemu like so:
> > 
> > $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> > unix:/tmp/ttyS0,server,nowait -drive 
> > "file=$HOME/Downloads/TEST/debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"
> 
> One non-obvious thing is missing here. After running the above command you 
> have
> a new unix socket under /tmp/ttyS0 that you should be able to connect to using
> a serial terminal emulator. This is how autopkgtest connects to qemu. So after
> running above command, check that everything works by running something like:
> 
> $ minicom -D 'unix#/tmp/ttyS0'

Hello and sorry for the long delay (I had been busy in too many things...).

I tried the test you suggested and found out that the virtual machine
fails to boot!
The attached screenshot is what is shown in the graphical window...
seemingly forever.

I haven't even tried to connect to /tmp/ttyS0, as I guess a virtual
machine that fails to boot will never bring the serial terminal
emulator up and running...

I am really ignorant about qemu: do you happen to have any idea on what
went wrong?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpZIGyLBOK0C.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Quoting Johannes Schauer (2020-02-02 20:22:49)
> Hi,
> 
> Quoting Francesco Poli (2020-02-02 16:58:05)
> > > >   • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
> > > >   some different hex values (I am not sure why, but it's compiled Python
> > > >   code, maybe it includes a compilation timestamp or something?!?)
> > > This is a known bug that I have yet to report to the Python maintainers.
> > Ah, interesting.  I encourage you to report this bug, as it might help the
> > Reproducible Builds effort...
> 
> I reported a very similar bug over a year ago:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917407
> 
> The bug then magically fixed itself when python 3.7.3 upgraded to 3.7.4 but
> nobody really ever investigated what happened.
> 
> > > > I am under the impression that the two .tar files are to be considered
> > > > equivalent.  Do you agree?
> > > Yes. :)
> > OK, in the meanwhile I got around to check whether the .qcow2 image is
> > actually working as autopkgtest testbed.
> > 
> > Unfortunately, no, it's not working!  :-(
> > 
> > 
> >   $ autopkgtest --output-dir ./${PKG}_${VERS}_autopkgtest.out \
> >--summary ./${PKG}_${VERS}_autopkgtest.summary \
> >   -B ./${PKG}_${VERS}_amd64.changes \
> > -- qemu ~/Downloads/TEST/debian-unstable.qcow2
> >   autopkgtest [16:23:45]: version 5.11
> >   autopkgtest [16:23:45]: host ${HOST}; command line: ${CMDLINE}
> >   qemu-system-x86_64: terminating on signal 15 from pid 8488 
> > (/usr/bin/python3)
> >   : failure: timed out waiting for "login prompt on ttyS0"
> >   autopkgtest [16:24:45]: ERROR: testbed failure: cannot send to testbed: 
> > [Errno 32] Broken pipe
> > 
> > Could you help me to investigate the issue?
> > Is the command line correct?  Where did I go wrong?
> 
> I can imagine two ways forward.
> 
> 1. you can try the qcow2 image you built without autopkgtest by just running 
> it
>inside qemu like so:
> 
> $ qemu-system-x86_64 -enable-kvm -m 512 -serial 
> unix:/tmp/ttyS0,server,nowait -drive 
> "file=$HOME/Downloads/TEST/debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"

One non-obvious thing is missing here. After running the above command you have
a new unix socket under /tmp/ttyS0 that you should be able to connect to using
a serial terminal emulator. This is how autopkgtest connects to qemu. So after
running above command, check that everything works by running something like:

$ minicom -D 'unix#/tmp/ttyS0'

> 2. you could upload your qcow2 image somewhere together with the right
>SOURCE_DATE_EPOCH value that you used. Then I could try to reproduce that
>image myself and diff it against what I have because what I have here is
>working just fine. :)



signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-02-02 16:58:05)
> > >   • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
> > >   some different hex values (I am not sure why, but it's compiled Python
> > >   code, maybe it includes a compilation timestamp or something?!?)
> > This is a known bug that I have yet to report to the Python maintainers.
> Ah, interesting.  I encourage you to report this bug, as it might help the
> Reproducible Builds effort...

I reported a very similar bug over a year ago:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917407

The bug then magically fixed itself when python 3.7.3 upgraded to 3.7.4 but
nobody really ever investigated what happened.

> > > I am under the impression that the two .tar files are to be considered
> > > equivalent.  Do you agree?
> > Yes. :)
> OK, in the meanwhile I got around to check whether the .qcow2 image is
> actually working as autopkgtest testbed.
> 
> Unfortunately, no, it's not working!  :-(
> 
> 
>   $ autopkgtest --output-dir ./${PKG}_${VERS}_autopkgtest.out \
>--summary ./${PKG}_${VERS}_autopkgtest.summary \
>   -B ./${PKG}_${VERS}_amd64.changes \
> -- qemu ~/Downloads/TEST/debian-unstable.qcow2
>   autopkgtest [16:23:45]: version 5.11
>   autopkgtest [16:23:45]: host ${HOST}; command line: ${CMDLINE}
>   qemu-system-x86_64: terminating on signal 15 from pid 8488 
> (/usr/bin/python3)
>   : failure: timed out waiting for "login prompt on ttyS0"
>   autopkgtest [16:24:45]: ERROR: testbed failure: cannot send to testbed: 
> [Errno 32] Broken pipe
> 
> Could you help me to investigate the issue?
> Is the command line correct?  Where did I go wrong?

I can imagine two ways forward.

1. you can try the qcow2 image you built without autopkgtest by just running it
   inside qemu like so:

$ qemu-system-x86_64 -enable-kvm -m 512 -serial 
unix:/tmp/ttyS0,server,nowait -drive 
"file=$HOME/Downloads/TEST/debian-unstable.qcow2,cache=unsafe,if=virtio,index=0"

2. you could upload your qcow2 image somewhere together with the right
   SOURCE_DATE_EPOCH value that you used. Then I could try to reproduce that
   image myself and diff it against what I have because what I have here is
   working just fine. :)

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 02 Feb 2020 00:23:59 +0100 Johannes Schauer wrote:

> Quoting Francesco Poli (2020-02-01 16:52:47)
> > The only differences shown in the resulting report_diffoscope.html file seem
> > to be:
> > 
> >   • the generated files in the content
> > of ./boot/initrd.img-5.4.0-3-amd64 have differing creation
> > timestamps (but this is obvious, since the two initrd were not
> > created exactly at the same time!)
> > 
> >   • ./var/lib/initramfs-tools/5.4.0-3-amd64 files differ (but they seem
> > to include some checksum of the initrd, hence the difference should
> > be consequence of the first point)
> 
> this should go away when you set SOURCE_DATE_EPOCH to something like $(date
> +%s) -- why didn't you set it in your tests?

It's plain simple: because I am an idiot!  ;-)
I just failed to think about it...

After setting the same SOURCE_DATE_EPOCH, the only remaining
differences were the machine-id and the hashlib.cpython-37.pyc file!

> 
> >   • ./etc/machine-id and ./var/lib/dbus/machine-id files differ (but I
> > think this should not be surprising...)
> 
> In the next mmdebstrap release /etc/machine-id will be set to an empty file.

OK.

> 
> >   • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
> > some different hex values (I am not sure why, but it's compiled
> > Python code, maybe it includes a compilation timestamp or
> > something?!?)
> 
> This is a known bug that I have yet to report to the Python maintainers.

Ah, interesting.
I encourage you to report this bug, as it might help the Reproducible
Builds effort...

> 
> > I am under the impression that the two .tar files are to be considered
> > equivalent.
> > Do you agree?
> 
> Yes. :)

OK, in the meanwhile I got around to check whether the .qcow2 image is
actually working as autopkgtest testbed.

Unfortunately, no, it's not working!  :-(


  $ autopkgtest --output-dir ./${PKG}_${VERS}_autopkgtest.out \
   --summary ./${PKG}_${VERS}_autopkgtest.summary \
  -B ./${PKG}_${VERS}_amd64.changes \
-- qemu ~/Downloads/TEST/debian-unstable.qcow2
  autopkgtest [16:23:45]: version 5.11
  autopkgtest [16:23:45]: host ${HOST}; command line: ${CMDLINE}
  qemu-system-x86_64: terminating on signal 15 from pid 8488 (/usr/bin/python3)
  : failure: timed out waiting for "login prompt on ttyS0"
  autopkgtest [16:24:45]: ERROR: testbed failure: cannot send to testbed: 
[Errno 32] Broken pipe

Could you help me to investigate the issue?
Is the command line correct?
Where did I go wrong?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgp86n5EiaIx2.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Quoting Francesco Poli (2020-02-01 16:52:47)
> The only differences shown in the resulting report_diffoscope.html file seem
> to be:
> 
>   • the generated files in the content
> of ./boot/initrd.img-5.4.0-3-amd64 have differing creation
> timestamps (but this is obvious, since the two initrd were not
> created exactly at the same time!)
> 
>   • ./var/lib/initramfs-tools/5.4.0-3-amd64 files differ (but they seem
> to include some checksum of the initrd, hence the difference should
> be consequence of the first point)

this should go away when you set SOURCE_DATE_EPOCH to something like $(date
+%s) -- why didn't you set it in your tests?

>   • ./etc/machine-id and ./var/lib/dbus/machine-id files differ (but I
> think this should not be surprising...)

In the next mmdebstrap release /etc/machine-id will be set to an empty file.

>   • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
> some different hex values (I am not sure why, but it's compiled
> Python code, maybe it includes a compilation timestamp or
> something?!?)

This is a known bug that I have yet to report to the Python maintainers.

> I am under the impression that the two .tar files are to be considered
> equivalent.
> Do you agree?

Yes. :)


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Fri, 31 Jan 2020 06:28:07 +0100 Johannes Schauer wrote:

[...]
> Quoting Francesco Poli (2020-01-30 23:41:11)
[...]
> > Wait, does this change the result?
[...]
> in your case, TMPDIR was a relative path.
[...]
> Your expectation was, that when you set TMPDIR, the only one who would consume
> that environment variable would be mmdebstrap itself. I think that thought is
> not unreasonable. That's why I think it's important that mmdebstrap unsets 
> that
> variable before it executes processes inside the chroot.

All your reasoning makes sense to me.
I agree that TMPDIR should be unset by mmdebstrap before running hooks.

I tried to check that the resulting .tar file is not affected by this
TMPDIR unsetting.
I did the following:

  $ cat ABSTMPDIR/test.sh 
  #!/bin/sh
  
  SETUP_TESTBED="/usr/share/autopkgtest/setup-commands/setup-testbed"
  
  LIMAVAR="amd64"
  SUITE="sid"  
  SIZE="2GiB"
  
  TMPDIR=$(pwd)
  export TMPDIR
  mmdebstrap --variant=important --include=linux-image-"$LIMAVAR" \
--customize-hook='chroot "$1" passwd --delete root' \
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home 
user' \
--customize-hook='chroot "$1" passwd --delete user' \
--customize-hook='echo host > "$1/etc/hostname"' \
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
--customize-hook="$SETUP_TESTBED" \
"$SUITE" debian-unstable.tar
  $ cat NOTMPDIR/test.sh 
  #!/bin/sh
  
  SETUP_TESTBED="/usr/share/autopkgtest/setup-commands/setup-testbed"
  
  LIMAVAR="amd64"
  SUITE="sid"  
  SIZE="2GiB"
  
  TMPDIR='.'
  export TMPDIR
  mmdebstrap --variant=important --include=linux-image-"$LIMAVAR" \
--customize-hook='chroot "$1" passwd --delete root' \
--customize-hook='chroot "$1" useradd --home-dir /home/user --create-home 
user' \
--customize-hook='chroot "$1" passwd --delete user' \
--customize-hook='echo host > "$1/etc/hostname"' \
--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
--customize-hook='env --unset=TMPDIR 
/usr/share/autopkgtest/setup-commands/setup-testbed "$1"' \
"$SUITE" debian-unstable.tar

After running the two test.sh scripts within their respective
directories, I used diffoscope to highlight the differences between the
two resulting .tar files:

  $ TMPDIR=/run/shm diffoscope \
 --max-diff-block-lines 15000 \
 --max-page-diff-block-lines 1500 \
 --html report_diffoscope.html \
 ABSTMPDIR/debian-unstable.tar \
  NOTMPDIR/debian-unstable.tar

The only differences shown in the resulting report_diffoscope.html file
seem to be:

  • the generated files in the content
of ./boot/initrd.img-5.4.0-3-amd64 have differing creation
timestamps (but this is obvious, since the two initrd were not
created exactly at the same time!)

  • ./var/lib/initramfs-tools/5.4.0-3-amd64 files differ (but they seem
to include some checksum of the initrd, hence the difference should
be consequence of the first point)

  • ./etc/machine-id and ./var/lib/dbus/machine-id files differ (but I
think this should not be surprising...)

  • ./usr/lib/python3.7/__pycache__/hashlib.cpython-37.pyc files have
some different hex values (I am not sure why, but it's compiled
Python code, maybe it includes a compilation timestamp or
something?!?)

I am under the impression that the two .tar files are to be considered
equivalent.
Do you agree?



P.S.: I still have to find the time to check the .qcow2 file I obtained
on last Wednesday...   :-(

-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpbwUbL69RDJ.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-30 23:41:11)
> On Thu, 30 Jan 2020 01:26:09 +0100 Johannes Schauer wrote:
> > the problem is fixed when TMPDIR gets unset for hook.
> Wait, does this change the result?
> 
> I mean: some commands executed (directly or indirectly) by
> setup-testbed perform operations on the directory which mmdebstrap set
> up as TMPDIR.
> If this is the case (as I think it is, at least for the udev hook
> called by update-initramfs), I wonder whether unsetting TMPDIR causes
> setup-testbed to perform some of its operations on the wrong directory...
> 
> Am I completely off-track?

in your case, TMPDIR was a relative path. It doesn't even matter whether the
script reading TMPDIR runs inside or outside the chroot. That script might
change its working directory to anywhere during its operation. If it then tries
to access a file that expects to be in $TMPDIR it will obviously not find it
because it changed working directory.

Another scenario: imagine you set TMPDIR to an absolute path outside the
chroot. Naturally, that path is not available inside the chroot and programs
inside the chroot might fail when they try to store data into the non-existing
TMPDIR.

Your expectation was, that when you set TMPDIR, the only one who would consume
that environment variable would be mmdebstrap itself. I think that thought is
not unreasonable. That's why I think it's important that mmdebstrap unsets that
variable before it executes processes inside the chroot.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Thu, 30 Jan 2020 01:26:09 +0100 Johannes Schauer wrote:

[...]
> the problem is fixed when TMPDIR gets unset for hook.

Wait, does this change the result?

I mean: some commands executed (directly or indirectly) by
setup-testbed perform operations on the directory which mmdebstrap set
up as TMPDIR.
If this is the case (as I think it is, at least for the udev hook
called by update-initramfs), I wonder whether unsetting TMPDIR causes
setup-testbed to perform some of its operations on the wrong directory...

Am I completely off-track?

-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgppmHhNb_ly0.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Quoting Johannes Schauer (2020-01-30 00:19:44)
> Quoting Francesco Poli (2020-01-29 23:34:21)
> > > I would love to help more but I already tried out the failing command on
> > > three different systems and I'm unable to reproduce it.
> > Which TMPDIR were you using?
> 
> apologies, I missed that you set TMPDIR. When I set it, then I can reproduce
> your problem in an even more minimal fashion:
> 
> TMPDIR='./' mmdebstrap --include=linux-image-amd64 --customize-hook='chroot 
> "$1" update-initramfs -u' sid debian-unstable.tar
> 
> I'll come back to you once I have fixed the issue.

the problem is fixed when TMPDIR gets unset for hook. I'm now unsure how to
best fix the problem. I could generally unset TMPDIR for hooks or just update
the instructions like this:

   $ mmdebstrap --variant=important --include=linux-image-amd64 \
   --customize-hook='chroot "$1" passwd --delete root' \
   --customize-hook='chroot "$1" useradd --home-dir /home/user 
--create-home user' \
   --customize-hook='chroot "$1" passwd --delete user' \
   --customize-hook='echo host > "$1/etc/hostname"' \
   --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
   --customize-hook='env --unset=TMPDIR 
/usr/share/autopkgtest/setup-commands/setup-testbed "$1"' \
   unstable debian-unstable.tar

hmm


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-29 23:34:21)
> > I would love to help more but I already tried out the failing command on
> > three different systems and I'm unable to reproduce it.
> Which TMPDIR were you using?

apologies, I missed that you set TMPDIR. When I set it, then I can reproduce
your problem in an even more minimal fashion:

TMPDIR='./' mmdebstrap --include=linux-image-amd64 --customize-hook='chroot 
"$1" update-initramfs -u' sid debian-unstable.tar

I'll come back to you once I have fixed the issue.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Wed, 29 Jan 2020 00:02:51 +0100 Johannes Schauer wrote:

[...]
> Quoting Francesco Poli (2020-01-28 23:41:38)
[...]
> > The lines that seem to fail are /usr/share/initramfs-tools/hooks/udev:
> > 25 and the following ones... But why?
> 
> yes, that's where it seems to fail. To further investigate you can try the
> following:
> 
> Copy /usr/share/autopkgtest/setup-commands/setup-testbed to a location that 
> you
> control so that you can edit it. Remove all the parts that are not necessary 
> to
> reproduce the problem. I suspect just keeping the line 'chroot "$root"
> update-initramfs -u' is not sufficient and something else has to happen before
> so that you can see the error? If that line alone is enough, then you should
> also be able to reproduce the problem by running mmdebstrap with:
> 
> --customize-hook='chroot "$1" update-initramfs -u'

I can get the error with just:

  $ cd Downloads/
  $ TMPDIR='./'
  $ export TMPDIR
  $ mmdebstrap --variant=important --include=linux-image-amd64 \
  >   --customize-hook='chroot "$1" passwd --delete root' \
  >   --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home 
user' \
  >   --customize-hook='chroot "$1" passwd --delete user' \
  >   --customize-hook='echo host > "$1/etc/hostname"' \
  >   --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
  >   --customize-hook='chroot "$1" update-initramfs -u' \
  > "sid" debian-unstable.tar

> 
> Once you are that far, you either already have found the problem or the next
> thing you can try is to debug /usr/share/initramfs-tools/hooks/udev. To do
> that, just insert a "set -x" somewhere at the top. For example by adding the
> following before you call the failing hook:
> 
> --customize-hook='awk "NR==2{print 'set -x'}1" 
> "$1/usr/share/initramfs-tools/hooks/udev"'

I think I am getting nearer to understand the issue.


  $ cd Downloads/
  $ TMPDIR='.'
  $ export TMPDIR
  $ mmdebstrap --variant=important --include=linux-image-amd64 \
  >   --customize-hook='chroot "$1" passwd --delete root' \
  >   --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home 
user' \
  >   --customize-hook='chroot "$1" passwd --delete user' \
  >   --customize-hook='echo host > "$1/etc/hostname"' \
  >   --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
  >   --customize-hook='sed -i "1a set -x" 
"$1/usr/share/initramfs-tools/hooks/udev"' \
  >   --customize-hook='chroot "$1" update-initramfs -u' \
  > "sid" debian-unstable.tar
  [...]
  + PREREQS=
  + prereqs
  + echo
  + exit 0
  + PREREQS=
  + . /usr/share/initramfs-tools/hook-functions
  + mkdir -p ./mkinitramfs_1itDzt/lib/systemd
  [...]
  + mkdir -p ./mkinitramfs_1itDzt/etc/udev
  + cp -p /etc/udev/udev.conf ./mkinitramfs_1itDzt/etc/udev/
  + mkdir -p ./mkinitramfs_1itDzt/lib/systemd/network/
  + find /lib/systemd/network -name *.link -execdir cp -pt 
./mkinitramfs_1itDzt/lib/systemd/network/ {} +
  cp: failed to access './mkinitramfs_1itDzt/lib/systemd/network/': No such 
file or directory
  E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
  update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
  E: run_chroot failed: E: command failed: chroot "$1" update-initramfs -u
  I: main() received signal PIPE: waiting for setup...
  W: listening on child socket failed: E: received eof on socket
  
  I: removing tempdir ${HOME}/Downloads/mmdebstrap._p8KPoNDcS...

The issue seems to be that the -execdir option passed to find causes
the command "cp" to be run from the subdirectory containing the matched
file "/lib/systemd/network", but the destination directory is a
relative path "./mkinitramfs_1itDzt/lib/systemd/network/", which has
been previously created, but not under "/lib/systemd/network" !

If I set TMPDIR=$(pwd) in stead of TMPDIR='.' or TMPDIR='./'
I get to the end of the process without major errors.
Except for some warnings such as:

  W: Unable to read ${HOME}/Downloads/48yluzcM8D - RealFileExists (2: No such 
file or directory)

Should I worry about them?!?

At least I obtain a non-empty debian-unstable.tar and I manage to
create debian-unstable.img with guestfish and then convert it to
debian-unstable.qcow2 with qemu-img...

Now I am running out of time, but I hope I will soon get to check the
resulting .qcow2 file (I have to test whether it can successfully be
used as an autopkgtest testbed).
I'll let you know, thanks for all you invaluable help so far!   :-)

> 
> I would love to help more but I already tried out the failing command on three
> different systems and I'm unable to reproduce it.

Which TMPDIR were you using?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpSFx32kRtKR.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-28 23:41:38)
>   + chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou update-initramfs -u
>   update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
>   W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module 
> r8169
>   [...]
>   W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module 
> r8169
>   cp: failed to access './/mkinitramfs_v8SvPb/lib/systemd/network/': No such 
> file or directory
>   E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
>   update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
>   E: run_chroot failed: E: command failed: sh -x 
> /usr/share/autopkgtest/setup-commands/setup-testbed "$1"
>   I: main() received signal PIPE: waiting for setup...
>   W: listening on child socket failed: E: received eof on socket
>   
>   I: removing tempdir ${HOME}/Downloads/mmdebstrap.pxni4vAjou...
> 
> 
> As you can see, a "cp" fails to access file 
> './/mkinitramfs_v8SvPb/lib/systemd/network/',
> and /usr/share/initramfs-tools/hooks/udev fails with exit status 1.
> But, honestly, I cannot understand why.
> 
> The lines that seem to fail are /usr/share/initramfs-tools/hooks/udev:
> 25 and the following ones... But why?

yes, that's where it seems to fail. To further investigate you can try the
following:

Copy /usr/share/autopkgtest/setup-commands/setup-testbed to a location that you
control so that you can edit it. Remove all the parts that are not necessary to
reproduce the problem. I suspect just keeping the line 'chroot "$root"
update-initramfs -u' is not sufficient and something else has to happen before
so that you can see the error? If that line alone is enough, then you should
also be able to reproduce the problem by running mmdebstrap with:

--customize-hook='chroot "$1" update-initramfs -u'

Once you are that far, you either already have found the problem or the next
thing you can try is to debug /usr/share/initramfs-tools/hooks/udev. To do
that, just insert a "set -x" somewhere at the top. For example by adding the
following before you call the failing hook:

--customize-hook='awk "NR==2{print 'set -x'}1" 
"$1/usr/share/initramfs-tools/hooks/udev"'

I would love to help more but I already tried out the failing command on three
different systems and I'm unable to reproduce it.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Mon, 27 Jan 2020 23:44:55 +0100 Johannes Schauer wrote:

[...]
> this looks as if the error comes from
> /usr/share/autopkgtest/setup-commands/setup-testbed. To figure out what goes
> wrong, maybe try running setup-testbed with sh -x like so:
> 
> --customize-hook='sh -x /usr/share/autopkgtest/setup-commands/setup-testbed 
> "$1"'
> 
> Your command works fine on my system, so this must be something specific to
> your setup.
> 
> Thanks!

Thanks to you for your super-prompt reply!   :-)

  $ cd Downloads/
  $ TMPDIR='./'
  $ export TMPDIR
  $ mmdebstrap --variant=important --include=linux-image-amd64 \
  >   --customize-hook='chroot "$1" passwd --delete root' \
  >   --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home 
user' \
  >   --customize-hook='chroot "$1" passwd --delete user' \
  >   --customize-hook='echo host > "$1/etc/hostname"' \
  >   --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
  >   --customize-hook='sh -x 
/usr/share/autopkgtest/setup-commands/setup-testbed "$1"' \
  > "sid" debian-unstable.tar
  I: automatically chosen mode: fakechroot
  [...]
  I: running --customize-hook in shell: sh -c 'sh -x 
/usr/share/autopkgtest/setup-commands/setup-testbed "$1"' exec 
${HOME}/Downloads/mmdebstrap.pxni4vAjou
  + set -eu
  + umask 0022
  + export DEBIAN_FRONTEND=noninteractive
  + [ ${HOME}/Downloads/mmdebstrap.pxni4vAjou = --help ]
  + root=${HOME}/Downloads/mmdebstrap.pxni4vAjou
  + [ ${HOME}/Downloads/mmdebstrap.pxni4vAjou != / ]
  + cat
  + chmod 755 ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/init.d/autopkgtest
  + chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou update-rc.d autopkgtest 
defaults
  + [ -d ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/system ]
  + cat
  + mkdir -p 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/system/multi-user.target.wants
  + ln -sf ../autopkgtest.service 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/system/multi-user.target.wants/autopkgtest.service
  + [ -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/init/tty2.conf -a ! -e 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/init/ttyS0.conf ]
  + chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou dpkg --print-architecture
  + ARCH=amd64
  + [ ! -e 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/default/grub.d/90-autopkgtest.cfg ]
  + chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou which update-grub
  + [ -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/os-release ]
  + . ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/os-release
  + PRETTY_NAME=Debian GNU/Linux bullseye/sid
  + NAME=Debian GNU/Linux
  + ID=debian
  + HOME_URL=https://www.debian.org/
  + SUPPORT_URL=https://www.debian.org/support
  + BUG_REPORT_URL=https://bugs.debian.org/
  + echo debian
  + DISTRO_ID=debian
  + [ -z  ]
  + awk /^deb .*debian/ { sub(/\[.*\]/, "", $0); print $2; exit } 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/apt/sources.list
  + MIRROR=http://deb.debian.org/debian
  + [ -z  ]
  + awk /^deb .*debian/ { sub(/\[.*\]/, "", $0); print $3; exit } 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/apt/sources.list
  + RELEASE=sid
  + [ -n  ]
  + [ -n  ]
  + [ -n  ]
  + echo /usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set 
up Debian/Ubuntu apt sources automatically
  /usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set up 
Debian/Ubuntu apt sources automatically
  + [ -z sid ]
  + [ -z http://deb.debian.org/debian ]
  + [ http://deb.debian.org/debian != http://deb.debian.org/debian ]
  + echo /usr/share/autopkgtest/setup-commands/setup-testbed: Distribution 
assumed to resemble Debian
  /usr/share/autopkgtest/setup-commands/setup-testbed: Distribution assumed to 
resemble Debian
  + cat
  + [ -e ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/cloud/cloud.cfg ]
  + [ -z  ]
  + ls ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/systemd/network/*.network
  + ls ${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/netplan/*.yaml
  + grep -q source.*interfaces.d 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/network/interfaces
  + IFACE=
  + [ ${HOME}/Downloads/mmdebstrap.pxni4vAjou = / ]
  + IFACE=eth0
  + [ -e 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/udev/rules.d/80-net-setup-link.rules
 ]
  + ln -s /dev/null 
${HOME}/Downloads/mmdebstrap.pxni4vAjou/etc/udev/rules.d/80-net-setup-link.rules
  + chroot ${HOME}/Downloads/mmdebstrap.pxni4vAjou update-initramfs -u
  update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
  W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module 
r8169
  [...]
  W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module 
r8169
  cp: failed to access './/mkinitramfs_v8SvPb/lib/systemd/network/': No such 
file or directory
  E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
  update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
  E: run_chroot failed: E: command failed: sh -x 
/usr/share/autopkgtest/setup-commands/setup-testbed "$1"
  I: main() received signal PIPE: waiting 

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2020-01-27 23:24:22)
>   I: running --customize-hook directly: 
> /usr/share/autopkgtest/setup-commands/setup-testbed 
> ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
>   /usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set up 
> Debian/Ubuntu apt sources automatically
>   /usr/share/autopkgtest/setup-commands/setup-testbed: Distribution assumed 
> to resemble Debian
>   update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
>   W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module 
> r8169
>   [...]
>   W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module 
> r8169
>   cp: failed to access './/mkinitramfs_SqDsE7/lib/systemd/network/': No such 
> file or directory
>   E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
>   update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
>   E: run_chroot failed: E: command failed: 
> /usr/share/autopkgtest/setup-commands/setup-testbed
>   I: main() received signal PIPE: waiting for setup...
>   W: listening on child socket failed: E: received eof on socket
>   
>   I: removing tempdir ${HOME}/Downloads/mmdebstrap.2GP5UNhTac...
>   $ ls -s
>   total 0
>   0 debian-unstable.tar
> 
> As you can see, I still obtain an empty file.
> 
> Did I misunderstand the situation?
> I thought it could work correctly, now...

this looks as if the error comes from
/usr/share/autopkgtest/setup-commands/setup-testbed. To figure out what goes
wrong, maybe try running setup-testbed with sh -x like so:

--customize-hook='sh -x /usr/share/autopkgtest/setup-commands/setup-testbed 
"$1"'

Your command works fine on my system, so this must be something specific to
your setup.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 17 Nov 2019 19:53:13 +0100 Johannes Schauer wrote:

> Quoting Johannes Schauer (2019-11-17 18:20:17)
> > > Maybe the recipe you suggested should be improved to address this issue
> > > with the selection of the mode...
> > 
> > Before doing that I'll see if this could be fixed by changing something in
> > either fakechroot or in update-initramfs.
> 
> See #944929

Hello Johannes!

After seeing bug #944929 closed as fixed in both unstable and testing,
I thought I could retry with mmdebstrap.
But I still get an error:

  $ cd Downloads
  $ TMPDIR='./'
  $ export TMPDIR
  $ mmdebstrap --variant=important --include=linux-image-amd64 \
  >   --customize-hook='chroot "$1" passwd --delete root' \
  >   --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home 
user' \
  >   --customize-hook='chroot "$1" passwd --delete user' \
  >   --customize-hook='echo host > "$1/etc/hostname"' \
  >   --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
  >   --customize-hook="/usr/share/autopkgtest/setup-commands/setup-testbed" \
  >   "sid" debian-unstable.tar
  I: automatically chosen mode: fakechroot
  I: chroot architecture amd64 is equal to the host's architecture
  I: using ${HOME}/Downloads/mmdebstrap.2GP5UNhTac as tempdir
  I: running apt-get update...
  done
  I: downloading packages with apt...
  done
  I: extracting archives...
  done
  I: installing packages...
  done
  I: downloading apt...
  done
  I: installing apt...
  done
  I: installing remaining packages inside the chroot...
  done
  I: running --customize-hook in shell: sh -c 'chroot "$1" passwd --delete 
root' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
  passwd: password expiry information changed.
  I: running --customize-hook in shell: sh -c 'chroot "$1" useradd --home-dir 
/home/user --create-home user' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
  I: running --customize-hook in shell: sh -c 'chroot "$1" passwd --delete 
user' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
  passwd: password expiry information changed.
  I: running --customize-hook in shell: sh -c 'echo host > "$1/etc/hostname"' 
exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
  I: running --customize-hook in shell: sh -c 'echo "127.0.0.1 localhost host" 
> "$1/etc/hosts"' exec ${HOME}/Downloads/mmdebstrap.2GP5UNhTac
  I: running --customize-hook directly: 
/usr/share/autopkgtest/setup-commands/setup-testbed 
${HOME}/Downloads/mmdebstrap.2GP5UNhTac
  /usr/share/autopkgtest/setup-commands/setup-testbed: Attempting to set up 
Debian/Ubuntu apt sources automatically
  /usr/share/autopkgtest/setup-commands/setup-testbed: Distribution assumed to 
resemble Debian
  update-initramfs: Generating /boot/initrd.img-5.4.0-3-amd64
  W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module 
r8169
  [...]
  W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module 
r8169
  cp: failed to access './/mkinitramfs_SqDsE7/lib/systemd/network/': No such 
file or directory
  E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
  update-initramfs: failed for /boot/initrd.img-5.4.0-3-amd64 with 1.
  E: run_chroot failed: E: command failed: 
/usr/share/autopkgtest/setup-commands/setup-testbed
  I: main() received signal PIPE: waiting for setup...
  W: listening on child socket failed: E: received eof on socket
  
  I: removing tempdir ${HOME}/Downloads/mmdebstrap.2GP5UNhTac...
  $ ls -s
  total 0
  0 debian-unstable.tar

As you can see, I still obtain an empty file.

Did I misunderstand the situation?
I thought it could work correctly, now...


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpxrtBFMLY7R.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Quoting Johannes Schauer (2019-11-17 18:20:17)
> > Maybe the recipe you suggested should be improved to address this issue
> > with the selection of the mode...
> 
> Before doing that I'll see if this could be fixed by changing something in
> either fakechroot or in update-initramfs.

See #944929


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2019-11-17 16:56:48)
> > Quoting Francesco Poli (2019-11-17 16:06:05)
> > > I am giving it a try, but I am getting an error I am not quite sure to
> > > understand:
> > >
> > > [...]
> > > 
> > >   I: automatically chosen mode: fakechroot
> > there is your problem. If you are not executing mmdebstrap with superuser
> > privileges and if you you did not manually specify which --mode to use,
> > then it will look for some way to create the tarball that works on your
> > system. For you it picked fakechroot.
> [...]
> 
> Well, I was trying the command you have previously suggested...
> 
> I understand that automatic mode depends on which packages I have
> installed on my system and on my sysctl configuration.

Yes, correct.

> The fact is: I was running as a regular user.
> I have:
> 
>   $ /sbin/sysctl kernel.unprivileged_userns_clone
>   kernel.unprivileged_userns_clone = 0
> 
> And I have fakechroot on my system, just because mmdebstrap recommends
> it:
> 
>   $ aptitude why fakechroot
>   i   mmdebstrap Recommends fakechroot
> 
> Hence, everything conspired to make mmdebstrap choose that mode!

Also correct.

> Maybe the recipe you suggested should be improved to address this issue with
> the selection of the mode...

Before doing that I'll see if this could be fixed by changing something in
either fakechroot or in update-initramfs.

> > That mode works by using an LD_PRELOAD library to fake root permissions for
> > all executed programs. Sometimes this fails badly and the postinst script
> > of update-initramfs is a known maintainer script that doesn't work under
> > fakechroot mode. I briefly looked into the problem a few days ago but
> > couldn't immediately see what was wrong. So this is not a bug in
> > mmdebstrap. It is maybe a bug in fakechroot or maybe something could be
> > done different by the postinst script of update-initramfs. I don't know
> > whether either maintainer considers what you see a bug. You can try and
> > file one and see if you get this fixed somehow.
> 
> I can try and report bugs, although I don't fully understand what's
> going on, and hence I will need your help to explain. Are you willing
> to join this bug reporting effort?

I'll take care of it. Before filing a bug I'll first see if I can maybe find
the root cause of it.

> Anyway, I am not sure I understand how *you* create an autopkgtest QEMU/KVM
> testbed with mmdebstrap. I thought you were successfully using the recipe you
> suggested.  Is it not the case? How do *you* do that?

I see two possibilities:

 1. back then when I wrote the docs a year ago, the bug in update-initramfs or
fakechroot might not've existed yet

 2. maybe I had "kernel.unprivileged_userns_clone = 1" set and thus unshare
mode was selected during my testing

> > Another way to solve the problem is by choosing a different mode. If you 
> > don't
> > mind using superuser privileges, you could run mmdebstrap as root.
> I really mind!   ;-)

Great! :D

> Creating QEMU/KVM images without superuser privileges was the whole
> point of giving mmdebstrap a try.
> Otherwise I would just be using autopkgtest-build-qemu, as I said in the
> original bug report...

You could also file a bug report against vmdb2 (which is used by
autopkgtest-build-qemu) and ask its maintainer whether they would consider
adding root-less operation to it. As shown by mmdebstrap it's possible to
create a qemu autopkgtest image without superuser privileges. Maybe other tools
can gain that functionality as well?

> > Alternatively, you could also try --mode=unshare which uses linux user
> > namespaces to avoid using root privileges to build your tarball.
> 
> This seems to require setting "kernel.unprivileged_userns_clone=1" with
> sysctl. But, after having a look at #898446, I am under the impression
> that doing so would reduce the security of my system...

I cannot give you good advice about security.

But if all you are creating a Debian chroot and running only software from
Debian, then you are trusting that software already, no?

I mean, we are not talking about using user namespaces to contain a potentially
malicious proprietary application but about the same code which otherwise
already is running with superuser privileges on millions of systems.

> Any other possibility to create autopkgtest QEMU/KVM images without root
> privileges?

You could also try using --mode=proot.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 17 Nov 2019 16:17:25 +0100 Johannes Schauer wrote:

> Hi,

Hi, thanks again for replying so promptly!

> 
> Quoting Francesco Poli (2019-11-17 16:06:05)
> > I am giving it a try, but I am getting an error I am not quite sure to
> > understand:
> >
> > [...]
> > 
> >   I: automatically chosen mode: fakechroot
> 
> there is your problem. If you are not executing mmdebstrap with superuser
> privileges and if you you did not manually specify which --mode to use, then 
> it
> will look for some way to create the tarball that works on your system. For 
> you
> it picked fakechroot.
[...]

Well, I was trying the command you have previously suggested...

I understand that automatic mode depends on which packages I have
installed on my system and on my sysctl configuration.
The fact is: I was running as a regular user.
I have:

  $ /sbin/sysctl kernel.unprivileged_userns_clone
  kernel.unprivileged_userns_clone = 0

And I have fakechroot on my system, just because mmdebstrap recommends
it:

  $ aptitude why fakechroot
  i   mmdebstrap Recommends fakechroot

Hence, everything conspired to make mmdebstrap choose that mode!

Maybe the recipe you suggested should be improved to address this issue
with the selection of the mode...

> That mode works by using an LD_PRELOAD library to fake
> root permissions for all executed programs. Sometimes this fails badly and the
> postinst script of update-initramfs is a known maintainer script that doesn't
> work under fakechroot mode. I briefly looked into the problem a few days ago
> but couldn't immediately see what was wrong. So this is not a bug in
> mmdebstrap. It is maybe a bug in fakechroot or maybe something could be done
> different by the postinst script of update-initramfs. I don't know whether
> either maintainer considers what you see a bug. You can try and file one and
> see if you get this fixed somehow.

I can try and report bugs, although I don't fully understand what's
going on, and hence I will need your help to explain. Are you willing
to join this bug reporting effort?

Anyway, I am not sure I understand how *you* create an autopkgtest
QEMU/KVM testbed with mmdebstrap. I thought you were successfully using
the recipe you suggested.
Is it not the case? How do *you* do that?

> 
> Another way to solve the problem is by choosing a different mode. If you don't
> mind using superuser privileges, you could run mmdebstrap as root.

I really mind!   ;-)
Creating QEMU/KVM images without superuser privileges was the whole
point of giving mmdebstrap a try.
Otherwise I would just be using autopkgtest-build-qemu, as I said in the
original bug report...

> Alternatively, you could also try --mode=unshare which uses linux user
> namespaces to avoid using root privileges to build your tarball.

This seems to require setting "kernel.unprivileged_userns_clone=1" with
sysctl. But, after having a look at #898446, I am under the impression
that doing so would reduce the security of my system...

Any other possibility to create autopkgtest QEMU/KVM images without
root privileges?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpkntL7vTRGL.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi,

Quoting Francesco Poli (2019-11-17 16:06:05)
> I am giving it a try, but I am getting an error I am not quite sure to
> understand:
>
> [...]
> 
>   I: automatically chosen mode: fakechroot

there is your problem. If you are not executing mmdebstrap with superuser
privileges and if you you did not manually specify which --mode to use, then it
will look for some way to create the tarball that works on your system. For you
it picked fakechroot. That mode works by using an LD_PRELOAD library to fake
root permissions for all executed programs. Sometimes this fails badly and the
postinst script of update-initramfs is a known maintainer script that doesn't
work under fakechroot mode. I briefly looked into the problem a few days ago
but couldn't immediately see what was wrong. So this is not a bug in
mmdebstrap. It is maybe a bug in fakechroot or maybe something could be done
different by the postinst script of update-initramfs. I don't know whether
either maintainer considers what you see a bug. You can try and file one and
see if you get this fixed somehow.

Another way to solve the problem is by choosing a different mode. If you don't
mind using superuser privileges, you could run mmdebstrap as root.
Alternatively, you could also try --mode=unshare which uses linux user
namespaces to avoid using root privileges to build your tarball.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Mon, 11 Nov 2019 00:22:46 +0100 Francesco Poli wrote:

[...]
> I think I will look into mmdebstrap in more depth and then get back to
> you to let you know how it went...

I am giving it a try, but I am getting an error I am not quite sure to
understand:

  $ mmdebstrap --variant=important --include=linux-image-amd64 \
  --customize-hook='chroot "$1" passwd --delete root' \
  --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home 
user' \
  --customize-hook='chroot "$1" passwd --delete user' \
  --customize-hook='echo host > "$1/etc/hostname"' \
  --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
  --customize-hook="/usr/share/autopkgtest/setup-commands/setup-testbed" \
  sid debian-unstable.tar
  I: automatically chosen mode: fakechroot
  I: chroot architecture amd64 is equal to the host's architecture
  [...]
  Setting up linux-image-5.3.0-2-amd64 (5.3.9-2) ...
  I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.3.0-2-amd64
  I: /initrd.img.old is now a symlink to boot/initrd.img-5.3.0-2-amd64
  I: /vmlinuz is now a symlink to boot/vmlinuz-5.3.0-2-amd64
  I: /initrd.img is now a symlink to boot/initrd.img-5.3.0-2-amd64
  /etc/kernel/postinst.d/initramfs-tools:
  update-initramfs: Generating /boot/initrd.img-5.3.0-2-amd64
  W: Possible missing firmware /lib/firmware/rtl_nic/rtl8107e-2.fw for module 
r8169
  [...]
  W: Possible missing firmware /lib/firmware/rtl_nic/rtl8168d-1.fw for module 
r8169
  E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
  update-initramfs: failed for /boot/initrd.img-5.3.0-2-amd64 with 1.
  run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
  dpkg: error processing package linux-image-5.3.0-2-amd64 (--configure):
   installed linux-image-5.3.0-2-amd64 package post-installation script 
subprocess returned error exit status 1
  dpkg: dependency problems prevent configuration of linux-image-amd64:
   linux-image-amd64 depends on linux-image-5.3.0-2-amd64 (= 5.3.9-2); however:
Package linux-image-5.3.0-2-amd64 is not configured yet.
  
  dpkg: error processing package linux-image-amd64 (--configure):
   dependency problems - leaving unconfigured
  Setting up tasksel (3.56) ...
  Setting up tasksel-data (3.56) ...
  Processing triggers for libc-bin (2.29-3) ...
  Processing triggers for systemd (243-7) ...
  Processing triggers for initramfs-tools (0.135) ...
  Errors were encountered while processing:
   linux-image-5.3.0-2-amd64
   linux-image-amd64
  E: Sub-process /usr/bin/dpkg returned an error code (1)
  E: run_chroot failed: E: /usr/sbin/chroot 
/home/frx/var/cache/autopkgtest_TEST1/mmdebstrap.DUEBqVAD7B env 
--unset=APT_CONFIG --unset=TMPDIR apt-get --yes install -oAPT::Status-Fd=<$fd> 
-oDpkg::Use-Pty=false udev procps iproute2 ifupdown apt-utils mount tasksel 
iputils-ping systemd-sysv tzdata linux-image-amd64 debconf-i18n vim-common 
gcc-9-base iptables libpam-runtime e2fsprogs libpam-modules whiptail cron 
passwd isc-dhcp-common libpam-modules-bin init vim-tiny cpio netbase apt 
tasksel-data readline-common systemd kmod dmidecode logrotate sensible-utils 
gpgv mawk debconf bsdmainutils libreadline8 isc-dhcp-client nano 
debian-archive-keyring fdisk adduser less rsyslog failed
  I: removing tempdir /tmp/mmdebstrap.DUEBqVAD7B...


I noticed that my /tmp was almost full, when mmdebstrap failed.
Since I have /tmp mounted on a physical partition:

  $ df --si /tmp/
  Filesystem  Size  Used Avail Use% Mounted on
  /dev/md3869M  934k  806M   1% /tmp

I thought it could be the cause of the issue.
I tried to set:

  $ TMPDIR='./'
  $ export TMPDIR

This forced mmdebstrap to create its temporary directory inside the
current working directory (which is inside my home directory, where at
least 150 GB are available), but did not solve the issue.

Could you please help me understand
why /usr/share/initramfs-tools/hooks/udev failed with exit status == 1 ?


Thanks for your time and patience!

-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgp3ooaUNMMnh.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli]

On Sun, 10 Nov 2019 23:17:04 +0100 Johannes Schauer wrote:

[...]
> Quoting Francesco Poli (wintermute) (2019-11-10 18:44:47)
[...]
> > Could this feature be implemented? It would really be awesome to have
> > a tool that allows a regular user to create a QEMU/KVM minimal Debian
> > image...
> 
> it does not need to be implemented because it is already possible.
[...]

Wow! Thanks for your super-prompt reply!   :-)
And above all, thanks for implementing this already!

I think I will look into mmdebstrap in more depth and then get back to
you to let you know how it went...


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpm5UwU7X6lC.pgp
Description: PGP signature

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Johannes Schauer]

Hi Francesco,

Quoting Francesco Poli (wintermute) (2019-11-10 18:44:47)
> Hello and thanks for developing/packaging this tool!
> 
> I wonder whether it can be used to create (without superuser privileges!)
> a QEMU/KVM image.
> I am especially interested in QEMU/KVM images suitable as autopkgtest
> testbeds (autopkgtest-virt-qemu), but the feature could perhaps be
> useful for building other minimal Debian base QEMU/KVM images as well...
> 
> As you most probably know, autopkgtest-build-qemu uses vmdb2 under the
> hood, and vmdb2 [requires] to be run as root. I wonder whether mmdebstrap
> can be used in stead of vmdb2, in order to lift the superuser privilege
> requirement.
> 
> [requires]: 
> 
> Could this feature be implemented? It would really be awesome to have
> a tool that allows a regular user to create a QEMU/KVM minimal Debian
> image...

it does not need to be implemented because it is already possible.

It works through currently undocumented options that allow for hooks. Well,
actually the documentation already exists but is commented out, so you don't
see it in the man page that is generated from Perl POD. You can read the
documentation by reading the POD at the end of /usr/bin/mmdebstrap. For your
convenience I'll paste you the missing docs at the end of this mail. Part of
the docs is precisely what you were asking for: how to use mmdebstrap to
replace autopkgtest-build-qemu.

Thanks!

cheers, josch


   --setup-hook=command
   Execute arbitrary commands right after initial setup (directory 
creation,
   configuration of apt and dpkg, ...) but before any packages are 
downloaded
   or installed. At that point, the chroot directory does not 
contain any
   executables and thus cannot be chroot-ed into.  The option can be
   specified multiple times and the commands are executed in the 
order in
   which they are given on the command line. If command is an 
existing
   executable file or if command does not contain any shell 
metacharacters,
   then command is directly exec-ed with the path to the chroot 
directory
   passed as the first argument. Otherwise, command is executed 
under sh and
   the chroot directory can be accessed via $1. All environment 
variables
   used by mmdebstrap (like "APT_CONFIG", "DEBIAN_FRONTEND", 
"LC_ALL" and
   "PATH") are preserved.

   Example: Setup merged-/usr via symlinks

   --setup-hook='for d in bin sbin lib; do ln -s usr/$d 
"$1/$d"; mkdir -p "$1/usr/$d"; done'

   Example: Setup chroot for installing a sub-essential 
busybox-based chroot
   with --variant=custom
   
--include=dpkg,busybox,libc-bin,base-files,base-passwd,debianutils

   --setup-hook='mkdir -p "$1/bin"'
   --setup-hook='for p in awk cat chmod chown cp diff echo env 
grep less ln mkdir mount rm rmdir sed sh sleep sort touch uname; do ln -s 
busybox "$1/bin/$p"; done'
   --setup-hook='echo root:x:0:0:root:/root:/bin/sh > 
"$1/etc/passwd"'
   --setup-hook='printf "root:x:0:\nmail:x:8:\nutmp:x:43:\n" > 
"$1/etc/group"'

   --essential-hook=command
   Execute arbitrary commands after the Essential:yes packages have 
been
   installed but before installing the remaining packages. The hook 
is not
   executed for the extract and custom variants. The option can be 
specified
   multiple times and the commands are executed in the order in 
which they
   are given on the command line. If command is an existing 
executable file
   or if command does not contain any shell metacharacters, then 
command is
   directly exec-ed with the path to the chroot directory passed as 
the first
   argument. Otherwise, command is executed under sh and the chroot 
directory
   can be accessed via $1. All environment variables used by 
mmdebstrap (like
   "APT_CONFIG", "DEBIAN_FRONTEND", "LC_ALL" and "PATH") are 
preserved.

   Example: Enable unattended upgrades

   --essential-hook='echo unattended-upgrades 
unattended-upgrades/enable_auto_updates boolean true | chroot "$1" 
debconf-set-selections'

   Example: Select Europe/Berlin as the timezone

   --essential-hook='echo tzdata tzdata/Areas select Europe | 
chroot "$1" debconf-set-selections'
   --essential-hook='echo tzdata tzdata/Zones/Europe select 
Berlin | chroot "$1" debconf-set-selections'
   --customize-hook=command
   Execute arbitrary commands after the chroot is set up and all 
packages got
   installed but before final cleanup actions are carried out.  The 
option
   can be specified multiple times and the commands 

Bug#944485: mmdebstrap: please implement the creation of QEMU/KVM images for autopkgtest-virt-qemu

[Francesco Poli (wintermute)]

Package: mmdebstrap
Version: 0.5.1-2
Severity: wishlist

Hello and thanks for developing/packaging this tool!

I wonder whether it can be used to create (without superuser privileges!)
a QEMU/KVM image.
I am especially interested in QEMU/KVM images suitable as autopkgtest
testbeds (autopkgtest-virt-qemu), but the feature could perhaps be
useful for building other minimal Debian base QEMU/KVM images as well...

As you most probably know, autopkgtest-build-qemu uses vmdb2 under the
hood, and vmdb2 [requires] to be run as root. I wonder whether mmdebstrap
can be used in stead of vmdb2, in order to lift the superuser privilege
requirement.

[requires]: 

Could this feature be implemented? It would really be awesome to have
a tool that allows a regular user to create a QEMU/KVM minimal Debian
image...

Please let me know your thoughts!
Thanks for your time and patience.
Bye.