Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Francesco Poli]

On Thu, 25 Jan 2024 13:32:00 +0100 Johannes Schauer Marin Rodrigues wrote:

> Hi Francesco,

Hello Johannes!

> 
> Quoting Johannes Schauer Marin Rodrigues (2024-01-23 08:56:24)
[...]
> > Amongst other improvements, if "$IMAGE" cannot be accessed by the unshared
> > user, it will error out early with a (hopefully) helpful error message.
> 
> this has now been uploaded as mmdebstrap 1.4.1.

Sorry, I haven't got around to testing it before the upload...

I've just tested it, after upgrading to mmdebstrap/1.4.1-1 from Debian
sid.

> Are you satisfied with the resolution?

That's certainly a significant improvement over the previous state of
affairs!

> Do you have other remarks? I'm going to make another bugfix release
> very soon, so if you like some more changes to be included, now is the time to
> speak up. :)

Yes, I have other remarks.   :-)
But maybe I should include them in brand new bug reports...


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgp3FJnMWKjNt.pgp
Description: PGP signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi Francesco,

Quoting Johannes Schauer Marin Rodrigues (2024-01-23 08:56:24)
> I'm planning to close this bug with this patch:
> 
> https://paste.debian.net/hidden/844859ff/
> 
> Amongst other improvements, if "$IMAGE" cannot be accessed by the unshared
> user, it will error out early with a (hopefully) helpful error message.

this has now been uploaded as mmdebstrap 1.4.1. Are you satisfied with the
resolution? Do you have other remarks? I'm going to make another bugfix release
very soon, so if you like some more changes to be included, now is the time to
speak up. :)

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Control: tag -1 + pending

Hi,

I'm planning to close this bug with this patch:

https://paste.debian.net/hidden/844859ff/

Amongst other improvements, if "$IMAGE" cannot be accessed by the unshared
user, it will error out early with a (hopefully) helpful error message.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi,

thank you Helmut for your (again) very thorough reply. Just as with your 
last large mail to this bug, I can only second the things you said and 
have very little to add.


On 2024-01-16 07:29, Helmut Grohne wrote:

On Tue, Jan 16, 2024 at 12:21:46AM +0100, Francesco Poli wrote:

Well, performance was not an issue here.
The copy of

  $ ls -lFs --si sid_amd64.img
  690M -rw-r-xrwx 1 $USER $USER 2.3G Jan 15 23:26 sid_amd64.img*

was really quick.


Probably. Copying huge chunks of data probably just feels wrong and
technically isn't. We could also generate the file in $TMPDIR in 
general

(hoping it is backed by tmpfs and that the system performing the image
build has at least 4GB ram) and then copy it to the final destination 
in

general. That's usability beating performance.


the patch I posted in an earlier mail to this bug also "solves" the 
problem of missing permission via copying things. Instead of letting 
/sbin/mkfs.ext4 (which runs inside the unshared user namespace) work on 
a file outside of it, the patch runs genext2fs on the outside (thus it 
will happily write to files inside your $HOME) and receives a tarball of 
the rootfs via a pipe from inside the unshared namespace. Then another 
copy is done to shift the image that genext2fs does by the correct 
offset to make room for the first partition.


  d.  configure sbuild-qemu and learn to use it to build Debian 
packages


Does this provide significant advantages over sbuild in unshare mode?


  e.  report to you and/or Christian Kastner (and suggest possible
  improvements for mmdebstrap-autopkgtest-build-qemu and/or
  sbuild-qemu)


I suspect mmdebstrap-autopkgtest-build-qemu very much is unfinished. We
got it into a barely-works.


Yes, but I think there are some immediate things that can be done to 
make the experience a bit better already. For starters, the error 
message that Francesco got initially is really bad and this thing should 
be caught much earlier and give the user an adequate explanation about 
what went wrong and what workaround exists.


I want to do another mmdebstrap release soon and will integrate such a 
check in it. I'll send a patch soon if Francesco wants to try it out 
first.



I think making this all more useful eventually requires finding a
solution for crossing uid boundaries in a sane way. At this time, I see
two fairly generic ways to do this:

a) Have a very simple proxy fuse driver in the initial namespace and
   mount it inside the target namespace. That way an individual
   directory from a different user can be exposed to the namespace
   without granting full access. (Good containment, non-trivial setup,
   slight performance degradation.)

b) Map the calling user in the user namespace. We don't have to map the
   user to root. We can map it to some high uid > 65535. Any process
   that has CAP_DAC_OVERRIDE (and that capability is initially 
available

   and typically available to any process running as uid 0 inside) can
   also access files by your user. (Bad containment, easy setup, no
   performance impact.)


I also think that getting this merged will improve the situation:

https://github.com/tytso/e2fsprogs/pull/118

With that feature, /sbin/mkfs.ext4 can be run on the outside with the 
desired offset for the partition location on the disk image and receive 
a tarball from mmdebstrap via a pipe. Yes, this again means to copy some 
stuff but while the disk itself is 25G, the actual rootfs is not and 
even on my ARM Cortex A53 this is not a noticeable performance impact.


Theodore Ts'o indicated that this pull request will be "shortly 
integrated" here: 
https://lists.debian.org/debian-devel/2023/11/msg00090.html So maybe 
this is another option.


Thanks!

cheers, josch

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Helmut Grohne]

On Tue, Jan 16, 2024 at 12:21:46AM +0100, Francesco Poli wrote:
> Why is my first subuid not my user, but a different user?
> Is it by design?

Yes, this is by design. The use of subuids provides isolation. It's not
that you have just one, but you have a whole range of uids - the first
of which happens to be mapped to root. In particular, this mapping must
be constructed in such a way that every uid inside the namespace is
mapped to a different uid outside the namespace. Hence, you really need
a lot of uids to construct such a namespace. Then the use of different
uids serves isolation: Processes that use the namespace have uids inside
your subuid range outside and then regular permission rules prevent such
processes from doing bad things with your user (such as killing your
processes or messing with your files). While we do not technically need
this property here (as e.g. fakeroot/fakechroot does not provide such
isolation either), it still is a safety-net for things that might go
wrong that we'd like to not give up.

> Or can my first subuid be forced to become my user?

In principle, yes. The mapping currently is:

0 - 65535 -> first_subuid - first_subuid+65535

This is the same mapping that you get with unshare --map-auto.

Instead we could also map like this:

0 -> your_uid
1 - 65535 -> first_subuid - first_subuid+65534

This is roughly what you get with unshare --map-root-user --map-auto.
mmdebstrap does not allow you to configure this aspect at this time.

> Well, performance was not an issue here.
> The copy of
> 
>   $ ls -lFs --si sid_amd64.img 
>   690M -rw-r-xrwx 1 $USER $USER 2.3G Jan 15 23:26 sid_amd64.img*
> 
> was really quick.

Probably. Copying huge chunks of data probably just feels wrong and
technically isn't. We could also generate the file in $TMPDIR in general
(hoping it is backed by tmpfs and that the system performing the image
build has at least 4GB ram) and then copy it to the final destination in
general. That's usability beating performance.

> However, even on a mechanical hard disk drive, I would be more than
> willing to spend some more time in the copy operation. After all, if I
> understand correctly, the image will only be created once, and then kept
> up-to-date with "sbuild-qemu-update" (which does not require superuser
> privileges, does it?).

Alternatively, you may periodically recreate your image. The official
Debian buildd network could also be using sbuild-update to update its
chroots, but instead they periodically recreate them.

> Would the use of the .qcow2 format even further help in this?

The qcow2 format is not using sparseness. Hence, copying cannot inflate
its size. Also qcow2 can be inline-compressed which can further reduce
the image size. autopkgtest will produce an overlay image in any case
rather than modify your base image, so you don't grow it over time while
using it.

>   d.  configure sbuild-qemu and learn to use it to build Debian packages

Does this provide significant advantages over sbuild in unshare mode?

>   e.  report to you and/or Christian Kastner (and suggest possible
>   improvements for mmdebstrap-autopkgtest-build-qemu and/or
>   sbuild-qemu)

I suspect mmdebstrap-autopkgtest-build-qemu very much is unfinished. We
got it into a barely-works.

I think making this all more useful eventually requires finding a
solution for crossing uid boundaries in a sane way. At this time, I see
two fairly generic ways to do this:

a) Have a very simple proxy fuse driver in the initial namespace and
   mount it inside the target namespace. That way an individual
   directory from a different user can be exposed to the namespace
   without granting full access. (Good containment, non-trivial setup,
   slight performance degradation.)

b) Map the calling user in the user namespace. We don't have to map the
   user to root. We can map it to some high uid > 65535. Any process
   that has CAP_DAC_OVERRIDE (and that capability is initially available
   and typically available to any process running as uid 0 inside) can
   also access files by your user. (Bad containment, easy setup, no
   performance impact.)

Helmut

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Francesco Poli]

On Sun, 14 Jan 2024 20:27:58 +0100 Helmut Grohne wrote:

> On Sun, Jan 14, 2024 at 06:36:29PM +0100, Francesco Poli wrote:
> > Of course, I have!   ;-)
> > 
> > For privacy reasons: I don't want other users to be able to enter my
> > home directory and to read any file within it that I might have
> > forgotten with world-readable permissions!
> 
> I agree that this is good practice. It's just that working with
> namespaces tends to make this annoying. We're still in search for a more
> satisfying solution. For now, I'm happy to have identified the cause of
> your failure.

Indeed!
Without understanding the cause of the failure, nobody can find a
(good) fix. Hence, identifying the cause is the first step.

> 
> > the final touches to sid_amd64.img will be put with the file in its
> > intended destination directory, which is /home/${USER}/mysubdir, since
> > the command-line argument was "sid_amd64.img", a relative path to a
> > file directly within the current working directory (~/mysubdir).
> 
> That command that fails is mkfs.ext4. This command is run inside the
> user namespace that mmdebstrap creates for constructing the chroot. The
> uids 0 to 65535 inside the user namespace correspond to your first
> subuid range that is large enough. The mkfs.ext4 operation is performed
> by the root user inside the user namespace. In the initial namespace
> that root user is your first subuid. In particular, it is not your user
> but a different user and this user has no permission to access your
> output image.

Why is my first subuid not my user, but a different user?
Is it by design?
Or can my first subuid be forced to become my user?

[...]
> Let me suggest
> some alternatives. One is storing the output image outside $HOME. If you
> put it into /tmp and the move it from /tmp into your home, that should
> work as well.

I attempted to do this by running mmdebstrap-autopkgtest-build-qemu
from within /dev/shm (and it works, more on this below...).

[...]
> The available options to improve this are not super nice. A simple
> workaround is creating the output image as a temporary file inside the
> namespace and then copying it out. This will perform a 1GB copy
> operation that we'd like to avoid (and rather construct the filesystem
> in-place) for performance reasons, but maybe usability beats
> performance?

Well, performance was not an issue here.
The copy of

  $ ls -lFs --si sid_amd64.img 
  690M -rw-r-xrwx 1 $USER $USER 2.3G Jan 15 23:26 sid_amd64.img*

was really quick.

Maybe it's because my home directory is on an SSD...

However, even on a mechanical hard disk drive, I would be more than
willing to spend some more time in the copy operation. After all, if I
understand correctly, the image will only be created once, and then kept
up-to-date with "sbuild-qemu-update" (which does not require superuser
privileges, does it?).

> I'm also not sure how mmdebstrap downloads sparse files. It
> might make them un-sparse and that'd be quite bad for this use case as
> you'd write 25GB of zeros.

I still have to test with a --size=25G (which is the default size):
a file with actual allocated size of 25 GiB would not fit within
my /dev/shm, but the above-mentioned test with --size=2G produced an
image with an allocated size of about 690 MB, so maybe it can work even
with --size=25G ?!?

Would the use of the .qcow2 format even further help in this?

[...]
> > It looks like it worked, but, unfortunately, it (again) fails to be
> > usable with autopkgtest:
> > 
> >   $ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out  --summary 
> > ./${SRCPKG}_autopkgtest.summary --apt-upgrade -B ./${SRCPKG}_amd64.changes 
> > -- qemu --overlay-dir /dev/shm ~/Downloads/sid_amd64.img
> >   autopkgtest [18:24:23]: starting date and time: 2024-01-14 18:24:23+0100
> >   autopkgtest [18:24:23]: version 5.32
> >   autopkgtest [18:24:23]: host ${HOST}; command line: /usr/bin/autopkgtest 
> > --output-dir './${SRCPKG}_autopkgtest.out' --summary 
> > './${SRCPKG}_autopkgtest.summary' --apt-upgrade -B 
> > './${SRCPKG}_amd64.changes' -- qemu --overlay-dir /dev/shm 
> > ${HOME}/Downloads/sid_amd64.img
> >   qemu-system-x86_64: terminating on signal 15 from pid 115770 
> > (/usr/bin/python3)
> >   : failure: timed out waiting for 'login prompt on serial 
> > console'
> >   autopkgtest [18:25:24]: ERROR: testbed failure: unexpected eof from the 
> > testbed
> > 
> > 
> > What did I fail to understand?
> 
> The difficulty resides in making a bootable image with no root
> privileges. mmdebstrap-autopkgtest-build-qemu is not fully compatible
> with autopkgtest-build-qemu. It specifically required you to pass
> --boot=efi, right? On amd64, autopkgtest-virt-qemu defaults to bios
> boot. In order to use this image, you likewise have to pass --boot=efi
> to the virtualization backend.

Adding the --boot=efi option made it work!   :-)

  $ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out  --summary 
./${SRCPKG}_autopkgtest.summary --apt-upgrade -B 

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2024-01-14 18:33:14)
>   The partition table has been altered.
>   Syncing disks.
>   $ echo $?
>   0
>   $ ls -lF --si sid_amd64.img 
>   -rw-r-xrwx 1 $USER $USER 670M Jan 14 17:21 sid_amd64.img*
> 
> However, if I attempt to use the resulting image, autopkgtest
> fails:
> 
>   $ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out  --summary 
> ./${SRCPKG}_autopkgtest.summary --apt-upgrade -B ./${SRCPKG}_amd64.changes -- 
> qemu --overlay-dir /dev/shm ~/Downloads/sid_amd64.img
>   autopkgtest [17:27:29]: starting date and time: 2024-01-14 17:27:29+0100
>   autopkgtest [17:27:29]: version 5.32
>   autopkgtest [17:27:29]: host ${HOST}; command line: /usr/bin/autopkgtest 
> --output-dir './${SRCPKG}_autopkgtest.out' --summary 
> './${SRCPKG}_autopkgtest.summary' --apt-upgrade -B 
> './${SRCPKG}_amd64.changes' -- qemu --overlay-dir /dev/shm 
> ${HOME}/Downloads/sid_amd64.img
>   qemu-system-x86_64: terminating on signal 15 from pid 115770 
> (/usr/bin/python3)
>   : failure: timed out waiting for 'login prompt on serial 
> console'
>   autopkgtest [17:28:29]: ERROR: testbed failure: unexpected eof from the 
> testbed
> 
> 
> Was it just a test to investigate further?
> Or did it have a chance to produce a usable image?

The qemu image you built uses efi booting. Try to add --boot=efi. The default
is --boot=auto which chooses bios boot on amd64.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Helmut Grohne]

On Sun, Jan 14, 2024 at 06:36:29PM +0100, Francesco Poli wrote:
> Of course, I have!   ;-)
> 
> For privacy reasons: I don't want other users to be able to enter my
> home directory and to read any file within it that I might have
> forgotten with world-readable permissions!

I agree that this is good practice. It's just that working with
namespaces tends to make this annoying. We're still in search for a more
satisfying solution. For now, I'm happy to have identified the cause of
your failure.

> the final touches to sid_amd64.img will be put with the file in its
> intended destination directory, which is /home/${USER}/mysubdir, since
> the command-line argument was "sid_amd64.img", a relative path to a
> file directly within the current working directory (~/mysubdir).

That command that fails is mkfs.ext4. This command is run inside the
user namespace that mmdebstrap creates for constructing the chroot. The
uids 0 to 65535 inside the user namespace correspond to your first
subuid range that is large enough. The mkfs.ext4 operation is performed
by the root user inside the user namespace. In the initial namespace
that root user is your first subuid. In particular, it is not your user
but a different user and this user has no permission to access your
output image.

> And these final touches require all parent directories (up to the root
> directory) to be world-executable.

Technically speaking, this is a sufficient but not strictly necessary
precondition. What we actually needs is your first subuid to be able to
open the output image for writing. This is difficult to achieve. The
method implemented in mmdebstrap-autopkgtest-build-qemu is to
temporarily change the ownership of the output image to the first
subuid. While chown is normally a privileged operation, you can chown
inside your namespace if both the original owner and the new owner are
mapped inside your user namespace. This is not the case for the
namespace that mmdebstrap creates, but before calling mmdebstrap, we
construct another namespace suitable for performing this chown.

What we don't do is adjust the permission of leading directory
components. The ability for opening the output image depends on both the
final component having a +w bit in an appropriate place (and since we
care fully chose the owner this is u+w) and leading directory components
being +x (and since these are neither owned by the first subuid nor the
first subgid, it amounts to o+x).

> In other words, the user would need:
> 
>   $ stat -c "%a %n" ~/mysubdir ~ /home /
>   771 /home/${USER}/mysubdir
>   701 /home/${USER}
>   755 /home
>   755 /
> 
> rather than:
> 
>   $ stat -c "%a %n" ~/mysubdir ~ /home /
>   770 /home/${USER}/mysubdir
>   700 /home/${USER}
>   755 /home
>   755 /
> 
> Is that correct?
> Or am I completely off-track?

This is one way to establish the actual precondition. Let me suggest
some alternatives. One is storing the output image outside $HOME. If you
put it into /tmp and the move it from /tmp into your home, that should
work as well. Another is using ACLs. You might add an ACL that
specifically grants your subuid range execute permission on your home
but not every user.

The available options to improve this are not super nice. A simple
workaround is creating the output image as a temporary file inside the
namespace and then copying it out. This will perform a 1GB copy
operation that we'd like to avoid (and rather construct the filesystem
in-place) for performance reasons, but maybe usability beats
performance? I'm also not sure how mmdebstrap downloads sparse files. It
might make them un-sparse and that'd be quite bad for this use case as
you'd write 25GB of zeros. Another option could be using fuse. In
principle, we could have a fuse driver running in the initial namespace
that just serves the output image. Then inside the namespace we could
mount this fuse filesystem (which becomes a regular file, not a
directory) and write to it. Having the fuse driver run in one namespace
and the fuse mount in a different namespace resolves the permission
problem. This is all non-trivial to implement though. The plumbing isn't
up to the task. Maybe there are other options that I don't see at this
time?

> It looks like it worked, but, unfortunately, it (again) fails to be
> usable with autopkgtest:
> 
>   $ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out  --summary 
> ./${SRCPKG}_autopkgtest.summary --apt-upgrade -B ./${SRCPKG}_amd64.changes -- 
> qemu --overlay-dir /dev/shm ~/Downloads/sid_amd64.img
>   autopkgtest [18:24:23]: starting date and time: 2024-01-14 18:24:23+0100
>   autopkgtest [18:24:23]: version 5.32
>   autopkgtest [18:24:23]: host ${HOST}; command line: /usr/bin/autopkgtest 
> --output-dir './${SRCPKG}_autopkgtest.out' --summary 
> './${SRCPKG}_autopkgtest.summary' --apt-upgrade -B 
> './${SRCPKG}_amd64.changes' -- qemu --overlay-dir /dev/shm 
> ${HOME}/Downloads/sid_amd64.img
>   qemu-system-x86_64: terminating on signal 15 from pid 

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Francesco Poli]

On Sat, 13 Jan 2024 22:09:22 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]
> Quoting Johannes Schauer Marin Rodrigues (2024-01-13 20:18:34)
[...]
> > The problem occurs for Francesco when
> > using either /tmp or /dev/shm as a temporary directly. By default, those two
> > locations should have the desired permission bits set. Lets check whether
> > world-execute permissions are set for all directories up until root. I have
> > this:
> > 
> > $ stat -c "%a %n" / /dev /dev/shm /tmp
> > 755 /
> > 755 /dev
> > 1777 /dev/shm
> > 1777 /tmp
> > 
> > Francesco, what are the world execute permissions on all path components up
> > to /tmp and /devv/shm in your case?

Just like yours:

  $ stat -c "%a %n" / /dev /dev/shm /tmp
  755 /
  755 /dev
  1777 /dev/shm
  1777 /tmp


> 
> scratch what I said in your last mail. Your problem is with the creation of 
> the
> image and not with the temporary chroot directory. What Helmut said is likely
> correct and exactly the problem you are facing. To verify, you could run the
> command you posted initially inside your $HOME directory and show us the
> permission bits of your $HOME. Mine are these:
> 
> stat -c "%a %n" $HOME
> 711 /home/josch
> 
> I assume in your case, you have a 0 at the end of the octal permissions?

Of course, I have!   ;-)

For privacy reasons: I don't want other users to be able to enter my
home directory and to read any file within it that I might have
forgotten with world-readable permissions!

So, the answer to your question is that I indeed have 0 at the end of
the octal permissions:

  $ stat -c "%a %n" $HOME
  700 /home/$USER


OK, please let me understand.

mmdebstrap-autopkgtest-build-qemu creates a chroot within the temporary
directory (TMPDIR=/dev/shm, in my case), but then performs some further
operations (such as mkfs.ext4) on the image file, which is being
created in the directory the user wanted it to be.

Hence, if the user issues the command

  $ cd ~/mysubdir
  $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
--boot=efi sid sid_amd64.img

the final touches to sid_amd64.img will be put with the file in its
intended destination directory, which is /home/${USER}/mysubdir, since
the command-line argument was "sid_amd64.img", a relative path to a
file directly within the current working directory (~/mysubdir).

And these final touches require all parent directories (up to the root
directory) to be world-executable.
In other words, the user would need:

  $ stat -c "%a %n" ~/mysubdir ~ /home /
  771 /home/${USER}/mysubdir
  701 /home/${USER}
  755 /home
  755 /

rather than:

  $ stat -c "%a %n" ~/mysubdir ~ /home /
  770 /home/${USER}/mysubdir
  700 /home/${USER}
  755 /home
  755 /

Is that correct?
Or am I completely off-track?


I tried to do everything within /dev/shm, with:

  $ cd /dev/shm
  $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
--size=2G --boot=efi sid sid_amd64.img
  [...]
  cp: warning: behavior of -n is non-portable and may change in future; use 
--update=none instead
  I: running special hook: download vmlinuz '/dev/shm/tmp.hhjurdQFhk/kernel'
  I: running special hook: download initrd.img '/dev/shm/tmp.hhjurdQFhk/initrd'
  I: running --customize-hook in shell: sh -c 'mount --bind "$1" "$1/mnt"' exec 
/dev/shm/mmdebstrap.PeOPcJxciV
  I: running --customize-hook in shell: sh -c 'mount --bind "$1/mnt/mnt" 
"$1/mnt/dev"' exec /dev/shm/mmdebstrap.PeOPcJxciV
  I: running --customize-hook in shell: sh -c '/sbin/mkfs.ext4 -d "$1/mnt" -L 
autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 'sid_amd64.img' 
'2G'' exec /dev/shm/mmdebstrap.PeOPcJxciV
  mke2fs 1.47.0 (5-Feb-2023)
  Creating filesystem with 524288 4k blocks and 131072 inodes
  Filesystem UUID: 6d0afa2d-5ee5-4a45-b2ef-1bdb74315ad7
  Superblock backups stored on blocks: 
  32768, 98304, 163840, 229376, 294912
  
  Allocating group tables: done
  Writing inode tables: done
  Creating journal (16384 blocks): done
  Copying files into the device: done
  Writing superblocks and filesystem accounting information: done 
  
  I: running --customize-hook in shell: sh -c 'umount --lazy "$1/mnt"' exec 
/dev/shm/mmdebstrap.PeOPcJxciV
  I: cleaning package lists and apt cache...
  done
  done
  I: removing tempdir /dev/shm/mmdebstrap.PeOPcJxciV...
  I: success in 120.0573 seconds
  determined efi vma alignment as 0x0001000
  determined minimum efi vma offset as 5603221504
  mkfs.fat 4.2 (2021-01-31)
  Checking that no-one is using this disk right now ... OK
  
  Disk sid_amd64.img: 2.13 GiB, 2281718784 bytes, 4456482 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  
  >>> Script header accepted.
  >>> Script header accepted.
  >>> Created a new GPT disklabel (GUID: 39BDEF0B-9FF8-4DB9-921B-FA7A6C2CDB54).
  

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Francesco Poli]

On Fri, 12 Jan 2024 23:39:34 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]
> could you try applying this patch and then try again:
> 
> --%<-
> --- a/mmdebstrap-autopkgtest-build-qemu
> +++ b/mmdebstrap-autopkgtest-build-qemu
> @@ -302,17 +302,12 @@ if test -n "$SCRIPT"; then
> '--customize-hook=rm -f "$1/userscript"'
>  fi
>  
> -EXT4_OFFSET_BYTES=$(( (FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS) * 512))
> -EXT4_OPTIONS="offset=$EXT4_OFFSET_BYTES,assume_storage_prezeroed=1"
>  set -- "$@" \
> "--customize-hook=download vmlinuz '$WORKDIR/kernel'" \
> "--customize-hook=download initrd.img '$WORKDIR/initrd'" \
> -   '--customize-hook=mount --bind "$1" "$1/mnt"' \
> -   '--customize-hook=mount --bind "$1/mnt/mnt" "$1/mnt/dev"' \
> -   '--customize-hook=/sbin/mkfs.ext4 -d "$1/mnt" -L autopkgtestvm -E 
> '"'$EXT4_OPTIONS' '$IMAGE' '$SIZE'" \
> -   '--customize-hook=umount --lazy "$1/mnt"' \
> +   --format=ext2 \
> "$RELEASE" \
> -   /dev/null
> +   "$IMAGE"
>  
>  test -n "$MIRROR" && set -- "$@" "$MIRROR"
>  test -n "$KEYRING" && set -- "$@" "--keyring=$KEYRING"
> @@ -320,6 +315,9 @@ test -n "$KEYRING" && set -- "$@" "--keyring=$KEYRING"
>  echo "mmdebstrap $*"
>  mmdebstrap "$@" || die "mmdebstrap failed"
>  
> +EXT4_OFFSET_BYTES=$(( (FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS) * 512))
> +fallocate --insert-range --offset=0 --length=$EXT4_OFFSET_BYTES "$IMAGE"
> +
>  unshare -U -r --map-groups=auto chown 0:0 "$IMAGE"
>  chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE"
>  
> -->%-

I have just tried.

But the result is not clear to me.
The image generation seems to succeed:

  $ cd ~/Downloads
  $ TMPDIR=/dev/shm ./test/mmdebstrap-autopkgtest-build-qemu \
--size=2G --boot=efi sid sid_amd64.img 
  [...]
  cp: warning: behavior of -n is non-portable and may change in future; use 
--update=none instead
  I: running special hook: download vmlinuz '/dev/shm/tmp.3NxJ7PeIpL/kernel'
  I: running special hook: download initrd.img '/dev/shm/tmp.3NxJ7PeIpL/initrd'
  I: cleaning package lists and apt cache...
  done
  done
  I: approximating disk usage...
  I: creating tarball...
  copying from tar archive -
  I: done
  I: removing tempdir /dev/shm/mmdebstrap.Dgp1UFANSi...
  I: success in 125.7682 seconds
  determined efi vma alignment as 0x0001000
  determined minimum efi vma offset as 5603221504
  mkfs.fat 4.2 (2021-01-31)
  Checking that no-one is using this disk right now ... OK
  
  Disk sid_amd64.img: 638.58 MiB, 669594624 bytes, 1307802 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  
  >>> Script header accepted.
  >>> Script header accepted.
  >>> Created a new GPT disklabel (GUID: EEDDF652-1EF3-4E3D-9994-1FD7E76B640A).
  sid_amd64.img1: Created a new partition 1 of type 'EFI System' and of size 
127 MiB.
  sid_amd64.img2: Created a new partition 2 of type 'Linux filesystem' and of 
size 510 MiB.
  Partition #2 contains a ext2 signature.
  sid_amd64.img3: Done.
  
  New situation:
  Disklabel type: gpt
  Disk identifier: EEDDF652-1EF3-4E3D-9994-1FD7E76B640A
  
  Device  Start End Sectors  Size Type
  sid_amd64.img1   2048  262143  260096  127M EFI System
  sid_amd64.img2 262144 1306623 1044480  510M Linux filesystem
  
  The partition table has been altered.
  Syncing disks.
  $ echo $?
  0
  $ ls -lF --si sid_amd64.img 
  -rw-r-xrwx 1 $USER $USER 670M Jan 14 17:21 sid_amd64.img*

However, if I attempt to use the resulting image, autopkgtest
fails:

  $ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out  --summary 
./${SRCPKG}_autopkgtest.summary --apt-upgrade -B ./${SRCPKG}_amd64.changes -- 
qemu --overlay-dir /dev/shm ~/Downloads/sid_amd64.img
  autopkgtest [17:27:29]: starting date and time: 2024-01-14 17:27:29+0100
  autopkgtest [17:27:29]: version 5.32
  autopkgtest [17:27:29]: host ${HOST}; command line: /usr/bin/autopkgtest 
--output-dir './${SRCPKG}_autopkgtest.out' --summary 
'./${SRCPKG}_autopkgtest.summary' --apt-upgrade -B './${SRCPKG}_amd64.changes' 
-- qemu --overlay-dir /dev/shm ${HOME}/Downloads/sid_amd64.img
  qemu-system-x86_64: terminating on signal 15 from pid 115770 
(/usr/bin/python3)
  : failure: timed out waiting for 'login prompt on serial console'
  autopkgtest [17:28:29]: ERROR: testbed failure: unexpected eof from the 
testbed


Was it just a test to investigate further?
Or did it have a chance to produce a usable image?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgp3i3hxNJ1Ye.pgp
Description: PGP signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Johannes Schauer Marin Rodrigues (2024-01-13 20:18:34)
> Quoting Helmut Grohne (2024-01-12 21:58:00)
> > > What can cause mkfs.ext4 to fail with a "Permission denied" error?
> > I think this is our typical problem when dealing with user namespaces. I
> > guess that the thing that fails here is mkfs.ext4 opening the target image
> > file (to be formatted). That file has earlier been chowned to the root uid
> > inside the namespace, so permission should be there, but you need more. You
> > also need execute permission (to the first uid of your namespace) for the
> > containing directory up until the root. I guess that none of those are
> > world-executable and not by chanced owned by your first subuid nor owned by
> > the first group in your subgid range.
> 
> I'm not yet convinced that this is it. The problem occurs for Francesco when
> using either /tmp or /dev/shm as a temporary directly. By default, those two
> locations should have the desired permission bits set. Lets check whether
> world-execute permissions are set for all directories up until root. I have
> this:
> 
> $ stat -c "%a %n" / /dev /dev/shm /tmp
> 755 /
> 755 /dev
> 1777 /dev/shm
> 1777 /tmp
> 
> Francesco, what are the world execute permissions on all path components up
> to /tmp and /devv/shm in your case?

scratch what I said in your last mail. Your problem is with the creation of the
image and not with the temporary chroot directory. What Helmut said is likely
correct and exactly the problem you are facing. To verify, you could run the
command you posted initially inside your $HOME directory and show us the
permission bits of your $HOME. Mine are these:

stat -c "%a %n" $HOME
711 /home/josch

I assume in your case, you have a 0 at the end of the octal permissions?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2024-01-12 21:58:00)
> > What can cause mkfs.ext4 to fail with a "Permission denied" error?
> I think this is our typical problem when dealing with user namespaces. I
> guess that the thing that fails here is mkfs.ext4 opening the target image
> file (to be formatted). That file has earlier been chowned to the root uid
> inside the namespace, so permission should be there, but you need more. You
> also need execute permission (to the first uid of your namespace) for the
> containing directory up until the root. I guess that none of those are
> world-executable and not by chanced owned by your first subuid nor owned by
> the first group in your subgid range.

I'm not yet convinced that this is it. The problem occurs for Francesco when
using either /tmp or /dev/shm as a temporary directly. By default, those two
locations should have the desired permission bits set. Lets check whether
world-execute permissions are set for all directories up until root. I have
this:

$ stat -c "%a %n" / /dev /dev/shm /tmp
755 /
755 /dev
1777 /dev/shm
1777 /tmp

Francesco, what are the world execute permissions on all path components up to
/tmp and /devv/shm in your case?

Arguably, mmdebstrap maybe should do this check at start-up. To have a
real-world test-case for such a feature, I'm eager to see how Francesco's
system looks like.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Helmut Grohne]

Hi Francesco,

On Fri, Jan 12, 2024 at 08:05:02PM +0100, Francesco Poli wrote:
> Indeed, the error does not seem to have anything to do with a "No space
> left on device" failure:
> 
> [...]
>   mke2fs 1.47.0 (5-Feb-2023)
>   mkfs.ext4: Permission denied while trying to determine filesystem size
>   E: setup failed: E: command failed: /sbin/mkfs.ext4 -d "$1/mnt" -L 
> autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 
> 'sid_amd64.img' '25G'
> [...]
> 
> What can cause mkfs.ext4 to fail with a "Permission denied" error?

I think this is our typical problem when dealing with user namespaces. I
guess that the thing that fails here is mkfs.ext4 opening the target
image file (to be formatted). That file has earlier been chowned to the
root uid inside the namespace, so permission should be there, but you
need more. You also need execute permission (to the first uid of your
namespace) for the containing directory up until the root. I guess that
none of those are world-executable and not by chanced owned by your
first subuid nor owned by the first group in your subgid range. It's an
odd requirement and unfortunately one that is very difficult to get rid
of.

Can we accept and document this limitation for now?

Helmut

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2024-01-12 20:05:02)
> On Thu, 11 Jan 2024 21:04:16 +0100 Johannes Schauer Marin Rodrigues wrote:
> > Do you see yourself debugging this further by yourself? You probably
> > understand that it's hard for me to investigate something that I cannot
> > test myself.
> 
> Well, you wrote the script, together with Helmut Grohne (who reads us
> in Cc).
> I rather see myself helping the two of you in debugging it...   :-)

could you try applying this patch and then try again:

--%<-
--- a/mmdebstrap-autopkgtest-build-qemu
+++ b/mmdebstrap-autopkgtest-build-qemu
@@ -302,17 +302,12 @@ if test -n "$SCRIPT"; then
'--customize-hook=rm -f "$1/userscript"'
 fi
 
-EXT4_OFFSET_BYTES=$(( (FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS) * 512))
-EXT4_OPTIONS="offset=$EXT4_OFFSET_BYTES,assume_storage_prezeroed=1"
 set -- "$@" \
"--customize-hook=download vmlinuz '$WORKDIR/kernel'" \
"--customize-hook=download initrd.img '$WORKDIR/initrd'" \
-   '--customize-hook=mount --bind "$1" "$1/mnt"' \
-   '--customize-hook=mount --bind "$1/mnt/mnt" "$1/mnt/dev"' \
-   '--customize-hook=/sbin/mkfs.ext4 -d "$1/mnt" -L autopkgtestvm -E 
'"'$EXT4_OPTIONS' '$IMAGE' '$SIZE'" \
-   '--customize-hook=umount --lazy "$1/mnt"' \
+   --format=ext2 \
"$RELEASE" \
-   /dev/null
+   "$IMAGE"
 
 test -n "$MIRROR" && set -- "$@" "$MIRROR"
 test -n "$KEYRING" && set -- "$@" "--keyring=$KEYRING"
@@ -320,6 +315,9 @@ test -n "$KEYRING" && set -- "$@" "--keyring=$KEYRING"
 echo "mmdebstrap $*"
 mmdebstrap "$@" || die "mmdebstrap failed"
 
+EXT4_OFFSET_BYTES=$(( (FAT_OFFSET_SECTORS + FAT_SIZE_SECTORS) * 512))
+fallocate --insert-range --offset=0 --length=$EXT4_OFFSET_BYTES "$IMAGE"
+
 unshare -U -r --map-groups=auto chown 0:0 "$IMAGE"
 chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE"
 
-->%-

> Indeed, the error does not seem to have anything to do with a "No space left
> on device" failure:
> 
> [...]
>   mke2fs 1.47.0 (5-Feb-2023)
>   mkfs.ext4: Permission denied while trying to determine filesystem size
>   E: setup failed: E: command failed: /sbin/mkfs.ext4 -d "$1/mnt" -L 
> autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 
> 'sid_amd64.img' '25G'
> [...]
> 
> What can cause mkfs.ext4 to fail with a "Permission denied" error?

This is unshare mode. Funny things happen. :)

The error message you see comes from misc/mke2fs.c in e2fsprogs as a result of
a failed call to ext2fs_open_file(). What puzzles me is, that it's permission
denied for you and working fine for me.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Francesco Poli]

On Thu, 11 Jan 2024 21:04:16 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]
> Do you see yourself debugging this further by yourself? You probably 
> understand
> that it's hard for me to investigate something that I cannot test myself.

Well, you wrote the script, together with Helmut Grohne (who reads us
in Cc).
I rather see myself helping the two of you in debugging it...   :-)

I've just performed another test, setting a much smaller size for the
image to be created:

  $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
--size=2G --boot=efi sid sid_amd64.img

but the result was the same exact error that I have previously reported
in the original [bug] report.

[bug]: 

Hence, it seems to me that the issue is not with the size of the
filesystem where the temporary directory is created.

Indeed, the error does not seem to have anything to do with a "No space
left on device" failure:

[...]
  mke2fs 1.47.0 (5-Feb-2023)
  mkfs.ext4: Permission denied while trying to determine filesystem size
  E: setup failed: E: command failed: /sbin/mkfs.ext4 -d "$1/mnt" -L 
autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 'sid_amd64.img' 
'25G'
[...]

What can cause mkfs.ext4 to fail with a "Permission denied" error?



-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpwrRkROgULm.pgp
Description: PGP signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2024-01-11 08:58:49)
> On Thu, 11 Jan 2024 00:45:50 +0100 Johannes Schauer Marin Rodrigues wrote:
> > Interesting! I'm unable to reproduce either of these issues and I'm also
> > puzzled why you get this "permission denied" error. On my system, I'm able 
> > run
> > your command with the default (in /tmp) as well as in /dev/shm. Here is the
> > free space each:
> > 
> > $ df --si /tmp /dev/shm
> > Filesystem  Size  Used Avail Use% Mounted on
> > tmpfs   8.6G  4.1k  8.6G   1% /tmp
> > tmpfs   2.0G  418k  2.0G   1% /dev/shm
> So 2.0 GB should be enough.
> And yet, it fails for me with 8.2 GB available on /dev/shm ...  :-(

it's hard for me to debug this as I don't see the error myself. I wonder how I
can be of further assistance to you to figure out what is going on?

> > Maybe something is mounted in a funny way? Both my /tmp and my /dev/shm are
> > tmpfs. The former is mounted with
> > rw,nosuid,nodev,relatime,size=8388608k,inode64 and the latter with
> > rw,nosuid,nodev,inode64.
> 
>   $ mount | grep '/tmp\|shm'
>   tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
>   /dev/md3 on /tmp type ext4 (rw,relatime,discard)

Yes, that looks harmless.

> > I doubt that it fails for lack of system memory unless you have less than 
> > 3.7
> > GB of RAM.
>   $ free --giga
>  totalusedfree  shared  buff/cache   
> available
>   Mem:  16   2  12   0   2
>   14
>   Swap: 17   0  17

$ free --giga
   totalusedfree  shared  buff/cache   
available
Mem:   3   3   0   0   1
   0
Swap:  8   0   8

Your values are all twice as big or more than mine.

> > Maybe permissions are somehow whacky? Both my /tmp and my /dev/shm are set 
> > to
> > drwxrwxrwt (so there is a sticky bit set) and owned by root:root.
> 
>   $ ls -ld /tmp /dev/shm
>   drwxrwxrwt  3 root root  200 Jan 11 08:25 /dev/shm
>   drwxrwxrwt 13 root root 4096 Jan 11 08:36 /tmp

Looks good!

> > What happens with other locations for TMPDIR?
> 
> Which other locations?
> I cannot use anything within my home directory, as you may recall from
> our past [discussions]
> 
> [discussions]: 

Yes, this also reminds me of #1052471 which found libpam-tmpdir directories
like /tmp/user/1000 to be unusable in unshare mode.

> > Maybe. It's currently .img because it's easier to debug stuff and use kpartx
> > with it without having to extract it first. :)
> Do I understand correctly that you intend to switch to .qcow2 in the future?
> Or otherwise, what's your plan?

I don't have one yet. It's a work in progress. Your input helps me to decide in
which direction to move. :)

> > > Please clarify and/or improve mmdebstrap-autopkgtest-build-qemu .
> > > Thanks for your time!
> > Thanks for your bug! Lets hope we get to the bottom of this.
> You're welcome.
> 
> Looking forward to hearing back from you!

Do you see yourself debugging this further by yourself? You probably understand
that it's hard for me to investigate something that I cannot test myself.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Francesco Poli]

On Thu, 11 Jan 2024 00:45:50 +0100 Johannes Schauer Marin Rodrigues wrote:

[...]
> Interesting! I'm unable to reproduce either of these issues and I'm also
> puzzled why you get this "permission denied" error. On my system, I'm able run
> your command with the default (in /tmp) as well as in /dev/shm. Here is the
> free space each:
> 
> $ df --si /tmp /dev/shm
> Filesystem  Size  Used Avail Use% Mounted on
> tmpfs   8.6G  4.1k  8.6G   1% /tmp
> tmpfs   2.0G  418k  2.0G   1% /dev/shm

So 2.0 GB should be enough.
And yet, it fails for me with 8.2 GB available on /dev/shm ...  :-(

> 
> Maybe something is mounted in a funny way? Both my /tmp and my /dev/shm are
> tmpfs. The former is mounted with
> rw,nosuid,nodev,relatime,size=8388608k,inode64 and the latter with
> rw,nosuid,nodev,inode64.

  $ mount | grep '/tmp\|shm'
  tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
  /dev/md3 on /tmp type ext4 (rw,relatime,discard)

> 
> I doubt that it fails for lack of system memory unless you have less than 3.7
> GB of RAM.

  $ free --giga
 totalusedfree  shared  buff/cache   
available
  Mem:  16   2  12   0   2  
14
  Swap: 17   0  17

> 
> Maybe permissions are somehow whacky? Both my /tmp and my /dev/shm are set to
> drwxrwxrwt (so there is a sticky bit set) and owned by root:root.

  $ ls -ld /tmp /dev/shm
  drwxrwxrwt  3 root root  200 Jan 11 08:25 /dev/shm
  drwxrwxrwt 13 root root 4096 Jan 11 08:36 /tmp

> 
> What happens with other locations for TMPDIR?

Which other locations?
I cannot use anything within my home directory, as you may recall from
our past [discussions]

[discussions]: 

And indeed, anything within my home directory is still unusable:

  $ cd ~/Downloads
  $ mkdir TEMP
  $ TMPDIR=./TEMP mmdebstrap-autopkgtest-build-qemu \
  --boot=efi sid sid_amd64.img
  [...]
  E: cannot create /home/$USER/Downloads: Permission denied; cannot create 
/home/$USER/Downloads/TEMP: Permission denied; cannot create 
/home/$USER/Downloads/TEMP/mmdebstrap.XSbRetZNZV: Permission denied; cannot 
create /home/$USER/Downloads/TEMP/mmdebstrap.XSbRetZNZV//etc: Permission 
denied; cannot create 
/home/$USER/Downloads/TEMP/mmdebstrap.XSbRetZNZV//etc/apt: Permission denied; 
cannot create 
/home/$USER/Downloads/TEMP/mmdebstrap.XSbRetZNZV//etc/apt/apt.conf.d: 
Permission denied
  W: hooklistener errored out: E: received eof on socket
  
  I: main() received signal PIPE: waiting for setup...
  I: removing tempdir /home/$USER/Downloads/TEMP/mmdebstrap.XSbRetZNZV...
  env: cannot change directory to 
'/home/$USER/Downloads/TEMP/mmdebstrap.XSbRetZNZV': Permission denied
  E: rm failed: 32000
  E: remove_tree failed
  mmdebstrap failed


So, what's left?!?
I think /dev/shm is the only possible choice (without root privileges).

> 
> > By the way, the old script that used guestfish (with all its limitations)
> > was able to create a QEMU image in .qcow2 format and my /dev/shm was
> > enough for it to work correctly.
> 
> It should be enough. Your /dev/shm is 8G large and it works with my 2G.
> 
> > Why does the current mmdebstrap-autopkgtest-build-qemu create a QEMU
> > image in .img format? Isn't the .qcow2 format better?
> 
> Maybe. It's currently .img because it's easier to debug stuff and use kpartx
> with it without having to extract it first. :)

Do I understand correctly that you intend to switch to .qcow2 in the future?
Or otherwise, what's your plan?

> 
> > Please clarify and/or improve mmdebstrap-autopkgtest-build-qemu .
> > Thanks for your time!
> 
> Thanks for your bug! Lets hope we get to the bottom of this.

You're welcome.

Looking forward to hearing back from you!


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpCrk8PurQoy.pgp
Description: PGP signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (wintermute) (2024-01-10 23:54:30)
> I am giving mmdebstrap-autopkgtest-build-qemu a try.

thank you!

> The following command fails:
> 
>   $ mmdebstrap-autopkgtest-build-qemu --boot=efi sid sid_amd64.img
> 
> during some package installation with "no space left on device" error,
> since I have /tmp on a somewhat small physical partition:
> 
>   $ df --si /tmp/
>   Filesystem  Size  Used Avail Use% Mounted on
>   /dev/md3868M   99k  806M   1% /tmp

yes, that looks too small.

> I tried with a TMPDIR in system memory:
> 
>   $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
> --boot=efi sid sid_amd64.img
> 
> but it again fails with the following (final chunk of) output:
> 
>   [...]
>   cp: warning: behavior of -n is non-portable and may change in future; use 
> --update=none instead
>   I: running special hook: download vmlinuz '/dev/shm/tmp.uNRUbVgiOu/kernel'
>   I: running special hook: download initrd.img 
> '/dev/shm/tmp.uNRUbVgiOu/initrd'
>   I: running --customize-hook in shell: sh -c 'mount --bind "$1" "$1/mnt"' 
> exec /dev/shm/mmdebstrap.IXehDNUWIf
>   I: running --customize-hook in shell: sh -c 'mount --bind "$1/mnt/mnt" 
> "$1/mnt/dev"' exec /dev/shm/mmdebstrap.IXehDNUWIf
>   I: running --customize-hook in shell: sh -c '/sbin/mkfs.ext4 -d "$1/mnt" -L 
> autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 
> 'sid_amd64.img' '25G'' exec /dev/shm/mmdebstrap.IXehDNUWIf
>   mke2fs 1.47.0 (5-Feb-2023)
>   mkfs.ext4: Permission denied while trying to determine filesystem size
>   E: setup failed: E: command failed: /sbin/mkfs.ext4 -d "$1/mnt" -L 
> autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 
> 'sid_amd64.img' '25G'
>   W: hooklistener errored out: E: received eof on socket
>   
>   I: main() received signal PIPE: waiting for setup...
>   I: removing tempdir /dev/shm/mmdebstrap.IXehDNUWIf...
>   E: mmdebstrap failed to run
>   mmdebstrap failed
> 
> Does it fail because I do not have enough system memory?
> 
>   $ df --si /dev/shm/
>   Filesystem  Size  Used Avail Use% Mounted on
>   tmpfs   8.3G  108M  8.2G   2% /dev/shm
> 
> Is this the explanation?
> Otherwise, what went wrong?

Interesting! I'm unable to reproduce either of these issues and I'm also
puzzled why you get this "permission denied" error. On my system, I'm able run
your command with the default (in /tmp) as well as in /dev/shm. Here is the
free space each:

$ df --si /tmp /dev/shm
Filesystem  Size  Used Avail Use% Mounted on
tmpfs   8.6G  4.1k  8.6G   1% /tmp
tmpfs   2.0G  418k  2.0G   1% /dev/shm

Maybe something is mounted in a funny way? Both my /tmp and my /dev/shm are
tmpfs. The former is mounted with
rw,nosuid,nodev,relatime,size=8388608k,inode64 and the latter with
rw,nosuid,nodev,inode64.

I doubt that it fails for lack of system memory unless you have less than 3.7
GB of RAM.

Maybe permissions are somehow whacky? Both my /tmp and my /dev/shm are set to
drwxrwxrwt (so there is a sticky bit set) and owned by root:root.

What happens with other locations for TMPDIR?

> By the way, the old script that used guestfish (with all its limitations)
> was able to create a QEMU image in .qcow2 format and my /dev/shm was
> enough for it to work correctly.

It should be enough. Your /dev/shm is 8G large and it works with my 2G.

> Why does the current mmdebstrap-autopkgtest-build-qemu create a QEMU
> image in .img format? Isn't the .qcow2 format better?

Maybe. It's currently .img because it's easier to debug stuff and use kpartx
with it without having to extract it first. :)

> Please clarify and/or improve mmdebstrap-autopkgtest-build-qemu .
> Thanks for your time!

Thanks for your bug! Lets hope we get to the bottom of this.

cheers, josch

signature.asc
Description: signature

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

[Francesco Poli (wintermute)]

Package: mmdebstrap
Version: 1.4.0-1
Severity: normal

Hello,
I am giving mmdebstrap-autopkgtest-build-qemu a try.

The following command fails:

  $ mmdebstrap-autopkgtest-build-qemu --boot=efi sid sid_amd64.img

during some package installation with "no space left on device" error,
since I have /tmp on a somewhat small physical partition:

  $ df --si /tmp/
  Filesystem  Size  Used Avail Use% Mounted on
  /dev/md3868M   99k  806M   1% /tmp

I tried with a TMPDIR in system memory:

  $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
--boot=efi sid sid_amd64.img

but it again fails with the following (final chunk of) output:

  [...]
  cp: warning: behavior of -n is non-portable and may change in future; use 
--update=none instead
  I: running special hook: download vmlinuz '/dev/shm/tmp.uNRUbVgiOu/kernel'
  I: running special hook: download initrd.img '/dev/shm/tmp.uNRUbVgiOu/initrd'
  I: running --customize-hook in shell: sh -c 'mount --bind "$1" "$1/mnt"' exec 
/dev/shm/mmdebstrap.IXehDNUWIf
  I: running --customize-hook in shell: sh -c 'mount --bind "$1/mnt/mnt" 
"$1/mnt/dev"' exec /dev/shm/mmdebstrap.IXehDNUWIf
  I: running --customize-hook in shell: sh -c '/sbin/mkfs.ext4 -d "$1/mnt" -L 
autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 'sid_amd64.img' 
'25G'' exec /dev/shm/mmdebstrap.IXehDNUWIf
  mke2fs 1.47.0 (5-Feb-2023)
  mkfs.ext4: Permission denied while trying to determine filesystem size
  E: setup failed: E: command failed: /sbin/mkfs.ext4 -d "$1/mnt" -L 
autopkgtestvm -E 'offset=134217728,assume_storage_prezeroed=1' 'sid_amd64.img' 
'25G'
  W: hooklistener errored out: E: received eof on socket
  
  I: main() received signal PIPE: waiting for setup...
  I: removing tempdir /dev/shm/mmdebstrap.IXehDNUWIf...
  E: mmdebstrap failed to run
  mmdebstrap failed

Does it fail because I do not have enough system memory?

  $ df --si /dev/shm/
  Filesystem  Size  Used Avail Use% Mounted on
  tmpfs   8.3G  108M  8.2G   2% /dev/shm

Is this the explanation?
Otherwise, what went wrong?

By the way, the old script that used guestfish (with all its limitations)
was able to create a QEMU image in .qcow2 format and my /dev/shm was
enough for it to work correctly.
Why does the current mmdebstrap-autopkgtest-build-qemu create a QEMU
image in .img format? Isn't the .qcow2 format better?

Please clarify and/or improve mmdebstrap-autopkgtest-build-qemu .
Thanks for your time!


-- System Information:
Debian Release: trixie/sid
  APT prefers testing
  APT policy: (800, 'testing'), (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.5.0-5-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mmdebstrap depends on:
ii  apt  2.7.6
ii  perl 5.36.0-10+b1
ii  python3  3.11.4-5+b1

Versions of packages mmdebstrap recommends:
ii  arch-test0.21-1
ii  fakechroot   2.20.1+ds-15
ii  fakeroot 1.32.2-1+b1
ii  gpg  2.2.40-1.1+b1
ii  libdistro-info-perl  1.7
ii  libdpkg-perl 1.22.2
ii  mount2.39.3-2
ii  uidmap   1:4.13+dfsg1-3+b1

Versions of packages mmdebstrap suggests:
pn  apt-transport-tor  
ii  apt-utils  2.7.6
ii  ca-certificates20230311
ii  debootstrap1.0.134
ii  distro-info-data   0.60
ii  dpkg-dev   1.22.2
pn  genext2fs  
ii  perl-doc   5.36.0-10
pn  qemu-user  
pn  qemu-user-static   
pn  squashfs-tools-ng  
ii  systemd255.2-3

-- no debconf information