Bug#1068775: sbuild-qemu: fails to work, if piuparts is requested, but not installed on the host

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Francesco Poli (2024-05-12 00:32:42)
> On Sat, 13 Apr 2024 14:50:05 +0200 Francesco Poli wrote:
> > On Fri, 12 Apr 2024 19:57:23 +0200 Johannes Schauer Marin Rodrigues wrote:
> > > Quoting Francesco Poli (wintermute) (2024-04-11 00:13:51)
> > > > Why does sbuild-qemu insist that piuparts be installed on the *host*
> > > > system?
> > > Because it needs to be installed on the host. In the same way as
> > > autopkgtest needs to be on the host. What can sbuild improve?
> > My point is that sbuild{,-qemu} should install piuparts inside the build
> > environment (chroot or VM guest system) and run it from within the build
> > environment, if this is possible.
> If this is possible for piuparts, could you please explain why sbuild-qemu
> does not do so?
> 
> Otherwise, if this is not possible, could please explain why?

it is possible if somebody writes the code for it.

If you want me to write the code for it, you'd first need to convince me that
it is a good idea to run piuparts inside the build environment as it is done
with lintian.

> > Does sbuild{,-qemu} do so for lintian?
> I checked: sbuild-qemu (temporarily) installs lintian inside the build
> environment and runs it from within the build environment.

Correct.

> I think this is the best strategy, especially for lintian, as I have
> previously explained.

Please refresh my memory: where did you explain why installing piuparts inside
the build environment would be the best strategy?

> That reinforces my opinion that the same strategy should probably be followed
> for piuparts, as well...

What is it that reinforces your opinion? If it is the fact that it is done for
Lintian, I do not see the connection. Why would it be a good idea to do this
for piuparts if it is done for lintian?

> Anyway, I am having issues in running piuparts, even after installing it on
> the host.
> 
> I tried to run
> 
>   $ sbuild-qemu --boot=efi --overlay-dir=/dev/shm
> 
> with the following configuration file:
> 
>   $ cat ~/.sbuildrc 
>   $source_only_changes = 1;
>   $autopkgtest_require_success = 1;
>   $lintian_require_success = 0;
>   $piuparts_require_success = 1;
>   $run_autopkgtest = 1;
>   $run_lintian = 1;
>   $run_piuparts = 1;
>   $build_dir = "$HOME/.cache/sbuild/build";
>   
>   # don't remove this, Perl needs it:
>   1;
> 
> It builds the package successfully, but, once it reaches the piuparts
> step, sudo asks for my (host system) regular user's password.
> My understanding is that it should not need to use sudo, since it should
> use the QEMU virtual machine image (where it has root access).
> Is that right? Or am I misunderstanding something?

This is not a functionality that piuparts has at the moment.

> The same thing happens at the autopkgtest step, where, again, sudo asks for
> my regular user's password. Once again, this should not be needed:
> autopkgtest should use the QEMU virtual machine as a testbed.  At least, when
> I manually run autopkgtest, I can use the QEMU virtual machine and no
> password is asked for.
> 
> I tried to modify the configuration file:
> 
>   $ cat ~/.sbuildrc
>   $source_only_changes = 1;
>   $run_lintian = 1;
>   $lintian_require_success = 0;
>   $run_piuparts = 1;
>   $piuparts_root_args = '';
>   $piuparts_require_success = 1;
>   $run_autopkgtest = 1;
>   $autopkgtest_root_args = '';
>   $autopkgtest_require_success = 1;
>   $build_dir = "$HOME/.cache/sbuild/build";
>   
>   # don't remove this, Perl needs it:
>   1;
> 
> It no longer asks for passwords, but fails to run piuparts:
> 
>   piuparts
>   
> 
>   You need to be root to use piuparts.
> 
>   E: Piuparts run failed.
> 
> And it also fails to run autopkgtest:
> 
>   autopkgtest
>   ---
>   
>   autopkgtest [19:56:04]: starting date and time: 2024-05-11 19:56:04+0200
>   [...]
>   autopkgtest [19:56:05]: test unit_test: preparing testbed
>   autopkgtest [19:56:05]: ERROR: "sh -ec
> type apt-ftparchive >/dev/null 2>&1 || DEBIAN_FRONTEND=noninteractive 
> apt-get install -y apt-utils 2>&1
> (cd /tmp/autopkgtest.jrHchc/binaries; apt-ftparchive packages . > 
> Packages; apt-ftparchive release . > Release)
> printf 'Package: *\nPin: origin ""\nPin-Priority: 1002\n' > 
> /etc/apt/preferences.d/90autopkgtest
> echo "deb [ trusted=yes ] file:///tmp/autopkgtest.jrHchc/binaries /" 
> >/etc/apt/sources.list.d/autopkgtest.list
> if [ "x`ls /var/lib/dpkg/updates`" != x ]; then
>   echo >&2 "/var/lib/dpkg/updates contains some files, aargh"; exit 1
> fi
> apt-get --quiet --no-list-cleanup -o 
> Dir::Etc::sourcelist=/etc/apt/sources.list.d/autopkgtest.list -o 
> Dir::Etc::sourceparts=/dev/null update 2>&1
> cp /var/lib/dpkg/status /tmp/autopkgtest.jrHchc/1-apt-update.out
> " failed with stderr "sh: 4: cannot create 
> /etc/apt/preferences.d/90autopkgtest: Permission denied
>   "
>   autopkgtest [19:56:05]: Binaries: resetting testbed apt configuration
>   Reading package lists...
>   E: Could not open lock 

Bug#1068775: sbuild-qemu: fails to work, if piuparts is requested, but not installed on the host

[Francesco Poli]

On Sat, 13 Apr 2024 14:50:05 +0200 Francesco Poli wrote:

[...]
> On Fri, 12 Apr 2024 19:57:23 +0200 Johannes Schauer Marin Rodrigues
> wrote:
[...] 
> > Quoting Francesco Poli (wintermute) (2024-04-11 00:13:51)
[...]
> > > Why does sbuild-qemu insist that piuparts be installed on the *host*
> > > system?
> > 
> > Because it needs to be installed on the host. In the same way as autopkgtest
> > needs to be on the host. What can sbuild improve?
> 
> My point is that sbuild{,-qemu} should install piuparts inside the build
> environment (chroot or VM guest system) and run it from within the
> build environment, if this is possible.

If this is possible for piuparts, could you please explain why
sbuild-qemu does not do so?

Otherwise, if this is not possible, could please explain why?

> 
> Does sbuild{,-qemu} do so for lintian?
[...]

I checked: sbuild-qemu (temporarily) installs lintian inside the build
environment and runs it from within the build environment.
I think this is the best strategy, especially for lintian, as I have
previously explained.
That reinforces my opinion that the same strategy should probably be
followed for piuparts, as well...


Anyway, I am having issues in running piuparts, even after installing
it on the host.

I tried to run

  $ sbuild-qemu --boot=efi --overlay-dir=/dev/shm

with the following configuration file:

  $ cat ~/.sbuildrc 
  $source_only_changes = 1;
  $autopkgtest_require_success = 1;
  $lintian_require_success = 0;
  $piuparts_require_success = 1;
  $run_autopkgtest = 1;
  $run_lintian = 1;
  $run_piuparts = 1;
  $build_dir = "$HOME/.cache/sbuild/build";
  
  # don't remove this, Perl needs it:
  1;

It builds the package successfully, but, once it reaches the piuparts
step, sudo asks for my (host system) regular user's password.
My understanding is that it should not need to use sudo, since it should
use the QEMU virtual machine image (where it has root access).
Is that right? Or am I misunderstanding something?

The same thing happens at the autopkgtest step, where, again, sudo asks
for my regular user's password. Once again, this should not be needed:
autopkgtest should use the QEMU virtual machine as a testbed.
At least, when I manually run autopkgtest, I can use the QEMU virtual
machine and no password is asked for.

I tried to modify the configuration file:

  $ cat ~/.sbuildrc
  $source_only_changes = 1;
  $run_lintian = 1;
  $lintian_require_success = 0;
  $run_piuparts = 1;
  $piuparts_root_args = '';
  $piuparts_require_success = 1;
  $run_autopkgtest = 1;
  $autopkgtest_root_args = '';
  $autopkgtest_require_success = 1;
  $build_dir = "$HOME/.cache/sbuild/build";
  
  # don't remove this, Perl needs it:
  1;

It no longer asks for passwords, but fails to run piuparts:

  piuparts
  

  You need to be root to use piuparts.

  E: Piuparts run failed.

And it also fails to run autopkgtest:

  autopkgtest
  ---
  
  autopkgtest [19:56:04]: starting date and time: 2024-05-11 19:56:04+0200
  [...]
  autopkgtest [19:56:05]: test unit_test: preparing testbed
  autopkgtest [19:56:05]: ERROR: "sh -ec
type apt-ftparchive >/dev/null 2>&1 || DEBIAN_FRONTEND=noninteractive 
apt-get install -y apt-utils 2>&1
(cd /tmp/autopkgtest.jrHchc/binaries; apt-ftparchive packages . > Packages; 
apt-ftparchive release . > Release)
printf 'Package: *\nPin: origin ""\nPin-Priority: 1002\n' > 
/etc/apt/preferences.d/90autopkgtest
echo "deb [ trusted=yes ] file:///tmp/autopkgtest.jrHchc/binaries /" 
>/etc/apt/sources.list.d/autopkgtest.list
if [ "x`ls /var/lib/dpkg/updates`" != x ]; then
  echo >&2 "/var/lib/dpkg/updates contains some files, aargh"; exit 1
fi
apt-get --quiet --no-list-cleanup -o 
Dir::Etc::sourcelist=/etc/apt/sources.list.d/autopkgtest.list -o 
Dir::Etc::sourceparts=/dev/null update 2>&1
cp /var/lib/dpkg/status /tmp/autopkgtest.jrHchc/1-apt-update.out
" failed with stderr "sh: 4: cannot create 
/etc/apt/preferences.d/90autopkgtest: Permission denied
  "
  autopkgtest [19:56:05]: Binaries: resetting testbed apt configuration
  Reading package lists...
  E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission 
denied)
  E: Unable to lock directory /var/lib/apt/lists/
  [...]
  E: Autopkgtest run failed.

It seems to me that autopkgtest is not using the QEMU virtual machine
as testbed, or am I wrong?


Why isn't all this working out of the box?
Is there any missing setting in ~/.sbuildrc ?
If this is the case, why isn't sbuild-qemu passing those needed options
by default?

Please explain.
Thanks for your time!



-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgp1PXbWEgVnH.pgp
Description: PGP signature

Bug#1068775: sbuild-qemu: fails to work, if piuparts is requested, but not installed on the host

[Richard Lewis]

On Fri, 12 Apr 2024, 19:00 Johannes Schauer Marin Rodrigues, <
jo...@debian.org> wrote:

> Hi Francesco,
>
> Quoting Francesco Poli (wintermute) (2024-04-11 00:13:51)
> >
> >   sbuild --dist unstable --purge-build=never --purge-deps=never
> --chroot-mode=autopkgtest --autopkgtest-virt-server=qemu
> --autopkgtest-virt-server-opt --overlay-dir=/dev/shm
> --autopkgtest-virt-server-opt --qemu-architecture=x86_64
> --autopkgtest-virt-server-opt --ram-size=2048 --autopkgtest-virt-server-opt
> --cpus=2 --autopkgtest-virt-server-opt --boot=efi
> --autopkgtest-virt-server-opt
> /home/$USER/.cache/sbuild/unstable-autopkgtest-amd64.img
> --bd-uninstallable-explainer apt
> > Error reading configuration: PIUPARTS binary 'piuparts' does not exist
> or is not executable at /usr/share/perl5/Sbuild/Conf.pm line 76.
> >
> >



> > However, I thought that setting $run_piuparts = 1 in ~/.sbuildrc caused
> > piuparts to be run inside the virtual machine guest system, not on the
> > host system!
>
> What made you think that? Is the error message above not clear enough? Do
> you
> think it should be improved to make things clearer? Do you have
> suggestions for
> a message that would've better explained that piuparts needs to be
> installed on
> the host system?
>

Adding a data point, i would also like the error message to be clearer - i
know that once you understand how sbuild and piuparts work it becomes
"obvious" that piuparts needs to be run from the host -- but this is not at
all clear for new users: would be great if the documentation and errors
could clearly distinguish between which things inside/outside the chroot


This particular error message could be improved in at least 3 ways:
1. "Error reading configuration" -- suggests it couldnt parse the config
file, which doesnt seem to be the case here (suggest either dropping this
from the error message or saying more clearly what the issue is)

2. "PIUPARTS binary 'piuparts' does not exist or is not executable"

not sure why the capitials and repetition?  just say more directly what the
issue is: eg : "Error: piuparts not found (it needs to be installed on the
host)"

3.  "at /usr/share/perl5/Sbuild/Conf.pm line 76." -- not sure this
implementation detail is helpful for a user -- it makes it look a bug
rather than a message for the user: could it be dropped from the error
message or hidden behind a "debug"/"verbose" setting?


thanks for working on sbuild, it's really great.

Bug#1068775: sbuild-qemu: fails to work, if piuparts is requested, but not installed on the host

[Francesco Poli]

Control: severity -1 wishlist


On Fri, 12 Apr 2024 19:57:23 +0200 Johannes Schauer Marin Rodrigues
wrote:

[...]
> Quoting Francesco Poli (wintermute) (2024-04-11 00:13:51)
[...]
> > I can update the virtual machine (if I create a symlink, see bug 
> > [#1061816]):
> >
> >   $ sbuild-qemu-update --boot=efi unstable-autopkgtest-amd64.img
> > 
> > [#1061816]: 
> 
> I think 1061816 was fixed with 0.85.7 and the changelog was just missing the
> "closes" entry:
> 
> https://tracker.debian.org/news/1518576/accepted-sbuild-0857-source-into-unstable/

Yes, I have just tested sbuild-qemu/0.85.7 from unstable, and I can
confirm that the symlink is no longer needed.
I have just sent a message to close bug report #1061816 ...

> 
> > But, when I try to build a package from withing the unpacked source
> > tree:
[...]
> > Error reading configuration: PIUPARTS binary 'piuparts' does not exist or 
> > is not executable at /usr/share/perl5/Sbuild/Conf.pm line 76.
> > 
> > Now, piuparts is indeed not installed on the host:
[...]
> > However, I thought that setting $run_piuparts = 1 in ~/.sbuildrc caused
> > piuparts to be run inside the virtual machine guest system, not on the
> > host system!
> 
> What made you think that? Is the error message above not clear enough? Do you
> think it should be improved to make things clearer? Do you have suggestions 
> for
> a message that would've better explained that piuparts needs to be installed 
> on
> the host system?

My point is not that the error message should be improved.
Actually, I think the error message clearness is basically OK.

> 
> > Hence, I thought that piuparts was going to be automatically installed
> > and executed on the guest system (actually on the throw-away overlay).
> > And I thought that the same was going to happen for $run_autopkgtest = 1
> > and for $run_lintian = 1 ...
> > 
> > Am I completely off-track?
> > What did I fail to understand?
> > 
> > Why does sbuild-qemu insist that piuparts be installed on the *host*
> > system?
> 
> Because it needs to be installed on the host. In the same way as autopkgtest
> needs to be on the host. What can sbuild improve?

My point is that sbuild{,-qemu} should install piuparts inside the build
environment (chroot or VM guest system) and run it from within the
build environment, if this is possible.

Does sbuild{,-qemu} do so for lintian? I think this is even more
important for lintian: if the host system runs Debian testing (or maybe
even Debian stable) and the package is being built for Debian unstable
(as upload target), then the package should be checked by the version
of lintian which is currently in Debian unstable. If it were checked by
the version of lintian from Debian testing (or even from Debian
stable), many undeserved "unknown Policy version" complaints could pop
up...

Do you agree with the reasoning?

Thanks for your time and patience!   :-)

-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpli10LtTDY8.pgp
Description: PGP signature

Bug#1068775: sbuild-qemu: fails to work, if piuparts is requested, but not installed on the host

[Johannes Schauer Marin Rodrigues]

Hi Francesco,

Quoting Francesco Poli (wintermute) (2024-04-11 00:13:51)
> I am trying to set up sbuild-qemu to build (and test) Debian packages.
> 
> After creating the virtual machine image:
> 
>   $ mkdir -p ~/.cache/sbuild/build
>   $ cd /dev/shm
>   $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
> --size=25G --boot=efi sid unstable-autopkgtest-amd64.img
>   $ mv -i unstable-autopkgtest-amd64.img ~/.cache/sbuild/
> 
> I prepared the following configuration file:
> 
>   $ cat ~/.sbuildrc 
>   $source_only_changes = 1;
>   $autopkgtest_require_success = 1;
>   $lintian_require_success = 0;
>   $piuparts_require_success = 1;
>   $run_autopkgtest = 1;
>   $run_lintian = 1;
>   $run_piuparts = 1;
>   $build_dir = "$HOME/.cache/sbuild/build";
>   
>   # don't remove this, Perl needs it:
>   1;
> 
> I can update the virtual machine (if I create a symlink, see bug [#1061816]):
>
>   $ sbuild-qemu-update --boot=efi unstable-autopkgtest-amd64.img
> 
> [#1061816]: 

I think 1061816 was fixed with 0.85.7 and the changelog was just missing the
"closes" entry:

https://tracker.debian.org/news/1518576/accepted-sbuild-0857-source-into-unstable/

> But, when I try to build a package from withing the unpacked source
> tree:
> 
>   $ sbuild-qemu --boot=efi --overlay-dir=/dev/shm
>   sbuild --dist unstable --purge-build=never --purge-deps=never 
> --chroot-mode=autopkgtest --autopkgtest-virt-server=qemu 
> --autopkgtest-virt-server-opt --overlay-dir=/dev/shm 
> --autopkgtest-virt-server-opt --qemu-architecture=x86_64 
> --autopkgtest-virt-server-opt --ram-size=2048 --autopkgtest-virt-server-opt 
> --cpus=2 --autopkgtest-virt-server-opt --boot=efi 
> --autopkgtest-virt-server-opt 
> /home/$USER/.cache/sbuild/unstable-autopkgtest-amd64.img 
> --bd-uninstallable-explainer apt
> Error reading configuration: PIUPARTS binary 'piuparts' does not exist or is 
> not executable at /usr/share/perl5/Sbuild/Conf.pm line 76.
> 
> Now, piuparts is indeed not installed on the host:
> 
>   $ apt policy piuparts
>   piuparts:
> Installed: (none)
> Candidate: 1.4.1
> Version table:
>1.4.1 800
>   800 http://deb.debian.org/debian testing/main amd64 Packages
>   500 http://deb.debian.org/debian unstable/main amd64 Packages
> 
> However, I thought that setting $run_piuparts = 1 in ~/.sbuildrc caused
> piuparts to be run inside the virtual machine guest system, not on the
> host system!

What made you think that? Is the error message above not clear enough? Do you
think it should be improved to make things clearer? Do you have suggestions for
a message that would've better explained that piuparts needs to be installed on
the host system?

> Hence, I thought that piuparts was going to be automatically installed
> and executed on the guest system (actually on the throw-away overlay).
> And I thought that the same was going to happen for $run_autopkgtest = 1
> and for $run_lintian = 1 ...
> 
> Am I completely off-track?
> What did I fail to understand?
> 
> Why does sbuild-qemu insist that piuparts be installed on the *host*
> system?

Because it needs to be installed on the host. In the same way as autopkgtest
needs to be on the host. What can sbuild improve?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1068775: sbuild-qemu: fails to work, if piuparts is requested, but not installed on the host

[Francesco Poli (wintermute)]

Package: sbuild-qemu
Version: 0.85.6
Severity: normal
X-Debbugs-Cc: invernom...@paranoici.org

Hi!

I am trying to set up sbuild-qemu to build (and test) Debian packages.

After creating the virtual machine image:

  $ mkdir -p ~/.cache/sbuild/build
  $ cd /dev/shm
  $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
--size=25G --boot=efi sid unstable-autopkgtest-amd64.img
  $ mv -i unstable-autopkgtest-amd64.img ~/.cache/sbuild/

I prepared the following configuration file:

  $ cat ~/.sbuildrc 
  $source_only_changes = 1;
  $autopkgtest_require_success = 1;
  $lintian_require_success = 0;
  $piuparts_require_success = 1;
  $run_autopkgtest = 1;
  $run_lintian = 1;
  $run_piuparts = 1;
  $build_dir = "$HOME/.cache/sbuild/build";
  
  # don't remove this, Perl needs it:
  1;

I can update the virtual machine (if I create a symlink, see bug [#1061816]):

  $ sbuild-qemu-update --boot=efi unstable-autopkgtest-amd64.img

[#1061816]: 

But, when I try to build a package from withing the unpacked source
tree:

  $ sbuild-qemu --boot=efi --overlay-dir=/dev/shm
  sbuild --dist unstable --purge-build=never --purge-deps=never 
--chroot-mode=autopkgtest --autopkgtest-virt-server=qemu 
--autopkgtest-virt-server-opt --overlay-dir=/dev/shm 
--autopkgtest-virt-server-opt --qemu-architecture=x86_64 
--autopkgtest-virt-server-opt --ram-size=2048 --autopkgtest-virt-server-opt 
--cpus=2 --autopkgtest-virt-server-opt --boot=efi --autopkgtest-virt-server-opt 
/home/$USER/.cache/sbuild/unstable-autopkgtest-amd64.img 
--bd-uninstallable-explainer apt
Error reading configuration: PIUPARTS binary 'piuparts' does not exist or is 
not executable at /usr/share/perl5/Sbuild/Conf.pm line 76.

Now, piuparts is indeed not installed on the host:

  $ apt policy piuparts
  piuparts:
Installed: (none)
Candidate: 1.4.1
Version table:
   1.4.1 800
  800 http://deb.debian.org/debian testing/main amd64 Packages
  500 http://deb.debian.org/debian unstable/main amd64 Packages

However, I thought that setting $run_piuparts = 1 in ~/.sbuildrc caused
piuparts to be run inside the virtual machine guest system, not on the
host system!
Hence, I thought that piuparts was going to be automatically installed
and executed on the guest system (actually on the throw-away overlay).
And I thought that the same was going to happen for $run_autopkgtest = 1
and for $run_lintian = 1 ...

Am I completely off-track?
What did I fail to understand?

Why does sbuild-qemu insist that piuparts be installed on the *host*
system?

Please clarify and/or fix this issue.
Thanks for your kind assistance!


-- System Information:
Debian Release: trixie/sid
  APT prefers testing
  APT policy: (800, 'testing'), (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.6.15-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages sbuild-qemu depends on:
ii  autopkgtest  5.34
ii  python3  3.11.6-1
ii  python3-pexpect  4.9-2
ii  python3-psutil   5.9.8-2
ii  qemu-system-x86  1:8.2.1+ds-2
ii  qemu-utils   1:8.2.1+ds-2
ii  sbuild   0.85.6
ii  vmdb20.28-2

Versions of packages sbuild-qemu recommends:
ii  qemu-system-arm [qemu-system-arm]  1:8.2.1+ds-2
ii  qemu-system-ppc [qemu-system-ppc]  1:8.2.1+ds-2

sbuild-qemu suggests no packages.

-- no debconf information