Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Julien Cristau]

On Thu, Sep  8, 2016 at 13:35:51 +0200, Lucas Nussbaum wrote:

> 1) packages failing to build when gnupg is not installed in the chroot.
> gnupg is priority: important, and is not installed by debootstrap
> --variant=buildd.
> 
> 2) packages failing to build when tzdata is not installed in the chroot.
> tzdata is priority: required, and it is installed by debootstrap
> --variant=buildd. But since it is not essential, not build-essential,
> and not a dependency of essential or build-essential packages, it can
> safely be removed (e.g. with debfoster -o MaxPriority=required).  I
> count about 150 packages failing in that case.
> 
> Both classes of bugs are valid bugs. However, I'm wondering if it's OK
> to consider both classes as release critical.
> 
I think they should both be serious, and tzdata's priority should be
fixed up by ftpmaster.

Cheers,
Julien

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Lucas Nussbaum]

On 08/09/16 at 11:31 +0200, Santiago Vila wrote:
> On Thu, 8 Sep 2016, Johannes Schauer wrote:
> 
> > as we are talking about testing packages in the most minimal environment
> > possible it must be noted that debootstrap --variant=minbase or
> > --variant=buildd does not only install Essential:yes or Essential:yes with
> > build-essential, respectively, (and their transitive dependencies of course)
> > but also Priority:required packages like lsb-base or tzdata which are not
> > depended upon by any Essential:yes package.
> 
> This is indeed the case in stretch and sid.
> 
> (In jessie, util-linux had a Depends on both lsb-base and tzdata).
> 
> > Thus, with the current situation, there could exist many source packages 
> > that
> > should explicitly build-depend on Priority:required packages but don't and 
> > were
> > never found so far because all build chroots created by debootstrap include
> > these packages already.
> 
> In theory, yes, there could exist many source packages with hidden
> missing build-dependencies on packages installed by default.
> 
> In practice, I don't think there are so many. In the gnupg case, for
> example, I only found 3 or 4 packages among the set of 15000 packages
> generating "Arch: all" binary packages.
> 
> But in either case I'm going to remove lsb-base and tzdata from my
> stretch chroots and see what happens. Are there more required packages
> in stretch that should be removed after using the buildd variant?
> 
> (I don't really care about minbase, as I think it is more oriented
> towards using it in debian-installer and similar scenarios to create
> actually bootable and usable systems).
> 
> > Maybe a bug against debootstrap is in order that adds a new variant 
> > allowing it
> > to create a more minimal chroot?
> 
> Yes, please, but we don't really need a new variant, we just need a
> buildd variant that honors the build-essential definition in policy
> (which does not include required packages as such, only essential and
> build essential packages).
> 
> Thanks.

Dear Release Team,

Could you please advice on what to do about the severity of those two
classes of bugs.

1) packages failing to build when gnupg is not installed in the chroot.
gnupg is priority: important, and is not installed by debootstrap
--variant=buildd.

2) packages failing to build when tzdata is not installed in the chroot.
tzdata is priority: required, and it is installed by debootstrap
--variant=buildd. But since it is not essential, not build-essential,
and not a dependency of essential or build-essential packages, it can
safely be removed (e.g. with debfoster -o MaxPriority=required).  I
count about 150 packages failing in that case.

Both classes of bugs are valid bugs. However, I'm wondering if it's OK
to consider both classes as release critical.

Thanks,

Lucas

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Santiago Vila]

On Thu, 8 Sep 2016, Johannes Schauer wrote:

> as we are talking about testing packages in the most minimal environment
> possible it must be noted that debootstrap --variant=minbase or
> --variant=buildd does not only install Essential:yes or Essential:yes with
> build-essential, respectively, (and their transitive dependencies of course)
> but also Priority:required packages like lsb-base or tzdata which are not
> depended upon by any Essential:yes package.

This is indeed the case in stretch and sid.

(In jessie, util-linux had a Depends on both lsb-base and tzdata).

> Thus, with the current situation, there could exist many source packages that
> should explicitly build-depend on Priority:required packages but don't and 
> were
> never found so far because all build chroots created by debootstrap include
> these packages already.

In theory, yes, there could exist many source packages with hidden
missing build-dependencies on packages installed by default.

In practice, I don't think there are so many. In the gnupg case, for
example, I only found 3 or 4 packages among the set of 15000 packages
generating "Arch: all" binary packages.

But in either case I'm going to remove lsb-base and tzdata from my
stretch chroots and see what happens. Are there more required packages
in stretch that should be removed after using the buildd variant?

(I don't really care about minbase, as I think it is more oriented
towards using it in debian-installer and similar scenarios to create
actually bootable and usable systems).

> Maybe a bug against debootstrap is in order that adds a new variant allowing 
> it
> to create a more minimal chroot?

Yes, please, but we don't really need a new variant, we just need a
buildd variant that honors the build-essential definition in policy
(which does not include required packages as such, only essential and
build essential packages).

Thanks.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Johannes Schauer]

Hi,

Quoting Thorsten Glaser (2016-09-07 22:53:37)
> Markus Koschany dixit:
> 
> >I have just created a new cowbuilder base chroot with
> >
> >sudo DIST=sid ARCH=amd64 cowbuilder --create
> >
> >and this command still installs gnupg. I don't know what I'm currently
> >missing
> 
> --variant=minbase or --variant=buildd (minbase+build-essential)

as we are talking about testing packages in the most minimal environment
possible it must be noted that debootstrap --variant=minbase or
--variant=buildd does not only install Essential:yes or Essential:yes with
build-essential, respectively, (and their transitive dependencies of course)
but also Priority:required packages like lsb-base or tzdata which are not
depended upon by any Essential:yes package.

Thus, with the current situation, there could exist many source packages that
should explicitly build-depend on Priority:required packages but don't and were
never found so far because all build chroots created by debootstrap include
these packages already.

Maybe a bug against debootstrap is in order that adds a new variant allowing it
to create a more minimal chroot?

Personally, I so far used multistrap to create my chroots, which doesn't suffer
from this problem.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[James Clarke]

> On 7 Sep 2016, at 23:01, Markus Koschany  wrote:
> 
> On 07.09.2016 23:24, Mattia Rizzolo wrote:
>> Pbuilder (and therefore cowbuilder) already use --variant=buildd
>> 
>> That afaik is --variant=minbase + build-essential.
>> 
>> I'm not sure why you still get gnupg installed.
>> 
>> Btw what version are you using (of cowbuilder and pbuilder)?
> 
> I'm using the latest packages from sid (cowbuilder 0.80, pbuilder
> 0.226). I'm not sure but I believe exporting
> DEBOOTSTRAPOPTS=--variant=buildd solved the issue for me. GnuPG is no
> longer installed by default. I'm using a slightly modified version of
> Ubuntu's .pbuilderrc which I have once copied from

--variant=buildd is the default; if not, that’s your pbuilderrc (either
/etc/pbuilderrc or ~/.pbuilderrc) at fault.

James

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Santiago Vila dixit:

>For the record: This is a false dichotomy, because xmlgraphics-commons
>didn't fail in the official buildds. it didn't fail in a

You miss a word there: “yet”

An undeclared build dependency on a non-Essential/Build-Essential
package is an RC bug in the package. That includes when needed
packages were previously provided by other build dependencies (by
virtue of being run-time dependencies of such) but no longer are.

bye,
//mirabilos
-- 
“ah that reminds me, thanks for the stellar entertainment that you and certain
other people provide on the Debian mailing lists │ sole reason I subscribed to
them (I'm not using Debian anywhere) is the entertainment factor │ Debian does
not strike me as a place for good humour, much less German admin-style humour”

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Santiago Vila]

On Wed, 7 Sep 2016, Markus Koschany wrote:

> Now we all agree that a package that fails to build
> on the official buildd network is RC buggy. But what about packages that
> neither fail there nor locally in a clean cowbuilder environment but
> under some obscure circumstances in a local sbuild environment?

For the record: This is a false dichotomy, because xmlgraphics-commons
didn't fail in the official buildds. it didn't fail in a
what-you-call-clean-but-it-was-not-really-clean cowbuilder
environment, but the circumstances in which it failed were not
obscrure at all (as explained).

It seems to me that you are looking for a generic passport to
downgrade FTBFS bugs. Please don't put policy violations and
bugs that we still don't know how to reproduce in the same bag.

Thanks.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Markus Koschany]

On 07.09.2016 23:24, Mattia Rizzolo wrote:
> Pbuilder (and therefore cowbuilder) already use --variant=buildd
> 
> That afaik is --variant=minbase + build-essential.
> 
> I'm not sure why you still get gnupg installed.
> 
> Btw what version are you using (of cowbuilder and pbuilder)?

I'm using the latest packages from sid (cowbuilder 0.80, pbuilder
0.226). I'm not sure but I believe exporting
DEBOOTSTRAPOPTS=--variant=buildd solved the issue for me. GnuPG is no
longer installed by default. I'm using a slightly modified version of
Ubuntu's .pbuilderrc which I have once copied from

https://wiki.ubuntu.com/PbuilderHowto

I'll check again tomorrow if gnupg will be installed without passing
this option to pbuilder. It may well be that I did something wrong a few
hours ago.




signature.asc
Description: OpenPGP digital signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Mattia Rizzolo dixit:

>I'm not sure why you still get gnupg installed.
>
>Btw what version are you using (of cowbuilder and pbuilder)?

I just installed (on an up-to-date sid/x32 system with latest
cowbuilder/pbuilder) a fresh sid chroot (from a local mirror
that’s updated daily) and get no gnupg. The full package list:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name   Version  Architecture Description
+++-==---=
ii  adduser3.115all  add and remove 
users and groups
ii  apt1.3~rc4  i386 commandline 
package manager
ii  aptitude   0.8.3-1  i386 terminal-based 
package manager
ii  aptitude-common0.8.3-1  all  architecture 
independent files for the aptitude p
ii  base-files 9.6  i386 Debian base system 
miscellaneous files
ii  base-passwd3.5.40   i386 Debian base system 
master password and group file
ii  bash   4.3-15   i386 GNU Bourne Again 
SHell
ii  binutils   2.27-8   i386 GNU assembler, 
linker and binary utilities
ii  bsdutils   1:2.28.1-1   i386 basic utilities 
from 4.4BSD-Lite
ii  build-essential12.2 i386 Informational list 
of build-essential packages
ii  bzip2  1.0.6-8  i386 high-quality 
block-sorting file compressor - util
ii  coreutils  8.25-2   i386 GNU core utilities
ii  cowdancer  0.80 i386 Copy-on-write 
directory tree utility
ii  cpp4:6.1.1-1i386 GNU C preprocessor 
(cpp)
ii  cpp-6  6.2.0-3  i386 GNU C preprocessor
ii  dash   0.5.8-2.3i386 POSIX-compliant 
shell
ii  debconf1.5.59   all  Debian 
configuration management system
ii  debian-archive-keyring 2014.3   all  GnuPG archive keys 
of the Debian archive
ii  debianutils4.8  i386 Miscellaneous 
utilities specific to Debian
ii  diffutils  1:3.5-1  i386 File comparison 
utilities
ii  dpkg   1.18.10  i386 Debian package 
management system
ii  dpkg-dev   1.18.10  all  Debian package 
development tools
ii  e2fslibs:i386  1.43.3-1 i386 ext2/ext3/ext4 
file system libraries
ii  e2fsprogs  1.43.3-1 i386 ext2/ext3/ext4 
file system utilities
ii  eatmydata  105-3all  Library and 
utilities designed to disable fsync a
ii  fakeroot   1.21-2   i386 tool for 
simulating superuser privileges
ii  findutils  4.6.0+git+201607 i386 utilities for 
finding files--find, xargs
ii  g++4:6.1.1-1i386 GNU C++ compiler
ii  g++-6  6.2.0-3  i386 GNU C++ compiler
ii  gcc4:6.1.1-1i386 GNU C compiler
ii  gcc-4.9-base:i386  4.9.4-2  i386 GCC, the GNU 
Compiler Collection (base package)
ii  gcc-5-base:i3865.4.1-2  i386 GCC, the GNU 
Compiler Collection (base package)
ii  gcc-6  6.2.0-3  i386 GNU C compiler
ii  gcc-6-base:i3866.2.0-3  i386 GCC, the GNU 
Compiler Collection (base package)
ii  gpgv   2.1.15-2 i386 GNU privacy guard 
- signature verification tool
ii  grep   2.25-6   i386 GNU grep, egrep 
and fgrep
ii  gzip   1.6-5i386 GNU compression 
utilities
ii  hostname   3.18 i386 utility to 
set/show the host name or domain name
ii  init-system-helpers1.42 all  helper tools for 
all init systems
ii  initscripts2.88dsf-59.8 i386 scripts for 
initializing and shutting down the sy
ii  insserv1.14.0-5.4   i386 boot sequence 
organizer using LSB init.d script d
ii  libacl1:i386   2.2.52-3 i386 Access control 
list shared library
ii  libapt-pkg5.0:i386 1.3~rc4  i386 package management 
runtime library
ii  libasan3:i386  6.2.0-3  i386 AddressSanitizer 
-- a fast memory error detector
ii  

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Mattia Rizzolo]

Pbuilder (and therefore cowbuilder) already use --variant=buildd

That afaik is --variant=minbase + build-essential.

I'm not sure why you still get gnupg installed.

Btw what version are you using (of cowbuilder and pbuilder)?

On Wed, 7 Sep 2016, 11:19 p.m. Thorsten Glaser,  wrote:

> Markus Koschany dixit:
>
> >Thanks. Could we make these variants the default in cowbuilder?
>
> Unsure, people *can* use cowbuilder to create normal chroots
> for use, just like schroot.
>
> One thing we can do is to document this better in the cowbuilder
> manpage (currently only in pbuilder’s at all, and IMHO it belongs
> mentioned in the --create option).
>
> Or add a --variant to both pbuilder and cowbuilder that gets
> mapped to --debootstraptoptions --variant, and make --create
> error out if it’s not given.
>
> One more problem with this is that alternative debootstraps
> can be used, which may not have this option. So if we do the
> latter, it can only fire with --debootstrap debootstrap (or
> some variant of it… remember it’s just a script that can be
> renamed…).
>
> bye,
> //mirabilos
> --
> Stéphane, I actually don’t block Googlemail, they’re just too utterly
> stupid to successfully deliver to me (or anyone else using Greylisting
> and not whitelisting their ranges). Same for a few other providers such
> as Hotmail. Some spammers (Yahoo) I do block.
>

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Santiago Vila]

On Wed, 7 Sep 2016, Markus Koschany wrote:

> But what about packages that
> neither fail there nor locally in a clean cowbuilder environment but
> under some obscure circumstances in a local sbuild environment?

Excuse me? A chroot without gnupg is now called "obscure circumstances
in a local sbuild environment"? I find such expression sligthly
offensive and unrespectful, even more than raising once the severity
of the bug to match current practice (with a reminder of what current
practice is).

I said the way to reproduce it from the very beginning: A chroot
without gnupg. The patch clearly added gnupg to build-depends.
The error message said 'Cannot run program "gpg"'.

But now this is "obscure circumstances", as if I had not given enough
information to reproduce the bug.

The only thing I didn't do was to attach a full build log, because I
naively thought that the error message, the patch, the bug title and
the explanation was more than enough to reproduce the bug.

Why you apparently refused to use all this information when trying to
reproduce the bug is still a mystery to me.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Markus Koschany dixit:

>Thanks. Could we make these variants the default in cowbuilder?

Unsure, people *can* use cowbuilder to create normal chroots
for use, just like schroot.

One thing we can do is to document this better in the cowbuilder
manpage (currently only in pbuilder’s at all, and IMHO it belongs
mentioned in the --create option).

Or add a --variant to both pbuilder and cowbuilder that gets
mapped to --debootstraptoptions --variant, and make --create
error out if it’s not given.

One more problem with this is that alternative debootstraps
can be used, which may not have this option. So if we do the
latter, it can only fire with --debootstrap debootstrap (or
some variant of it… remember it’s just a script that can be
renamed…).

bye,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Markus Koschany]

On 07.09.2016 22:53, Thorsten Glaser wrote:
> Markus Koschany dixit:
> 
>> I have just created a new cowbuilder base chroot with
>>
>> sudo DIST=sid ARCH=amd64 cowbuilder --create
>>
>> and this command still installs gnupg. I don't know what I'm currently
>> missing
> 
> --variant=minbase or --variant=buildd (minbase+build-essential)
> 
> TYS,
> //mirabilos

Thanks. Could we make these variants the default in cowbuilder?

Cheers,

Markus



signature.asc
Description: OpenPGP digital signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Markus Koschany dixit:

>I have just created a new cowbuilder base chroot with
>
>sudo DIST=sid ARCH=amd64 cowbuilder --create
>
>and this command still installs gnupg. I don't know what I'm currently
>missing

--variant=minbase or --variant=buildd (minbase+build-essential)

TYS,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Markus Koschany]

On 07.09.2016 16:40, gregor herrmann wrote:
> On Wed, 07 Sep 2016 16:05:24 +0200, Johannes Schauer wrote:
> 
>>> The package xmlgraphics-commons started recently failing to build from
>>> source in a clean sbuild environment although it was built successfully
>>> on the buildd network a few months ago. This behavior cannot be observed
>>> in a clean cowbuilder environment though. [1]
> 
>>> 3. or if cowbuilder should not install gnupg by default
>> I think it should not because I think that source packages should be compiled
>> in an environment that is as minimal as possible for the reasons given above.
>> But of course this is up to the cowbuilder maintainers.
> 
> FWIW, I don't have a /usr/bin/gpg in my {amd64,i386} {stretch,sid}
> cowbuilder chroots. But they are present in older ones.
> 
> I suspect I don't have them anymore because I clean my chroots (with
> debfoster), and others might still have gnupg installed because it
> was originally installed but never removed.
> 
> (I haven't tried, but I guess creating a new cowbuilder chroot won't
> have gnupg either.) 

Hi,

I have just created a new cowbuilder base chroot with

sudo DIST=sid ARCH=amd64 cowbuilder --create

and this command still installs gnupg. I don't know what I'm currently
missing but this is the issue that I would like to get resolved with
this bug report. In my opinion cowbuilder and sbuild should always
install only the minimum build requirements by default which are
demanded by Policy and the tools should behave identically. That means
no matter which tool I use the same package should build or fail in both
chroots.

Quoting Johannes Schauer
> indeed, there already is a definition of the "common build environment
> standard" and that definition is given by the relationships between packages 
> in
> the archive and the implicit dependencies on Essential:yes and 
> build-essential,
> just as given in the section of Debian policy that you described.
> 
> I would be interested to hear what Markus means by defining a "common build
> environment standard" because then I'd like to present reasons why our current
> standard (Essential:yes plus build-essential plus Build-Depends minus
> Build-Conflicts) is an excellent way to define the set.

I was talking about the build environment tools like cowbuilder,
pbuilder and sbuild which are actually used in practice for building
packages in a commonly called "clean" environment. By using the word
"clean" I refer to the same properties that you have described as
installing Essential: yes and build-essential packages. Everything else
is just the defined build-dependencies in debian/control. I completely
agree with that.

My point is that I have come across a lot of bug reports in the past
that either claim the build failed in a clean cowbuilder or sbuild
environment or both. Now we all agree that a package that fails to build
on the official buildd network is RC buggy. But what about packages that
neither fail there nor locally in a clean cowbuilder environment but
under some obscure circumstances in a local sbuild environment? Is this
automatically release critical because the package FTBFS or do we accept
that not every FTBFS is automatically RC and that there might be corner
cases, so rare that they should not be deemed "serious", "grave" or
"critical" but just to be normal issues?

#834682 [1] is an example of such a dilemma. For me this package builds
fine in cowbuilder and sbuild but it fails for Santiago. Since I can't
reproduce the issue I have downgraded the severity and marked the bug as
"unreproducible". Now I don't want to allege that this bug is some kind
of imagination but I'm responsible as the maintainer to determine the
severity of a bug report. Just ignoring the bug report and waiting for
the autoremoval can't really be a solution. I personally find it rude
and disrespectful if Santiago immediately raises the severity again and
claims that each and every one of his local FTBFS bugs is always RC.

Thus I have used the phrase "common build environment standard" because
we need _tools_ that implement the Policy, are reliable and deliver a
reproducible and verifiable output. I would rather like to see that we
require only one tool for building packages before we upload them
"source-only" into the archive which would certainly bring synergy
effects. Then we all would have less to worry about FTBFS bug reports in
the future.

And regarding #834744 [2] I have never disputed that this is a bug in
xmlgraphics-commons despite the fact that the package builds fine in a
clean cowbuilder environment. My point is that we should be more careful
when we assign severities to bug reports because there might be other
solutions like the ones I have mentioned in my initial bug report. Now
we know that this behavioral change was intended and there is actually
some kind of bug in cowbuilder that "tricked" us into believing gnupg
was part of a default clean environment. Adding gnupg to Build-Depends

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Mattia Rizzolo]

On Wed, Sep 07, 2016 at 06:33:06PM +, Thorsten Glaser wrote:
> Johannes Schauer dixit:
> 
> >APT::AutoRemove::RecommendsImportant "false";
> >
> >This is what sbuild does and how it will remove gnupg on chroot upgrades.
> 
> That’s useful for cowbuilder, indeed!

pbuilder (and therefor cowbuilder) has that option since 0.222.

Though I just realized that with the way I added it (just writing them
in /etc/apt/apt.conf.d/15pbuilder, where there used to be only
'APT::Install-Recommends "false";' back in time) won't be applied unless
the --override-config is used while doing the update.

I wonder whether I should be more enforcing of the default APT opts and
pass them to every apt-get call as opposed as just using the config
file.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Johannes Schauer]

Hi,

Quoting Santiago Vila (2016-09-07 20:33:54)
> Moreover, Markus suggested that I work towards "defining a common
> build environment standard". Not sure what he meant by that. Do we
> need such standard or we can still use the already existing set of
> build essential packages?
> 
> I would prefer not to bother the release managers or the technical
> committee about this if anybody here could convince Markus that Bug #834744
> should be serious (not just because of policy but also because that's
> how we usually file FTBFS bugs that happen because of missing build-depends).

indeed, there already is a definition of the "common build environment
standard" and that definition is given by the relationships between packages in
the archive and the implicit dependencies on Essential:yes and build-essential,
just as given in the section of Debian policy that you described.

I would be interested to hear what Markus means by defining a "common build
environment standard" because then I'd like to present reasons why our current
standard (Essential:yes plus build-essential plus Build-Depends minus
Build-Conflicts) is an excellent way to define the set.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Johannes Schauer dixit:

>APT::AutoRemove::RecommendsImportant "false";
>
>This is what sbuild does and how it will remove gnupg on chroot upgrades.

That’s useful for cowbuilder, indeed!

Thanks,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Mattia Rizzolo dixit:

>Given this, and other instance (like the presence of all ggc-X(.Y)-base
>binaries that are installed because of their priority), I recommend to
>recreate the building chroots from time to time, to take account changes
>in that set.

Agreed, except I do manual cleanup instead because recreating does
NOT save you from stray gcc-X.Y-base packages: they’re kept in the
archive for as long as they’re still used, or (in the debian-ports
case) until the porters request aurel32 to manually remove them(!)
which can… take a while.

bye,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Johannes Schauer]

Hi,

Quoting Mattia Rizzolo (2016-09-07 19:54:10)
> On Wed, Sep 07, 2016 at 04:40:52PM +0200, gregor herrmann wrote:
> > On Wed, 07 Sep 2016 16:05:24 +0200, Johannes Schauer wrote:
> > 
> > > > The package xmlgraphics-commons started recently failing to build from
> > > > source in a clean sbuild environment although it was built successfully
> > > > on the buildd network a few months ago. This behavior cannot be observed
> > > > in a clean cowbuilder environment though. [1]
> > 
> > > > 3. or if cowbuilder should not install gnupg by default
> > > I think it should not because I think that source packages should be 
> > > compiled
> > > in an environment that is as minimal as possible for the reasons given 
> > > above.
> > > But of course this is up to the cowbuilder maintainers.
> > 
> > FWIW, I don't have a /usr/bin/gpg in my {amd64,i386} {stretch,sid}
> > cowbuilder chroots. But they are present in older ones.
> > 
> > I suspect I don't have them anymore because I clean my chroots (with
> > debfoster), and others might still have gnupg installed because it
> > was originally installed but never removed.
> > 
> > (I haven't tried, but I guess creating a new cowbuilder chroot won't
> > have gnupg either.) 
> 
> The reason some people still have gnupg installed in old chroots but not
> newer is because `apt-get autoremove` won't remove packages that are:
>  * marked as installed automatically  AND
>  * with no reverse-dependencies (anymore) AND
>  * with reverse-recommends.

have a look at the options that sbuild passes to apt. You can change the
default behaviour you describe by adding:

APT::AutoRemove::RecommendsImportant "false";

This is what sbuild does and how it will remove gnupg on chroot upgrades.

> AFAIK that's because of the idea that that people could have come to rely to
> the optional feature provided by the presence of the recommended package,
> even if that recommended package was installed in a different time than the
> recommending package.

That makes sense on user's systems (hence this is the default in apt) but not
in build chroots which is why sbuild adds above option.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Johannes Schauer]

Hi Mattia,

Quoting Mattia Rizzolo (2016-09-07 19:50:14)
> On Wed, Sep 07, 2016 at 02:36:50PM +, Thorsten Glaser wrote:
> > >In fact, to further minimize the number of packages installed into the 
> > >build
> > >chroot, I have plans to even get rid of apt and its dependencies during the
> > >build and only leave build-essential, Essential:yes packages, the build
> > >dependencies and their transitive dependencies.
> > 
> > And libcowdancer and libeatmydata, I presume… or we
> > could inject them via /tmp.
> > 
> > Good idea! (Though please make this optional. For
> > slower architectures, adding and removing apt all
> > the time will suck; on m68k I even added debhelper
> > into the base chroot as 99% of all packages need
> > it anyway, as a speed hack.)
> 
> If people would like this for pbuilder please file a bug, as I don't
> have any plan right away, and I'll surely forget it.  (and of course patches
> welcome :P)

once you do implement this in pbuilder, come back to me because I'm currently
implementing this for sbuild using three different approaches and you might
want to copy some stuff from sbuild to not write everything from scratch. :)

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Mattia Rizzolo]

On Wed, Sep 07, 2016 at 04:40:52PM +0200, gregor herrmann wrote:
> On Wed, 07 Sep 2016 16:05:24 +0200, Johannes Schauer wrote:
> 
> > > The package xmlgraphics-commons started recently failing to build from
> > > source in a clean sbuild environment although it was built successfully
> > > on the buildd network a few months ago. This behavior cannot be observed
> > > in a clean cowbuilder environment though. [1]
> 
> > > 3. or if cowbuilder should not install gnupg by default
> > I think it should not because I think that source packages should be 
> > compiled
> > in an environment that is as minimal as possible for the reasons given 
> > above.
> > But of course this is up to the cowbuilder maintainers.
> 
> FWIW, I don't have a /usr/bin/gpg in my {amd64,i386} {stretch,sid}
> cowbuilder chroots. But they are present in older ones.
> 
> I suspect I don't have them anymore because I clean my chroots (with
> debfoster), and others might still have gnupg installed because it
> was originally installed but never removed.
> 
> (I haven't tried, but I guess creating a new cowbuilder chroot won't
> have gnupg either.) 

The reason some people still have gnupg installed in old chroots but not
newer is because `apt-get autoremove` won't remove packages that are:
 * marked as installed automatically  AND
 * with no reverse-dependencies (anymore) AND
 * with reverse-recommends.

AFAIK that's because of the idea that that people could have come to
rely to the optional feature provided by the presence of the recommended
package, even if that recommended package was installed in a different
time than the recommending package.


Given this, and other instance (like the presence of all ggc-X(.Y)-base
binaries that are installed because of their priority), I recommend to
recreate the building chroots from time to time, to take account changes
in that set.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Mattia Rizzolo]

On Wed, Sep 07, 2016 at 02:36:50PM +, Thorsten Glaser wrote:
> >In fact, to further minimize the number of packages installed into the build
> >chroot, I have plans to even get rid of apt and its dependencies during the
> >build and only leave build-essential, Essential:yes packages, the build
> >dependencies and their transitive dependencies.
> 
> And libcowdancer and libeatmydata, I presume… or we
> could inject them via /tmp.
> 
> Good idea! (Though please make this optional. For
> slower architectures, adding and removing apt all
> the time will suck; on m68k I even added debhelper
> into the base chroot as 99% of all packages need
> it anyway, as a speed hack.)

If people would like this for pbuilder please file a bug, as I don't
have any plan right away, and I'll surely forget it.
(and of course patches welcome :P)

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Johannes Schauer]

Hi,

Quoting Thorsten Glaser (2016-09-07 18:49:28)
> Johannes Schauer dixit:
> 
> >   --auto-remove --allow-remove-essential apt;
> 
> Do note you may not* remove any package that is Essential: yes,
> or the package build-essential and its dependencies.

the --allow-remove-essential flag is necessary for apt to remove itself because
internally, apt considers itself as essential even though the package is not
marked as Essential:yes.

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Johannes Schauer dixit:

>   --auto-remove --allow-remove-essential apt;

Do note you may not* remove any package that is Essential: yes,
or the package build-essential and its dependencies.

*) Removing gcc-X.Y-base (and its dependencies) when newer ones
   exist and they are otherwise unused is file, though, of course.
   Same for old libraries (like liblzma2 used to be (transitively,
   I think) Essential, was replaced by liblzma5, but the old one
   still present in the chroots).

bye,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Johannes Schauer]

Hi,

Quoting Thorsten Glaser (2016-09-07 16:36:50)
> >In fact, to further minimize the number of packages installed into the build
> >chroot, I have plans to even get rid of apt and its dependencies during the
> >build and only leave build-essential, Essential:yes packages, the build
> >dependencies and their transitive dependencies.
> 
> And libcowdancer and libeatmydata, I presume… or we could inject them via
> /tmp.
> 
> Good idea! (Though please make this optional. For slower architectures,
> adding and removing apt all the time will suck; on m68k I even added
> debhelper into the base chroot as 99% of all packages need it anyway, as a
> speed hack.)

Optional: definitely.

Though removing everything that is not needed afterwards is only one way to
achieve this. All options I came up with are:

 - apt-mark auto '*'; apt-mark manual sbuild-foo-dummy; apt-get remove
   --auto-remove --allow-remove-essential apt;
 - compute the minimum set using apt-cudf and a fitting clasp or aspcud
   minimization criterion and use the solution to remove all the unneeded cruft
 - do not have apt installed inside the chroot at all but use apt from the host
   running sbuild to install packages into the chroot. This would allow sbuild
   to drop lots of cruft and workarounds which are not needed anymore with
   newer apt but are only still present to support older apt in chroots of
   older Debian releases. Though this method will fail for anything that is not
   a plain chroot but a schroot, qemu, ssh or lxc so it is not very flexible.

Obviously the first two options would only work in temporary chroot sessions
which are automatically restored into their previous state after the build (as
when schroot is used). For other chroot setups, sbuild has to make sure to
somehow re-install apt after the build. This could be tricky because we would
have to keep all the .debs needed to install apt and its dependencies around
for after the build is done...

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Thorsten Glaser]

Johannes Schauer dixit:

>> I think it is important that all maintainers can rely on the same default
>> chroot environment to test their packages before uploading to avoid possible
>> build failures.

The default chroot environment is one created with
debootstrap --variant=minbase, and then kept up to
date. Some chroot maintainers recreate it, or clean
up extra packages, more often than others; cowbuilder
--update runs apt-get autoremove, but that sometimes
doesn’t pick up packages that are part of minbase.

Either way, you can rely on whatever minbase is, and
nothing more. You can’t rely on that “nothing more”
either, which is why Build-Conflicts exist for those
rare cases that need it.

>In fact, to further minimize the number of packages installed into the build
>chroot, I have plans to even get rid of apt and its dependencies during the
>build and only leave build-essential, Essential:yes packages, the build
>dependencies and their transitive dependencies.

And libcowdancer and libeatmydata, I presume… or we
could inject them via /tmp.

Good idea! (Though please make this optional. For
slower architectures, adding and removing apt all
the time will suck; on m68k I even added debhelper
into the base chroot as 99% of all packages need
it anyway, as a speed hack.)

bye,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[gregor herrmann]

On Wed, 07 Sep 2016 16:05:24 +0200, Johannes Schauer wrote:

> > The package xmlgraphics-commons started recently failing to build from
> > source in a clean sbuild environment although it was built successfully
> > on the buildd network a few months ago. This behavior cannot be observed
> > in a clean cowbuilder environment though. [1]

> > 3. or if cowbuilder should not install gnupg by default
> I think it should not because I think that source packages should be compiled
> in an environment that is as minimal as possible for the reasons given above.
> But of course this is up to the cowbuilder maintainers.

FWIW, I don't have a /usr/bin/gpg in my {amd64,i386} {stretch,sid}
cowbuilder chroots. But they are present in older ones.

I suspect I don't have them anymore because I clean my chroots (with
debfoster), and others might still have gnupg installed because it
was originally installed but never removed.

(I haven't tried, but I guess creating a new cowbuilder chroot won't
have gnupg either.) 

Cheers,
gregor


-- 
 .''`.  Homepage https://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06
 : :' : Debian GNU/Linux user, admin, and developer -  https://www.debian.org/
 `. `'  Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
   `-   NP: Ludwig Hirsch: Der Kater


signature.asc
Description: Digital Signature

Bug#836940: [buildd-tools-devel] Bug#836940: cowbuilder, sbuild: should behave identically in regard to default gnupg installation

[Johannes Schauer]

Hi Markus!

Quoting Markus Koschany (2016-09-07 13:28:41)
> I am assigning this bug report to both of you in order to determine the
> best course of action. Please feel free to reassign and change the
> severity as appropriate.
> 
> The package xmlgraphics-commons started recently failing to build from
> source in a clean sbuild environment although it was built successfully
> on the buildd network a few months ago. This behavior cannot be observed
> in a clean cowbuilder environment though. [1]
> 
> The reasoning for the FTBFS is a missing dependency on gnupg which was
> always present in default cowbuilder and sbuild environments until
> recently. According to [2] this behavioral change was introduced by the
> apt maintainers, more accurately in version 1.3~exp1 of apt.
> 
> * move gnupg|gnupg2 from apt Depends to Recommends
> 
> This is causing build failures in sbuild now, not only for
> xmlgraphics-commons but for all packages that relied on gnupg being
> installed by default.
> 
> Please clarify if
> 
> 1. This change should be reverted in apt to restore the old behavior

apt will not revert that change. Moving gnupg to Recommends was a very
intentional step toward getting rid of gnupg as a dependency of apt completely.
So the plan of the apt maintainers is to even remove gnupg from Recommends at
some point in the future.

> 2. or if sbuild should install gnupg by default

I don't think it should. By it having it installed by default (through apt
depending on it) bugs like the missing build dependency of xmlgraphics-commons
on gnupg were never found until now.

Source packages should be built in a very minimal environment, not only to
reduce the influence that extra packages might have on the build in a non-clean
environment but also because it makes sure that the source package has all its
build dependencies declared correctly. Without programs that make it easy to
build source packages in a clean chroot we'd have lots of packages in the
archive that were only built on developer's machines and might thus potentially
miss declaring build dependencies that the maintainer happened to have
installed.

The rule is, that a source package must declare all packages it build depends
on except build-essential and all Essential:yes packages (and their transitive
dependencies) on which every source package implicitly depends. Thus, if a
source package like xmlgraphics-commons needs gnupg, then it must build depend
on it independent on whether sbuild or cowbuilder install gnupg by default.

The packages that sbuild and cowbuilder install by default are in no way meant
to express what every source package can expect to find in a minimal system.

> 3. or if cowbuilder should not install gnupg by default

I think it should not because I think that source packages should be compiled
in an environment that is as minimal as possible for the reasons given above.
But of course this is up to the cowbuilder maintainers.

> I think it is important that all maintainers can rely on the same default
> chroot environment to test their packages before uploading to avoid possible
> build failures.
> 
> Thanks for maintaining these important tools and keep up the good work.

In fact, to further minimize the number of packages installed into the build
chroot, I have plans to even get rid of apt and its dependencies during the
build and only leave build-essential, Essential:yes packages, the build
dependencies and their transitive dependencies.

Thanks!

cheers, josch


signature.asc
Description: signature