Re: Easy discovery of ‘debian/rules’ build problems (was: Unsustainable debian/rules as official build entry point?)

[Ian Jackson]

Ben Finney writes ("Easy discovery of ‘debian/rules’ build problems (was: 
Unsustainable debian/rules as official build entry point?)"):
> From the rest of your message I infer that the mention of “one consumer”
> there refers to (current or future) ‘dpkg-buildpackage’, is that correct?

Yes.

> Ian Jackson  writes:
> > Important consequences of my views include:
> >
> > * The package-provided rules interface needs to remain managed as part
> >   of policy (and to continue to have a controlled approach to updates,
> >   etc.).
> >
> > * The interface is not *defined by* dpkg-buildpackage: ie it is still
> >   possible for dpkg-buildpackage to have a bug where it does not
> >   implement the de-jure interface.
> >
> > * Packages may still need to work around bugs in old versions of
> >   dpkg-buildpackage; conversely, new versions of dpkg-buildpackage may
> >   need to work around bugs in old packages.
> >
> > * For a long time, packages should try to be compatible with old
> >   builders which invoke rules directly, even old builders other than
> >   dpkg-buildpackage.
> 
> I had been under the impression the build tools (SBuild, PBuilder, etc.)
> invoke ‘debian/rules’ directly, and so are a good way to test that
> compatibility. Evidently that impression is wrong, and the use of those
> build tools is not helping to find such bugs.

I think some of those tools may invoke rules directly some of the
time.  I haven't done a survey.

> What tools exist to allow package maintainers – as many as possible – to
> get easy (automatic?) notification when the package they maintain is not
> presenting a compatible ‘debian/rules’ build interface?

I'm not aware of anything.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Unsustainable debian/rules as official build entry point?

[Simon Richter]

Hi,

On 18.10.2017 11:36, Guillem Jover wrote:

> Using dpkg-buildpackage as the official build entry point would allow
> for much debian/rules refactoring and reduction, and optimizations.

The important bit isn't whether dpkg-buildpackage is the official entry
point, because that isn't what Policy cares about. Policy defines an
interface to be used by whatever build tool is used, and all conforming
packages implement this interface.

If a package doesn't build correctly without environment variables that
aren't part of the interface, then the package is buggy and needs to be
fixed.

If we want to extend the interface, then we need to change Policy. We
can just define that all the variables provided by dpkg-buildpackage are
part of the interface now (and I don't have a real problem with that),
but I'd rather not delegate the Policy definition of a source package to
the dpkg-buildpackage manpage, last but not least because we have
multiple versions of that manpage in different releases.

   Simon



signature.asc
Description: OpenPGP digital signature

Re: Unsustainable debian/rules as official build entry point?

[Russ Allbery]

Guillem Jover  writes:

> Given the above, and that these are clear regressions, it seems
> obvious to me that we are (collectively) not checking/using debian/rules
> as the official build entry point interface.

> And I've got to question whether we should keep supporting it or just
> declare dpkg-buildpackage to be that entry point.

I'm definitely in favor of declaring dpkg-buildpackage to be the only
officially supported entry point for kicking off a package build.  Insofar
as we can without losing anything significant, I'm in favor of
standardizing and simplifying how Debian packages are built, if for no
other reason than that it means we're not theoretically promising things
we're not actually testing.

It's rarely a good idea in computing to promise that something works when
no one uses it seriously or consistently.  In that case, it rarely
actually works, and people who rely on that promise will just be
frustrated.

-- 
Russ Allbery (r...@debian.org)   

Re: Unsustainable debian/rules as official build entry point?

[Felipe Sateler]

On Wed, 18 Oct 2017 16:31:58 +0100, Wookey wrote:

> On 2017-10-18 12:08 +, Felipe Sateler wrote:
>> On Wed, 18 Oct 2017 11:36:41 +0200, Guillem Jover wrote:
>> 
>> > And I've got to question whether we should keep supporting it or just
>> > declare dpkg-buildpackage to be that entry point.
>> 
>> I think it makes sense to declare dpkg-buildpackage the official entry
>> point.
> 
> I too am reasonably happy with this idea, for the reasons you and Felipe
> gave, however I can think of one potential issue.
> 
>> Reasons against:
> 
> I quite often use the debian/rules binary{-arch,-indep} interface when
> doing porting/bootstrapping work (i.e the package built but something
> goes wrong in the packaging process so I want to retry with a tweak or a
> bodge)
> 
> In theory I should be able to do dpkg-buildpackage -nc --target=binary

Two things:

1. -nc is not necessary when passing --target. 
2. You mentioned binary{-arch,-indep} earlier but here --target=binary.
   Perhaps that is the source of the discrepancy? Note that you can call
   any target, not just the official ones. (You may need to also pass 
   --as-root too though).

> 
> but in practice I find that this often doesn't work as intended and it
> tries to do the whole build again. I have not investigated exactly why
> this is, and I guess you'll want me to give you a concrete example next.

This sound a lot like a bug somewhere. So yeah, discussion in the 
abstract is not going to do much because what you describe is not 
supposed to happen. Clearly something is failing somewhere, but without 
concrete examples I don't think there's anything actionable.



-- 
Saludos,
Felipe Sateler

Easy discovery of ‘debian/rules’ build problems (was: Unsustainable debian/rules as official build entry point?)

[Ben Finney]

Ian Jackson  writes:

> After there is only one consumer [of the package-provided
> ‘debian/rules’ build interface], it will be somewhat easier to change
> [the policy so that interface is not the standard].

>From the rest of your message I infer that the mention of “one consumer”
there refers to (current or future) ‘dpkg-buildpackage’, is that correct?

> Important consequences of my views include:
>
> * The package-provided rules interface needs to remain managed as part
>   of policy (and to continue to have a controlled approach to updates,
>   etc.).
>
> * The interface is not *defined by* dpkg-buildpackage: ie it is still
>   possible for dpkg-buildpackage to have a bug where it does not
>   implement the de-jure interface.
>
> * Packages may still need to work around bugs in old versions of
>   dpkg-buildpackage; conversely, new versions of dpkg-buildpackage may
>   need to work around bugs in old packages.
>
> * For a long time, packages should try to be compatible with old
>   builders which invoke rules directly, even old builders other than
>   dpkg-buildpackage.

I had been under the impression the build tools (SBuild, PBuilder, etc.)
invoke ‘debian/rules’ directly, and so are a good way to test that
compatibility. Evidently that impression is wrong, and the use of those
build tools is not helping to find such bugs.

What tools exist to allow package maintainers – as many as possible – to
get easy (automatic?) notification when the package they maintain is not
presenting a compatible ‘debian/rules’ build interface?

-- 
 \  “Very few things happen at the right time, and the rest do not |
  `\ happen at all. The conscientious historian will correct these |
_o__)  defects.” —Mark Twain, _A Horse's Tale_ |
Ben Finney

Re: Unsustainable debian/rules as official build entry point?

[Svante Signell]

On Wed, 2017-10-18 at 16:31 +0100, Wookey wrote:
> On 2017-10-18 12:08 +, Felipe Sateler wrote:

> I quite often use the debian/rules binary{-arch,-indep} interface when
> doing porting/bootstrapping work (i.e the package built but something
> goes wrong in the packaging process so I want to retry with a tweak or a
> bodge)
> 
> In theory I should be able to do 
> dpkg-buildpackage -nc --target=binary
> 
> but in practice I find that this often doesn't work as intended and it
> tries to do the whole build again. I have not investigated exactly why
> this is, and I guess you'll want me to give you a concrete example next.

Couldn't agree more. Porting packages is mostly the task of my work on GNU/Hurd
and being able to issue debian/rules configure, build, binary, clean, etc is
extremely useful. Especially for source packages building a lot of binaries.

> Doing the whole build again is sometimes just slow (very slow!), but
> can also be a PITA when porting, and you really do just want to
> package up what you have.

See above. Also, when testing if a package can be built twice in a row (not
necessarily for Hurd) it is important. I've encountered several such packages in
the Debian repository (still there).

Re: Unsustainable debian/rules as official build entry point?

[Wookey]

On 2017-10-18 12:08 +, Felipe Sateler wrote:
> On Wed, 18 Oct 2017 11:36:41 +0200, Guillem Jover wrote:
> 
> > And I've got to question whether we should keep supporting it or just
> > declare dpkg-buildpackage to be that entry point.
> 
> I think it makes sense to declare dpkg-buildpackage the official entry 
> point.

I too am reasonably happy with this idea, for the reasons you and
Felipe gave, however I can think of one potential issue.

> Reasons against:

I quite often use the debian/rules binary{-arch,-indep} interface when
doing porting/bootstrapping work (i.e the package built but something
goes wrong in the packaging process so I want to retry with a tweak or a bodge)

In theory I should be able to do 
dpkg-buildpackage -nc --target=binary

but in practice I find that this often doesn't work as intended and it
tries to do the whole build again. I have not investigated exactly why
this is, and I guess you'll want me to give you a concrete example next.

Doing the whole build again is sometimes just slow (very slow!), but
can also be a PITA when porting, and you really do just want to
package up what you have.

I guess my point is that I do not find these interfaces to be
equivalent in practice so there is value for me in retaining the
debian/rules binary interface at least until the dpkg-buidpackage one
reliably does the same thing. Perhaps I am missing some magic switch.

It seems like this is a bug/interaction, rather than a fundamental
reason for retaining the debian/rules interface in the long term, but
I do see it as a reason to proceed cautiously. 

Sadly I am failing to remember whch package did this to me most
recently, even though it wasn't long ago. (Something in the bottom
'debootstrap' 144 :-)

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Re: Unsustainable debian/rules as official build entry point?

[Samuel Thibault]

Svante Signell, on mer. 18 oct. 2017 13:51:25 +0200, wrote:
> On Wed, 2017-10-18 at 11:54 +0100, Simon McVittie wrote:
> > On Wed, 18 Oct 2017 at 11:57:55 +0200, Svante Signell wrote:
> > > Building some packages for GNU/Hurd has been impossible in the
> > > past, since also tests are run under fakeroot.
> > 
> > Is there some reason why this would be Hurd-specific? Is fakeroot's
> > emulation of real root significantly more limited on Hurd, or do the
> > Hurd buildds invoke builds differently, or something?
> 
> The problems was revealed when fakeroot was flaky (native
> implementation). That fakeroot problem is now solved.

It's also not always flakiness. Sometimes there are just several ways to
emulate root behavior, and the hurdish implementation happened to use a
different one from the non-Hurd fakeroot.

Samuel

Re: Unsustainable debian/rules as official build entry point?

[Ian Jackson]

Guillem Jover writes ("Unsustainable debian/rules as official build entry 
point?"):
> Given the above, and that these are clear regressions, it seems
> obvious to me that we are (collectively) not checking/using debian/rules
> as the official build entry point interface.
> 
> And I've got to question whether we should keep supporting it or just
> declare dpkg-buildpackage to be that entry point.

tl;dr: I think I agree with your suggestion but I would definitely
  express it rather differently.


There are two entry points which are relevant:

1. The interface provided by packages.
2. The interface consumed by programs which want to build packages.

Up to now, both of these have been, officially, the same:
debian/rules.

But for purpose 2 most programs invoke dpkg-buildpackage now.  This is
because the official interface 1 keeps becoming more complex,
particularly because of backward compatibility requirements.

The compatibility and protocol negotiation functionality in
dpkg-buildpackage, where dpkg-buildpackage arranges to work with new
packages in new ways and old packages in old ways, becomes harder and
harder to reproduce.  So de facto, most consumers of source packages
consume them via dpkg-buildpackage.

I think there is a lot of value in interposing a common
compatibility/adjustment layer between the two interfaces I mention
above.  So I would support a proposal to official declare that
programs which want to build (unrelated) packages should do so via
dpkg-buildpackage.

However:

I would _not_ support a proposal to abolish the package-provided rules
interface as de-jure standard.  It's just that this standard would
gradually lose its other consumers.  After there is only one consumer
it will be somewhat easier to change it.

Additionally, although there being one consumer makes it seem as if we
could just add new mandatory features to packages (ie, changes which
require new dpkg-buildpackage), I don't think we should do that.

It is important for backporting, downstreams, etc. etc., that old
versions of dpkg-buildpackage can build new packages.

Important consequences of my views include:

* The package-provided rules interface needs to remain managed as part
  of policy (and to continue to have a controlled approach to updates,
  etc.).

* The interface is not *defined by* dpkg-buildpackage: ie it is still
  possible for dpkg-buildpackage to have a bug where it does not
  implement the de-jure interface.

* Packages may still need to work around bugs in old versions of
  dpkg-buildpackage; conversely, new versions of dpkg-buildpackage may
  need to work around bugs in old packages.

* For a long time, packages should try to be compatible with old
  builders which invoke rules directly, even old builders other than
  dpkg-buildpackage.


I think you will probably agree, but I wanted to set it all out.

Ian.

Re: Unsustainable debian/rules as official build entry point?

[Felipe Sateler]

On Wed, 18 Oct 2017 11:36:41 +0200, Guillem Jover wrote:

> And I've got to question whether we should keep supporting it or just
> declare dpkg-buildpackage to be that entry point.

I think it makes sense to declare dpkg-buildpackage the official entry 
point. Reasons for:

1. It is already the de-facto entry point, since buildds use dpkg-
buildpackage, and so do pbuilder and sbuild.

2. Most (all?) of the archive already depends on dpkg-dev, by using the /
usr/share/dpkg/*.mk helpers (either directly or indirectly via CDBS) and 
using dpkg-* tools (either directly or indirectly via debhelper).

3. Providing a clean environment helps the reproducibility efforts, by 
providing a single place where many weird environment variables can be 
neutered. It probably will also help other archive-wide efforts

4. dpkg-buildpackage has the --target and --as-root flags to invoke any 
debian/rules target since 1.15 (oldoldstable has 1.16).


Reasons against:

1. Finger memory. This can be alleviated via a suitable alias 
drules='dpkg-buildpackage --target'.

I can't come up with other reasons.

Let's make dpkg-buildpackage the official entry point.

-- 
Saludos,
Felipe Sateler

Re: Unsustainable debian/rules as official build entry point?

[Svante Signell]

On Wed, 2017-10-18 at 11:54 +0100, Simon McVittie wrote:
> On Wed, 18 Oct 2017 at 11:57:55 +0200, Svante Signell wrote:
> > Building some packages for GNU/Hurd has been impossible in the
> > past, since also tests are run under fakeroot.
> 
> Is there some reason why this would be Hurd-specific? Is fakeroot's
> emulation of real root significantly more limited on Hurd, or do the
> Hurd buildds invoke builds differently, or something?

The problems was revealed when fakeroot was flaky (native
implementation). That fakeroot problem is now solved.

> dbus fails build-time tests under fakeroot on all architectures
> (or at least, I would expect it to, because it tests AF_UNIX
> credentials-passing), which makes me wonder why this failure mode
> would have manifested on Hurd and not Linux in the past. I would have
> expected packages whose tests don't like fakeroot to either succeed
> equally on both, or fail equally on both, if unrelated reasons for
> failure are ignored.

Yes, you are right. From what I remember most packages either succeed
or fail when tests are run when run under fakeroot for all
architectures. In my opinion tests should never be run under fakeroot
(i.e. in the binary target) except when requested specifically (cannot
be solved by a buildd?). The problem is the same as for tests requiring
internet connections, which are not allowed, right?

Re: Unsustainable debian/rules as official build entry point?

[Guillem Jover]

On Wed, 2017-10-18 at 11:54:57 +0100, Simon McVittie wrote:
> On Wed, 18 Oct 2017 at 11:57:55 +0200, Svante Signell wrote:
> > Building some packages for GNU/Hurd has been impossible in the past,
> > since also tests are run under fakeroot.
> 
> Is there some reason why this would be Hurd-specific? Is fakeroot's
> emulation of real root significantly more limited on Hurd, or do the
> Hurd buildds invoke builds differently, or something?

The Hurd has had its own fakeroot implementation for a long time.

Regards,
Guillem

Re: Unsustainable debian/rules as official build entry point?

[Simon McVittie]

On Wed, 18 Oct 2017 at 11:57:55 +0200, Svante Signell wrote:
> Building some packages for GNU/Hurd has been impossible in the past,
> since also tests are run under fakeroot.

Is there some reason why this would be Hurd-specific? Is fakeroot's
emulation of real root significantly more limited on Hurd, or do the
Hurd buildds invoke builds differently, or something?

dbus fails build-time tests under fakeroot on all architectures
(or at least, I would expect it to, because it tests AF_UNIX
credentials-passing), which makes me wonder why this failure mode would
have manifested on Hurd and not Linux in the past. I would have expected
packages whose tests don't like fakeroot to either succeed equally
on both, or fail equally on both, if unrelated reasons for failure
are ignored.

smcv

Re: Unsustainable debian/rules as official build entry point?

[Svante Signell]

On Wed, 2017-10-18 at 11:36 +0200, Guillem Jover wrote:
> Hi!
> 
> So, dpkg 1.19.0 and 1.19.0.1 had a bug where the build target was not
> being called when building packages.

Thanks, this problem has finally been revealed officially. Are you sure
this problem is not older than version 1.19.x? 

Building some packages for GNU/Hurd has been impossible in the past,
since also tests are run under fakeroot. And they should not, they are
part of the build target, not the binary target. We have tried to point
this out several times, but until now was not taken seriously.

Unsustainable debian/rules as official build entry point?

[Guillem Jover]

Hi!

So, dpkg 1.19.0 and 1.19.0.1 had a bug where the build target was not
being called when building packages.

Apparently this caused mass build failures, all due to packages (or
their helpers) being very Debian policy non-compliant! These are all
MUST requirements. Stuff like:

  - binary-targets are the only thing required to build a package, and
the needed build-targets must be depended on by the binary-targets.
  - build-targets must not require root privs, but must be able to be
run with root privs (via binary-target invocation).
  - debian/rules must not assume any pre-existing environment variable,
as it still needs to be runnable as debian/rules.

From what I've been told (thanks Niels and Helmut! :), we have lots of
packages in the archive that FTBFS due to, at least:

  - debhelper compat levels < 9 not recursing into debian/rules to
call the build target.
  - dh-elpa's test failing when executed as root.
  - debian/rules assuming DEB_*_MULTIARCH being present w/o defining it.

The first will just solve itself as people upgrade compat, the second
does really need fixing (although might get shadowed again with the
upcoming R³ support), the last is a recurring problem.


Given the above, and that these are clear regressions, it seems
obvious to me that we are (collectively) not checking/using debian/rules
as the official build entry point interface.

And I've got to question whether we should keep supporting it or just
declare dpkg-buildpackage to be that entry point.


Using debian/rules as the official build entry point has the beauty
that it makes source packages be somewhat self-contained (although
they will still require dpkg-dev tools internally), has perhaps less
interface surface than dpkg-buildpackage provides, and does not tie
it to a specific implementation (dpkg in this case).

Using dpkg-buildpackage as the official build entry point would allow
for much debian/rules refactoring and reduction, and optimizations. We
could predefine multiple environment variables from data sources
dpkg-buildpackage has already needed to parse, or to get a defined and
cleaner environment (see recent LC_ALL=C.UTF-8 thread). It could allow
us to make packaging easier by possbly (and conditionally) reversing the
current inside-out build nature, where the bulk of the building logic is
inside debian/rules via some helper, and even perhaps reduce the need
for a helper, making packaging easier. Etc.


In the past there's been quite strong opposition to switching the
official build entry point from debian/rules to dpkg-buildpackage
(well, even to debian/rules being a Makefile! :). But it seems obvious
that the current situation is not very sustainable. So if there's
still opposition, it might make sense for those opposing to at least
try to attest the conformance of that interface from time to time, and
file bugs etc.? Thoughts?

Thanks,
Guillem