Re: Help requested: Packages which FTBFS randomly

[Simon McVittie]

On Thu, 23 Feb 2017 at 10:02:55 -0800, Nikolaus Rath wrote:
> Yes, but last time I checked a "apt-get source && debuild -us -uc" still
> defaults to using just a single CPU unless you explicitly set
> DEB_BUILD_OPTIONS=parallel.

Check again - dpkg-buildpackage now defaults to -Jauto
(it sets DEB_BUILD_OPTIONS=parallel=n where n is the detected number of
CPUs), on the basis that every Debian buildd already seems to set
parallel >= 2, so any package that fails to build in that situation is
already effectively unreleasable. (#842845)

If you maintain packages where this can't work, use dh --no-parallel
or put .NOPARALLEL: in your Makefile.

S

Re: Help requested: Packages which FTBFS randomly

[Vincent Bernat]

 ❦ 23 février 2017 10:02 -0800, Nikolaus Rath  :

 Your chosen build environment is not common [...]
>>>
>>> This has come up a few times now. Could someone explain what is so odd
>>> about his envirnoment? It does not look unusual to me.
>>
>> Official buildd have several CPU. Most "important" downstream
>> distribution builders have several CPU. [...]
>
> Yes, but last time I checked a "apt-get source && debuild -us -uc" still
> defaults to using just a single CPU unless you explicitly set
> DEB_BUILD_OPTIONS=parallel. So if this causes the build to fail, it will
> directly affect those users least able to figure out what may be
> wrong.

When the build system is not make-based, it doesn't care for this option
(for example, Go programs compile using all the available
cores). Moreover, tests can do whatever they want.
-- 
Extreme fear can neither fight nor fly.
-- William Shakespeare, "The Rape of Lucrece"


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Nikolaus Rath]

On Feb 21 2017, Vincent Bernat  wrote:
>  ❦ 21 février 2017 09:48 -0800, Nikolaus Rath  :
>
>>> Your chosen build environment is not common [...]
>>
>> This has come up a few times now. Could someone explain what is so odd
>> about his envirnoment? It does not look unusual to me.
>
> Official buildd have several CPU. Most "important" downstream
> distribution builders have several CPU. [...]

Yes, but last time I checked a "apt-get source && debuild -us -uc" still
defaults to using just a single CPU unless you explicitly set
DEB_BUILD_OPTIONS=parallel. So if this causes the build to fail, it will
directly affect those users least able to figure out what may be wrong.

> Accomodating for all build environments is a slippery slope.

In my opinion there is a big difference between asking builds to work
with a single CPU, and supporting "all build environments".


Best,
-Nikoalus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Help requested: Packages which FTBFS randomly

[Simon McVittie]

On Tue, 21 Feb 2017 at 23:35:44 +, Ben Hutchings wrote:
> Having said that, some ioctls that make sense for block-backed
> filesystems, such as FS_IOC_FIEMAP, won't work on a tmpfs (or nfs,
> ubifs, etc.).

One notable omission is that tmpfs doesn't do generic "user." extended
attributes (due to worries about quota control), which results in ostree's
tests preferring /var/tmp over /tmp, and skipping many of the tests if
/var/tmp is a tmpfs too.

S

Re: Help requested: Packages which FTBFS randomly

[Vincent Bernat]

 ❦ 22 février 2017 00:46 +0100, Adam Borowski  :

>> > > * using a qemu build chroot (Debian doesn't do this, other might)
>> > 
>> > Is that because QEMU is slow, or some other reason?
>> 
>> AIUI qemu(-static) cannot handle threading very well. So if a build
>> process uses such applications, things turn bad. Typical observation was
>> msgmerge stuck in an endless loop at 100% CPU. Lesson learned: There is
>> a reason why Debian builds do not use emulation.
>
> That's qemu-user.  On the other hand, qemu-system, while nowhere as
> convenient, is safe and reliable.  Its bugs don't exceed the differences
> between revisions of real hardware.

It doesn't emulate correctly inability of some architectures to do
unaligned memory accesses.
-- 
Make it right before you make it faster.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Ben Hutchings]

On Tue, 2017-02-21 at 14:18 -0700, Sean Whitton wrote:
> On Tue, Feb 21, 2017 at 02:37:23AM +, Ben Hutchings wrote:
> > > * with /tmp on tmpfs on some archs
> > 
> > [...]
> > 
> > You mean the 64-bit PowerPC architectures?  tmpfs allocates at
> > least a
> > page per file, and they have 64K pages...
> 
> I'm not sure why you're mentioned powerpc archs, but as a hopefully
> relevant data point, two of my arch:all packages FTBFS when / is a
> tmpfs.  There are some weird permissions errors from the test suites.

That sounds like a kernel bug - tmpfs is meant to implement permissions
(including ACLs) just like a "normal" filesystem (e.g. ext4).

Having said that, some ioctls that make sense for block-backed
filesystems, such as FS_IOC_FIEMAP, won't work on a tmpfs (or nfs,
ubifs, etc.).

Ben.

-- 
Ben Hutchings
73.46% of all statistics are made up.



signature.asc
Description: This is a digitally signed message part

Re: Help requested: Packages which FTBFS randomly

[Adam Borowski]

On Tue, Feb 21, 2017 at 11:04:52PM +0100, Christoph Biedl wrote:
> > > * using a qemu build chroot (Debian doesn't do this, other might)
> > 
> > Is that because QEMU is slow, or some other reason?
> 
> AIUI qemu(-static) cannot handle threading very well. So if a build
> process uses such applications, things turn bad. Typical observation was
> msgmerge stuck in an endless loop at 100% CPU. Lesson learned: There is
> a reason why Debian builds do not use emulation.

That's qemu-user.  On the other hand, qemu-system, while nowhere as
convenient, is safe and reliable.  Its bugs don't exceed the differences
between revisions of real hardware.


Meow!
-- 
Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
  ./configure --host=zx-spectrum --build=pdp11

Re: Help requested: Packages which FTBFS randomly

[Christoph Biedl]

Ben Hutchings wrote...

> > * when using eatmydata
> 
> I can certainly think of a test case that would be broken by eatmydata
> and I would not want to rule out such test cases.  But still, I am
> suprised by this.

#667965 - don't know whether this still exists. I later decided to patch
dpkg so "unsafe-io" skips *all* calls to *sync. This also works better
in mixed-arch environments.

> > * on sbuild using overlayfs
> 
> overlayfs is sadly quite a way from being a POSIX-compliant filesystem.
>  So it seems unreasonable to expect every package to be buildable in
> such a build environment.

Indeed. That's why I'm asking for such a hint file: It would allow me to
switch to a more expensive build mode, but a mode that makes the build
pass.

> > * using a qemu build chroot (Debian doesn't do this, other might)
> 
> Is that because QEMU is slow, or some other reason?

AIUI qemu(-static) cannot handle threading very well. So if a build
process uses such applications, things turn bad. Typical observation was
msgmerge stuck in an endless loop at 100% CPU. Lesson learned: There is
a reason why Debian builds do not use emulation.

Christoph


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Christoph Biedl]

Sean Whitton wrote...

> I'm not sure why you're mentioned powerpc archs

Because that's a surprising feature of that arch and once you've
realized you were caught by this, you will not forget it. Bonus:
Rebuilding on a porter box passes since /home is not a tmpfs.

Christoph

PS: 
https://sources.debian.net/patches/logrotate/3.11.0-0.1/fix-test-pagesize.patch/


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Sean Whitton]

On Tue, Feb 21, 2017 at 02:37:23AM +, Ben Hutchings wrote:
> > * with /tmp on tmpfs on some archs
> [...]
> 
> You mean the 64-bit PowerPC architectures?  tmpfs allocates at least a
> page per file, and they have 64K pages...

I'm not sure why you're mentioned powerpc archs, but as a hopefully
relevant data point, two of my arch:all packages FTBFS when / is a
tmpfs.  There are some weird permissions errors from the test suites.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Tue, Feb 21, 2017 at 09:36:58PM +0100, Vincent Bernat wrote:

> Accomodating for all build environments is a slippery slope. What if I
> use a 128MB host with 64GB of swap? Timing-related tests will start to
> fail. Is it Debian job to fix all the test suites? Should I be able to
> build packages on a system not supporting ACL? On a kernel not
> supporting users? On a grsec kernel? With a low ulimit value for the
> number of opened files or on the stack size? On a chroot on an Android
> phone? On the Windows Subsystem for Linux?

I don't really have a problem with multi-core being the only supported
build setup in some distant future, but if such thing ever happens, it
should be because Debian decides it, either the Release Managers at
the start of a release cycle (not at the end), or the Debian
Developers as a whole by way of a General Resolution.

The current scenario where some maintainers think they have the power
to decide about this on their own for their own packages is not
acceptable at all.

Having said that, there are practical reasons why we should keep
support for single-CPU build systems, for example, you can split the
work better if you have a build farm and you can save real money.

Also, the number of packages that we have to fix to support this is
still extremely small, and most of them have been fixed as they have
been reported. The number of packages in stretch which are not
buildable on a single-CPU system may be counted with the fingers of
just one hand.

Nothing to do with the number of problems that would arise if you use
a 128MB host with 64GB of swap or any of the really weird things you
mention. Really, building with a single CPU is not a sin, and people
should not be punished for doing so.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Vincent Bernat]

 ❦ 21 février 2017 09:48 -0800, Nikolaus Rath  :

>> Your chosen build environment is not common [...]
>
> This has come up a few times now. Could someone explain what is so odd
> about his envirnoment? It does not look unusual to me.

Official buildd have several CPU. Most "important" downstream
distribution builders have several CPU. Your own personal
laptop/workstation have several CPU. Free CI environments usually have
several CPU (eg. Travis CI). Private CI usually have several CPU
(eg. jenkins.debian.net, debomatic).

Just checked with Ubuntu, ri-li builds successfully and reliably on
their platform. Moreover, I don't think we ever had downstream
distributions complained about the inability to compile packages. From
my point of view, we are told to fix bugs that don't affect our users.

Accomodating for all build environments is a slippery slope. What if I
use a 128MB host with 64GB of swap? Timing-related tests will start to
fail. Is it Debian job to fix all the test suites? Should I be able to
build packages on a system not supporting ACL? On a kernel not
supporting users? On a grsec kernel? With a low ulimit value for the
number of opened files or on the stack size? On a chroot on an Android
phone? On the Windows Subsystem for Linux?
-- 
Use variable names that mean something.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Nikolaus Rath]

> Your chosen build environment is not common [...]


This has come up a few times now. Could someone explain what is so odd
about his envirnoment? It does not look unusual to me.


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Help requested: Packages which FTBFS randomly

[Ben Hutchings]

On Tue, 2017-02-21 at 11:50 +0800, Paul Wise wrote:
> On Tue, Feb 21, 2017 at 6:36 AM, Christoph Biedl wrote:
> 
> > This is a charming idea altough I have doubt it will work out: As
> > usual the information has to be kept up-to-date, so unless it is
> > collected and verified every now and then automatically, it will
> > become unsuable pretty soon.
> 
> FYI the buildds are automatically collecting disk usage information,
> see the last line of each build log.
> 
> Of course, that information isn't parsed and stored anywhere yet.
> 
> I guess collecting memory usage would be much harder, especially if
> multiple packages are built in parallel.

I think that can be done by putting each build in its own memory
controller cgroup (with high limits, at least for now).

Ben.

-- 
Ben Hutchings
73.46% of all statistics are made up.



signature.asc
Description: This is a digitally signed message part

Re: Help requested: Packages which FTBFS randomly

[Holger Levsen]

On Mon, Feb 20, 2017 at 04:42:49PM -0800, Russ Allbery wrote:
> To say my opinion explicitly, since there's been a lot of discussion here,
> some of which I've been involved in somewhat ambiguously:
 
Thanks for writing this up, Russ. I fully agree with *everything* you said here:

> I think this is a reasonably short and workable list of bugs, and I think
> the *default* assumption should be that any FTBFS, even intermittant, is
> RC.  This is both for fundamental reasons for what we're trying to achieve
> as a free software distribution that's modifyable and rebuildable by our
> users, and for practical reasons to support further development of our
> packages (including security fixes).
> 
> We already have an exception mechanism for handling bugs that shouldn't be
> RC for the current release because the actual impact of the bug turns out
> to be minor for some reason, or because they would have too negative of an
> impact on the release.  Normally, how we handle this is to mark the bugs
> with an RC severity (serious in this case), and then, if the release team
> feels this package warrants a special exception, mark it as ignored for
> the current release.
> 
> To me, that seems like the obvious approach to take here, and given the
> reasonably short list of bugs, I don't feel like that would cause
> unreasonable disruption.  In the cases where the build failures are from
> flaky tests, I think disabling the test is a perfectly reasonable fix.  If
> there is some significant merit to a test that fails in Santiago's
> environment but doesn't on the buildd network for some reason *and* has
> significant value in catching regressions on release architectures, that's
> an obvious case for an exception and could be handled like any other RC
> bug exception.
> 
> I think we're going to keep getting lost in the weeds when we try to
> discuss this in general hypotheticals.  If individual package maintainers
> request individual RC exceptions for their specific cases, the discussion
> can be far more concrete and in most cases will have an obvious outcome.
> 
> So, to be explicit, my opinion (as just a Debian Developer, with no
> special ability to decide this, so just take this as another of the
> contributions to the thread) is that all of these bugs should be set to
> Severity: serious, and the maintainers can ask for stretch-ignore tags (or
> downgrades for their specific bug if that seems more correct for specific
> reasons related to their packge, whatever those may be) if they feel that
> is appropriate.

And with this "+1" mail I shall stop contributing to this thread. Everything
has been said at least once :)

Now let's deal with individual bugs and work on getting Stretch released
eventually!


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Steve Cotton]

On Mon, Feb 20, 2017 at 01:12:33PM +0100, Santiago Vila wrote:
> Believe me, this is also frustrating for me. If you absolutely need a
> machine to reproduce this, contact me privately.

If someone creates a patch, then the bug is much more likely to get fixed,
whether it's RC or not.  Any FTBFS that needs an entire VM with an
unusual-configuration to reproduce may be RC, but it's another RC bug that when
volunteers look at the RC bug list, they think "I don't know where to start on
that one".

For these single-CPU buildds, there's a simple way to reproduce it at least the
bug that I looked at in a debugging environment, by using the taskset command
from package util-linux to run the build on CPU #0 only:

$ taskset --cpu-list 0 dpkg-buildpackage -A

For changing policy, I think "must be able to build using a single CPU, for
example using (taskset and the FreeBSD and Hurd equivalents), because otherwise
there's obviously a race condition somewhere" would be much clearer.

Steve

Re: Help requested: Packages which FTBFS randomly

[Adam Borowski]

On Tue, Feb 21, 2017 at 11:50:15AM +0800, Paul Wise wrote:
> On Tue, Feb 21, 2017 at 6:36 AM, Christoph Biedl wrote:
> 
> > This is a charming idea altough I have doubt it will work out: As
> > usual the information has to be kept up-to-date, so unless it is
> > collected and verified every now and then automatically, it will
> > become unsuable pretty soon.
> 
> FYI the buildds are automatically collecting disk usage information,
> see the last line of each build log.
> 
> Of course, that information isn't parsed and stored anywhere yet.

So here's my data, from multiple rebuilds since Sep 2014:
https://angband.pl/tmp/builds.sql.xz
(Postgres, but it should be easy to edit the schema for any SQL dialect)

It's all from one machine, running the same kernel line (3.8 vendor for
Odroid-U2), on the same filesystem (btrfs noatime,compress=lzo,ssd), sameish
sbuild settings (eatmydata; regularly upgraded with unstable though).
The data is not ideal -- I run multiple sbuild instances which with only 2GB
memory notoriously ends in swappeathons, thus a package's build time is
affected by what was running concurrently.

Obviously you want only records with status='successful'; I haven't removed
failures in case someone is interested -- for example, there's a bunch of
FTBFSes I haven't yet investigated/filed.

Note that the disk space data is misleading -- sbuild notes only the final
size of the build dir after finished build, peak size may differ.  For
example, ceph ENOSPCes with 51GB free despite sbuild saying only 21GB.

> I guess collecting memory usage would be much harder, especially if
> multiple packages are built in parallel.

Packages being built in parallel are not a problem -- neither overlayfs nor
btrfs support sharing pages yet even if they use the same on-disk blocks.
Getting peak usage for a set of processes is a very tricky task, though:
if a process uses 50MB then forks, each copy taking 20MB more, the usage
is 90MB not 140MB.  Processes come and go, executable/library pages are
shared, and so on.

The only tool that _looks_ like it gets it right seems to be cgmemtime, not
packaged in Debian yet and requiring some setup.

Having peak memory data from builds would be awesome.


Meow!
-- 
Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
  ./configure --host=zx-spectrum --build=pdp11

Re: Help requested: Packages which FTBFS randomly

[Paul Wise]

On Tue, Feb 21, 2017 at 6:36 AM, Christoph Biedl wrote:

> This is a charming idea altough I have doubt it will work out: As
> usual the information has to be kept up-to-date, so unless it is
> collected and verified every now and then automatically, it will
> become unsuable pretty soon.

FYI the buildds are automatically collecting disk usage information,
see the last line of each build log.

Of course, that information isn't parsed and stored anywhere yet.

I guess collecting memory usage would be much harder, especially if
multiple packages are built in parallel.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Help requested: Packages which FTBFS randomly

[Ben Hutchings]

On Mon, 2017-02-20 at 23:36 +0100, Christoph Biedl wrote:
> Niels Thykier wrote...
> 
> [ topic shift ]
> 
> > On a related note: Having some way to declare minimum requirements for
> > e.g. disk space and memory (a la "base GB usage + GB usage/core") used
> > would be great.
> >   Especially if it is available in metadata, so wanna-build can see
> > whether it makes sense to assign a given package to a given build-node.
> 
> This is a charming idea altough I have doubt it will work out: As
> usual the information has to be kept up-to-date, so unless it is
> collected and verified every now and then automatically, it will
> become unsuable pretty soon.
> 
> Otherwise, there are more things I could use in such a buildd routing
> table: I remember I've seen packages failing to build
> 
> * when using eatmydata

I can certainly think of a test case that would be broken by eatmydata
and I would not want to rule out such test cases.  But still, I am
suprised by this.

> * on sbuild using overlayfs

overlayfs is sadly quite a way from being a POSIX-compliant filesystem.
 So it seems unreasonable to expect every package to be buildable in
such a build environment.

> * using a qemu build chroot (Debian doesn't do this, other might)

Is that because QEMU is slow, or some other reason?

> * with /tmp on tmpfs on some archs
[...]

You mean the 64-bit PowerPC architectures?  tmpfs allocates at least a
page per file, and they have 64K pages...

Ben.

-- 
Ben Hutchings
73.46% of all statistics are made up.



signature.asc
Description: This is a digitally signed message part

Re: Help requested: Packages which FTBFS randomly

[Russ Allbery]

Santiago Vila  writes:

> The following packages FTBFS for me randomly. First column is the bug
> number, second column is the estimated probability of failure in my
> build environment, which is described here:

> https://people.debian.org/~sanvila/my-building-environment.txt

> Before I ask the Release Managers that they make the bugs at the top
> of this list serious again (using some kind of threshold, as discussed
> in Bug #844264), I would appreciate if somebody else could double check
> that those packages have indeed very flaky tests (or buggy Makefiles,
> in some cases).

> [ Please report any findings to the nnn...@bugs.debian.org address ].

> While I agree that none of those individual bugs is more important
> than a bug of "FTBFS-always" type, some packages have a failure ratio
> so high that if we accepted that they "build from source" with such a
> high failure rate, they would effectively honor policy if we had it
> reversed, as in "packages must *not* build from source". I call this
> "the FTBFS-randomly Schrödinger paradox".

To say my opinion explicitly, since there's been a lot of discussion here,
some of which I've been involved in somewhat ambiguously:

I think this is a reasonably short and workable list of bugs, and I think
the *default* assumption should be that any FTBFS, even intermittant, is
RC.  This is both for fundamental reasons for what we're trying to achieve
as a free software distribution that's modifyable and rebuildable by our
users, and for practical reasons to support further development of our
packages (including security fixes).

We already have an exception mechanism for handling bugs that shouldn't be
RC for the current release because the actual impact of the bug turns out
to be minor for some reason, or because they would have too negative of an
impact on the release.  Normally, how we handle this is to mark the bugs
with an RC severity (serious in this case), and then, if the release team
feels this package warrants a special exception, mark it as ignored for
the current release.

To me, that seems like the obvious approach to take here, and given the
reasonably short list of bugs, I don't feel like that would cause
unreasonable disruption.  In the cases where the build failures are from
flaky tests, I think disabling the test is a perfectly reasonable fix.  If
there is some significant merit to a test that fails in Santiago's
environment but doesn't on the buildd network for some reason *and* has
significant value in catching regressions on release architectures, that's
an obvious case for an exception and could be handled like any other RC
bug exception.

I think we're going to keep getting lost in the weeds when we try to
discuss this in general hypotheticals.  If individual package maintainers
request individual RC exceptions for their specific cases, the discussion
can be far more concrete and in most cases will have an obvious outcome.

So, to be explicit, my opinion (as just a Debian Developer, with no
special ability to decide this, so just take this as another of the
contributions to the thread) is that all of these bugs should be set to
Severity: serious, and the maintainers can ask for stretch-ignore tags (or
downgrades for their specific bug if that seems more correct for specific
reasons related to their packge, whatever those may be) if they feel that
is appropriate.

-- 
Russ Allbery (r...@debian.org)   

Re: Help requested: Packages which FTBFS randomly

[Russ Allbery]

Holger Levsen  writes:
> On Mon, Feb 20, 2017 at 09:59:29PM +, Ian Jackson wrote:

>>  + A FTBFS bug is still RC even if:
>>  + it doesn't happens on buildd.debian.net;
>>  + it only happens randomly;
>>  + it only happens on machines with one (virtual) cpu core.

> while I do agree that these are *not* good criterias to justify whether
> something is *not* RC, I fear there are bugs for which this is true,
> however the justification why they are not RC is different.

> And this is why patches to policy are hard :)

Also, Policy is not the tool for declaring things RC.  Policy states
whether or not things are bugs, and draws a must vs. should distinction
that is the *default* for RCness, but I don't think anyone has been
arguing about the default.  Most of this discussion has either been about
general principles (and that probably hasn't been hugely productive) or
about whether an exception should be made for one reason or another for
some specific bug that Santiago found.

Those exceptions are exactly why the release team sets RC criteria
somewhat independently of Policy, so that the practical implications of
making a bug RC can be considered separately from the general question of
what our default requirements for the archive are.  Sometimes exceptions
are needed based on release timing or impact to the release, but should be
evaluated by a human who can use reasonable discretion.

That said, Policy doesn't clearly state in the expected place that
packages are required to build from source, probably mostly because it's
been one of those base assumptions that everyone working on Policy shared.
I think it would make sense to be more explicit and obvious about this in
Policy.  If someone wants to take a crack at that, probably for section 4
before 4.1 since it's really fundamental to the whole concept of source
packages, I'd be happy to review.  (I could have sworn that we had an open
bug about this and some requirement other than the DFSG stuff in 2.2, but
I wasn't able to find either, which underscores that this would be
useful.)

-- 
Russ Allbery (r...@debian.org)   

Re: Help requested: Packages which FTBFS randomly

[Christoph Biedl]

Santiago Vila wrote...

> I fully agree with the underlying idea, however: If we can measure the
> failure rate, then it means it already fails too often to be acceptable.

Cannot deny I somehow like that approach.

> For that to happen, the around 50 packages which FTBFS randomly should
> do so less than 1% of the time (I'm assuming here that all the others
> "never" fail to build).
> 
> I think this is feasible, but only if we start not allowing
> (i.e. making RC) things that we seem to be currently allowing.

We still could make this a buster release goal but stretch-ignore some
packages; at least those with a failure rate below five percent.

> BTW: Could anybody tell me when exactly "FTBFS on a single-CPU machine"
> stopped being serious and RC? Did such thing ever happened?

Wasntme, and I doubt this is a good idea. And although nobody likes the
bringer of bad news, I'm glad people like you build-test the Debian
archive in a setup that is a bit off mainstream but not completely
unrealistic. While this means asking for trouble I consider it a good
idea to identify corner cases while they are in the corner.

Christoph


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Christoph Biedl]

Niels Thykier wrote...

[ topic shift ]

> On a related note: Having some way to declare minimum requirements for
> e.g. disk space and memory (a la "base GB usage + GB usage/core") used
> would be great.
>   Especially if it is available in metadata, so wanna-build can see
> whether it makes sense to assign a given package to a given build-node.

This is a charming idea altough I have doubt it will work out: As
usual the information has to be kept up-to-date, so unless it is
collected and verified every now and then automatically, it will
become unsuable pretty soon.

Otherwise, there are more things I could use in such a buildd routing
table: I remember I've seen packages failing to build

* when using eatmydata
* on sbuild using overlayfs
* using a qemu build chroot (Debian doesn't do this, other might)
* with /tmp on tmpfs on some archs

and it was nice I could reconfigure a (private) buildd's behaviour on
a per-package base, especially for those that could not be fixed by a
patch. This would allow me to continue using optimizations for the
packages that are less pick^W demanding about the build environment.

Christoph


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Holger Levsen]

On Mon, Feb 20, 2017 at 09:59:29PM +, Ian Jackson wrote:
> My advice would be to avoid asking the TC such general questions.

My advice too.

> There are two sensible questions:
> 
>  * Should the following bugs be RC ?
> [ list of 20 bugs or whatever ]
 
with a list of 20 bugs I dont think it's sensible. With one, yes, sure. With
two, probably…

>  * I proposed the following patch to policy. 

I haven't seen this yet, or rather I haven't heard that this was done yet.

>Despite it being
>seconded,

So this has neither happened…

>the policy editors have not responded to my request
>to amend policy. [or whatever]

… so this neither…

>  + A FTBFS bug is still RC even if:
>  + it doesn't happens on buildd.debian.net;
>  + it only happens randomly;
>  + it only happens on machines with one (virtual) cpu core.

while I do agree that these are *not* good criterias to justify
whether something is *not* RC, I fear there are bugs for which this is true,
however the justification why they are not RC is different.

And this is why patches to policy are hard :)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Ian Jackson]

Santiago Vila writes ("Re: Help requested: Packages which FTBFS randomly"):
> Maybe this one for a start?:
> 
>  Should building Debian source packages on a single-CPU machine be
>  supported at the same level as building on a multi-core machine, to
>  the point that a FTBFS bug on a single-CPU machine should be serious
>  regardless of what happens on buildd.debian.org? (The RM can still
>  stretch-ignore if they wish).
> 
> Do we really need a ruling from the TC regarding this?

My advice would be to avoid asking the TC such general questions.
The TC is wholly incapable of answering them coherently.  It lacks the
will to solve general problems.

There are two sensble questions:

 * Should the following bugs be RC ?

[ list of 20 bugs or whatever ]

   [As far as I am aware] all of these bugs have the following
   properties:
 [ whatever commonality you think is important ]

   I have asked the release team but they
   have declined to decide until after stretch is relased.

Or

 * I proposed the following patch to policy.  Despite it being
   seconded, the policy editors have not responded to my request
   to amend policy. [or whatever]

 + A FTBFS bug is still RC even if:
 + it doesn't happens on buildd.debian.net;
 + it only happens randomly;
 + it only happens on machines with one (virtual) cpu core.

Ian.

Re: Help requested: Packages which FTBFS randomly

[Ian Jackson]

Vincent Bernat writes ("Re: Help requested: Packages which FTBFS randomly"):
>  ❦ 20 février 2017 21:13 GMT, Ian Jackson <ijack...@chiark.greenend.org.uk> :
> > With the correct infrastructure (which is not that hard) the causes
> > can be completely eliminated.  I don't ever experience random build
> > failures of any of my own packages and if I did I would hunt them down
> > with a vengeance.[1]
> 
> Good for you.
> 
> https://tests.reproducible-builds.org/debian/rbuild/unstable/armhf/autopkgtest_4.3.rbuild.log

Thanks for the reminder that I should probably ask to be removed from
Maintainers for that package.  Martin Pitt has been doing most of the
work, I don't seem to be seeing eye to eye with him lately.[1]]

> It's also quite easier to not have a problems with reliable tests when
> most of your packages don't have tests.

I didn't say all my tests were reliable.  Did you miss my mention of
the dgit test suite ?  I know quite what it's like to have a heisenbug
in the test suite, in a program which you don't maintain, and whose
upstream denies that there can possibly be any races.[2]

There is no excuse for leaving an unreliable test as an FTBFS.
They're a nuisance for QA folks, trouble for downstreams, and a
landmine for users.

Ian.

[1] https://bugs.debian.org/833407

[2] Luckily another upstream developer eventually came to the rescue
and now the failure probability is down to about one in 10-20 runs of
dgit's adt tests.

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Mon, Feb 20, 2017 at 09:22:35PM +, Ian Jackson wrote:

> If you do not get good answers, please take this to the TC.[1]

Thanks a lot for your support, Ian.

What kind of question do you think I could make to the TC?

Maybe this one for a start?:

 Should building Debian source packages on a single-CPU machine be
 supported at the same level as building on a multi-core machine, to
 the point that a FTBFS bug on a single-CPU machine should be serious
 regardless of what happens on buildd.debian.org? (The RM can still
 stretch-ignore if they wish).

Do we really need a ruling from the TC regarding this?

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Vincent Bernat]

 ❦ 20 février 2017 21:13 GMT, Ian Jackson  :

> With the correct infrastructure (which is not that hard) the causes
> can be completely eliminated.  I don't ever experience random build
> failures of any of my own packages and if I did I would hunt them down
> with a vengeance.[1]

Good for you.

https://tests.reproducible-builds.org/debian/rbuild/unstable/armhf/autopkgtest_4.3.rbuild.log

It's also quite easier to not have a problems with reliable tests when
most of your packages don't have tests.
-- 
"... all the modern inconveniences ..."
-- Mark Twain


signature.asc
Description: PGP signature

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Russ Allbery]

Ian Jackson  writes:
> Russ Allbery writes:

>> The point is that they don't randomly fail in the sense that they don't
>> fail n% of the time when run in any possible build environment.
>> Rather, in the subset of cases we're talking about in this thread, the
>> tests work reliably on the developer's machine and on the buildd
>> network, but fail either reliably or randomly in other build
>> environments.
>> 
>> This is, in a sense, an unreliable test, but it's not unreliable in a
>> way that directly affects the main line of package development.

> I find this argument, from a Debian Developer, utterly outrageous.

I find it mildly irritating that you are attributing to me some sort of
argument, when instead I made several factual statements and carefully and
very intentionally refused to draw a conclusion from them.  I was simply
trying to provide some additional context because I thought two thread
participants were talking past each other and not talking about the same
thing.

Nowhere in the above did I say that we shouldn't care about the bugs
because they don't directly affect the main line of package development.

In retrospect, I could have made it clearer that I was doing that and that
I wasn't intending to hint at an argument, but also please extend to other
people the benefit of the doubt.

-- 
Russ Allbery (r...@debian.org)   

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Ian Jackson]

Russ Allbery writes ("Re: aren't unreliable tests worse than none? (Re: Help 
requested: Packages which FTBFS randomly)"):
> The point is that they don't randomly fail in the sense that they don't
> fail n% of the time when run in any possible build environment.  Rather,
> in the subset of cases we're talking about in this thread, the tests work
> reliably on the developer's machine and on the buildd network, but fail
> either reliably or randomly in other build environments.
> 
> This is, in a sense, an unreliable test, but it's not unreliable in a way
> that directly affects the main line of package development.

I find this argument, from a Debian Developer, utterly outrageous.

We are supposed, in Debian, not just to care about what affects us
directly.  We are supposed to be acting in the interests of our users
and our downstreams.

We don't overlook licence problems that have Debian-specific
exceptions.  Neither should we not overlook an RC bug, on the grounds
that it doesn't "directly [affect] the main line of [Debian] package
development."

Ian.
(totally boggled)

Re: Help requested: Packages which FTBFS randomly

[Ian Jackson]

Santiago Vila writes ("Re: Help requested: Packages which FTBFS randomly"):
> I don't think that's what we want.

Picking on this message at random:

Many of the messages in this thread are IMO defending the
indefensible, using unreasonable arguments.  Reading them, and some of
the bug reports you are dealing with, are making me quite cross.

Santigo, you are 100% right.  Please keep up the good work.
If you do not get good answers, please take this to the TC.[1]

Ian.

[1] Maybe the TC will surprise me again by being, eventually, useful !

Re: Help requested: Packages which FTBFS randomly

[Ian Jackson]

Christoph Biedl writes ("Re: Help requested: Packages which FTBFS randomly"):
> Ian Jackson wrote...
> > IMO all of these bugs should be RC.  A randomly-reproducible build
> > failure with more than negligible probabilty is likely to show up for
> > some of Debian's users and downstreams and cause them mysterious
> > trouble.  It also causes trouble for stalwarts like Santiago, doing
> > much needed and largely-unloved QA work.
> 
> Tracking down race conditions in the build that occur every now and then
> only isn't joy either.

Well, yes, but if there is some difficult work to be done it should be
done by those who care about the package, not by those who are doing
us all a favour by finding the problems.

> At least some of the packages have the random failures in the test
> suite. If this justifies an RC, the maintainer will rather disable the
> particular test, or even the entire test suite. Something I consider
> the much worse scenario than having to trigger a rebuild.

Most downstream folks who trip over something like this are not going
to think "oh it must be a heisenbug I will just try it again".  They
will wonder what they did wrong.  If the failure probability is
nontrivial they will start flailing, producing and discarding false
theories about the cause.

Far better to disable the problematic test.

> Random build failures do happen, for reasons inside or outside the
> packages.

This is simply not true.  "Random failures" do not "just happen".
They have causes.

With the correct infrastructure (which is not that hard) the causes
can be completely eliminated.  I don't ever experience random build
failures of any of my own packages and if I did I would hunt them down
with a vengeance.[1]

> So downstream (also: security and point release builds) must
> be prepared for a failing build anyway. Therefore I'd rather accept
> random build failures, grudgingly, with the threshold set to somewhere
> between 5% and 10%. Rationale: Build may fail one time, the second one
> should very likely succeed then.

I don't think that security and point release builds should be
prepared to retry failing builds.

IMO any failing build in a security, point release, or QA activity,
should immediately produce an RC bug.

Ian.

[1] Don't get me started on gnupg vs the dgit test suite, but of
that's not an FTBFS.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Santiago Vila]

On Mon, Feb 20, 2017 at 06:46:25PM +, Holger Levsen wrote:
> On Mon, Feb 20, 2017 at 10:29:28AM -0800, Russ Allbery wrote:
> > The point is that they don't randomly fail in the sense that they don't
> > fail n% of the time when run in any possible build environment.

We don't really know. Some FTBFS-randomly bugs *are* that way indeed:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838828

In that one, unless you cheat with the random seed, the probability of
failure is mathematically guaranteed.

> …but point taken, not all FTBFS bugs are RC(!) as
> <20170220152410.3mkm5tebg5i2y...@perpetual.pseudorandom.co.uk> nicely
> explained and whose explainations where mostly ignored by Santiago in his
> reply…

I agree that not all FTBFS bugs are RC, but I reject the idea that we
can take buildd.debian.org as the only and single criteria to decide
about RC-ness as some people seem to do.

This will make bugs not RC "because my computer is too slow" or
"because my computer is too fast" or "because my computer is already
running the kernel of stretch", or "because I didn't installed this
package which is no longer essential", none of which I consider
acceptable reasons to downgrade a FTBFS bug.

Thanks.

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Russ Allbery]

Holger Levsen  writes:
> On Mon, Feb 20, 2017 at 10:29:28AM -0800, Russ Allbery wrote:

>> This is, in a sense, an unreliable test, but it's not unreliable in a
>> way that directly affects the main line of package development.

> until someone affected wants to contribute…

Sure.  I think it's a hard classification question and I'm not fully
comfortable with any of the answers given.  For my own packages, I've
tried to stomp these bugs out entirely so that I don't have to worry about
whether they're RC or not (but was aided by the fact that I've mostly run
into them in cases where I'm also upstream, and as upstream I have a
committment to ensuring the test suite works on any platform on which it's
reasonable to install the software itself).  I've not run into any really
hard ones yet, though.

-- 
Russ Allbery (r...@debian.org)   

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Holger Levsen]

On Mon, Feb 20, 2017 at 10:29:28AM -0800, Russ Allbery wrote:
> The point is that they don't randomly fail in the sense that they don't
> fail n% of the time when run in any possible build environment.

…but point taken, not all FTBFS bugs are RC(!) as
<20170220152410.3mkm5tebg5i2y...@perpetual.pseudorandom.co.uk> nicely
explained and whose explainations where mostly ignored by Santiago in his
reply…


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Holger Levsen]

On Mon, Feb 20, 2017 at 10:29:28AM -0800, Russ Allbery wrote:
> This is, in a sense, an unreliable test, but it's not unreliable in a way
> that directly affects the main line of package development.
 
until someone affected wants to contribute…


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Russ Allbery]

Holger Levsen  writes:

> While I agree with Niels that it would be very worthwhile to be able to
> define ressource requirements for a package to build (and thus know I
> have to life with some packages having trouble sometimes) I find it
> *very* strange to be content with test suites which randomly fail.

The point is that they don't randomly fail in the sense that they don't
fail n% of the time when run in any possible build environment.  Rather,
in the subset of cases we're talking about in this thread, the tests work
reliably on the developer's machine and on the buildd network, but fail
either reliably or randomly in other build environments.

This is, in a sense, an unreliable test, but it's not unreliable in a way
that directly affects the main line of package development.

-- 
Russ Allbery (r...@debian.org)   

Re: Help requested: Packages which FTBFS randomly

[Jonathan Dowland]

On Mon, Feb 20, 2017 at 11:45:00AM +0100, Vincent Bernat wrote:
> Time is a limited resource and we need to set our priorities. Having
> test suites that work 100% of the time with constrained resources is not
> a goal I find worthy of the time I can spend on Debian. Unfortunately,
> those bugs are popping up as RC and I have the choice between
> downgrading them and appear as a bad player or fix them.

The pragmatic solution should also be a quick one (depending on the specifics
of the test suite, of course): adjust things so the failing test does not 
cause the test suite to fail: then, the builds aren't broken; the test still
runs, the bug still exists but it is not RC.

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Santiago Vila]

On Mon, Feb 20, 2017 at 06:10:23PM +0100, Vincent Bernat wrote:
>  ❦ 20 février 2017 13:44 GMT, Holger Levsen  :
> 
> >> As a rule of thumb, upstream usually knows better than me which tests
> >> are important. Tests are quite important for the packager to know if
> >> they didn't make an obvious mistake when updating a package (e.g new
> >> dependency missing, something easy to miss by testing manually). Test
> >> are quite important in a team as you may have to update the package
> >> while knowing little about it (e.g adding a security patch).
> >
> > I get why tests are useful. I just think unreliable tests do more harm then
> > good. (Mostly because this leads to ignoring those tests completely. BTDT.)
> 
> We are speaking of tests that succeed reliably on the packager's own builder.

We are also speaking of tests which are well known to be unreliable
under certain circumstances but the maintainer refuses to acknowledge
it or does not care at all.

The rygel bug comes to mind (#841098). Again, because "it does not
happen in buildd.debian.org", except that it was not even true in this
case and it actually happened in buildd.debian.org.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Bastien Roucaries]

Le 20 février 2017 16:55:49 GMT+01:00, Julien Cristau  a 
écrit :
>On 02/20/2017 07:57 AM, Niels Thykier wrote:
>> Vincent Bernat:
>>> [...]
>>>
>>> [...] The policy doesn't state that a package
>>> must build when there is not enough disk space or memory. Maybe it
>would
>>> be far simpler to allow packages to fail to build if there is not
>enough
>>> CPUs.
>>>
>> 
>> On a related note: Having some way to declare minimum requirements
>for
>> e.g. disk space and memory (a la "base GB usage + GB usage/core")
>used
>> would be great.
>>   Especially if it is available in metadata, so wanna-build can see
>> whether it makes sense to assign a given package to a given
>build-node.
>> 
>> That way we could presumably fix most of the "resource
>> exhausted"-failures by not over-committing resources on the buildds.
>> 
>I think most of the "resource exhausted" failures are either "buildd
>has
>lots of old build trees that need to be cleaned up" or "build exhausted
>virtual address space".  Neither of which would be fixed by the above.


Require fpu to build will be needef for imagemagick...

Bastien
>
>Cheers,
>Julien

-- 
Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.

Re: Help requested: Packages which FTBFS randomly

[Ben Hutchings]

On Mon, 2017-02-20 at 08:30 +, Simon McVittie wrote:
> On Mon, 20 Feb 2017 at 01:00:33 +0100, Santiago Vila wrote:
> > Wait a moment. How we do define "common" when applied to a "build
> > environment"?
> 
> Do we rely on it for Debian to function, or was it set up to determine
> what works (e.g. for QA)? The former is common; the latter might not
> be, and failing there is evidence of a possible RC bug but not
> *necessarily* an RC bug itself.
> 
> Debian is an operating system, not an academic exercise. If a package
> builds successfully reliably enough on buildds, porterboxes, and
> developers' hardware or VMs that we can prepare security updates and
> other urgent patches for it in a reasonable time, then it's close
> enough.

...and downstream distributions' build infrastructure, which might be
different from ours.

[...]
> For packages that sometimes FTBFS due to intermittent test failures, I
> for one would rather have the test coverage than not have it.
[...]

While test coverage is obviously a Good Thing in general, any
unreliable test has negative value and should be fixed, disabled, or
made non-fatal to the build.

Ben.

-- 
Ben Hutchings
If at first you don't succeed, you're doing about average.



signature.asc
Description: This is a digitally signed message part

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Mon, Feb 20, 2017 at 03:24:10PM +, Simon McVittie wrote:
> On Mon, 20 Feb 2017 at 10:41:49 +0100, Santiago Vila wrote:
> > You are somehow trying to equate RC-ness with "it FTBFS in 
> > buildd.debian.org".
> 
> No, I'm saying that a sufficiently repeatable FTBFS on buildd.debian.org
> is effectively release-critical whether Policy says it is or not,

Well, I don't really understand the "whether" part, considering that
Release Policy already says that packages must autobuild without failure.

> because if we can't build the package, we can't release it (or can't
> release security updates for it later, or both).

That's the problem, the excessively pragmatical approach to
buildability. For the release, we seem to care only about *our*
ability to build the package, but: What about the user's ability to
rebuild the package?

We don't just ship binary packages. We ship source packages that the
user might want to modify and rebuild (outside buildd.debian.org).

Consider an extreme case: What would happen if we can build the
package but nobody else could?

The package would have an implicit "build-depends: buildd.debian.org"
and it would be effectively non-free, as nobody else could rebuild it.

However, I can imagine some maintainers that would happily downgrade
the FTBFS bug below serious "because it builds ok in buildd.debian.org".

I don't think that's what we want.

Thanks.

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Vincent Bernat]

 ❦ 20 février 2017 13:44 GMT, Holger Levsen  :

>> As a rule of thumb, upstream usually knows better than me which tests
>> are important. Tests are quite important for the packager to know if
>> they didn't make an obvious mistake when updating a package (e.g new
>> dependency missing, something easy to miss by testing manually). Test
>> are quite important in a team as you may have to update the package
>> while knowing little about it (e.g adding a security patch).
>
> I get why tests are useful. I just think unreliable tests do more harm then
> good. (Mostly because this leads to ignoring those tests completely. BTDT.)

We are speaking of tests that succeed reliably on the packager's own builder.
-- 
Replace repetitive expressions by calls to a common function.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Julien Cristau]

On 02/20/2017 07:57 AM, Niels Thykier wrote:
> Vincent Bernat:
>> [...]
>>
>> [...] The policy doesn't state that a package
>> must build when there is not enough disk space or memory. Maybe it would
>> be far simpler to allow packages to fail to build if there is not enough
>> CPUs.
>>
> 
> On a related note: Having some way to declare minimum requirements for
> e.g. disk space and memory (a la "base GB usage + GB usage/core") used
> would be great.
>   Especially if it is available in metadata, so wanna-build can see
> whether it makes sense to assign a given package to a given build-node.
> 
> That way we could presumably fix most of the "resource
> exhausted"-failures by not over-committing resources on the buildds.
> 
I think most of the "resource exhausted" failures are either "buildd has
lots of old build trees that need to be cleaned up" or "build exhausted
virtual address space".  Neither of which would be fixed by the above.

Cheers,
Julien

Re: Help requested: Packages which FTBFS randomly

[Simon McVittie]

On Mon, 20 Feb 2017 at 10:41:49 +0100, Santiago Vila wrote:
> You are somehow trying to equate RC-ness with "it FTBFS in buildd.debian.org".

No, I'm saying that a sufficiently repeatable FTBFS on buildd.debian.org
is effectively release-critical whether Policy says it is or not,
because if we can't build the package, we can't release it (or can't
release security updates for it later, or both).

There are many other forms of release-critical bug. Not all of them
are FTBFS, and (perhaps more controversially) not all FTBFS are
release-critical.

Is it a bug that the packages you're talking about sometimes fail
their tests? Absolutely. I'm not arguing that it isn't - it indicates
that either the code under test is wrong, or the test is wrong, or
perhaps even both. Given enough developer time, it should be fixed;
but given enough developer time, we'd have all sorts of nice things.
A release-critical bug is a really big hammer, and I think it's
important to distinguish between fixes we *want*, and fixes we *need*.

By asking for these bugs to be escalated to RC, you're essentially
saying that they are bad enough bugs that Debian would be a better
operating system for its users and developers if the affected packages
were removed than if they were left in unfixed; or if they have too
many dependencies to consider that, then you're essentially saying that
Debian 9 cannot be released until someone fixes those bugs. If Debian's
official infrastructure, which provides production binaries to our users,
can and does build these packages, then that seems like it might be
an exaggeration.

Again, I am not saying these bugs aren't bugs, merely that we have no
shortage of bugs, and perhaps some of the others are a more effective
use of our developers' limited supplies of time and patience.

> And the couterexample has been provided by yourself: Bugs about tzdata
> and lsb-base were declared RC by a Release Manager well before they
> were removed from buildds (thanks to Lucas Nussbaum).

Sure; the release managers made the judgement that those bugs
*are* problematic for the release. They make the release, they make
the rules for what they are willing to put in it.

> Does this mean we were doing an "academic exercise"?

In the case of tzdata and lsb-base, presumably they were removed from
buildd chroots because someone had a practical reason to want them to be;
and before they were removed, someone did the work to determine how
much fallout there was going to be, which is the best estimate we're
likely to get of the effort required to fix that fallout.

Without knowing the circumstances in detail, I don't know whether
the effort/benefit trade-off between the effort required to fix those
bugs was worth the benefit we derived from removing those packages
from the minimal chroot. Someone presumably concluded that it was
worth it. If the conclusion had been that it wasn't worth it, then
the answer would have instead been to add tzdata or lsb-base
to Essential, Build-Essential or some similar set of
guaranteed/assumed packages.

S

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Holger Levsen]

On Mon, Feb 20, 2017 at 01:46:20PM +0100, Vincent Bernat wrote:
> As a rule of thumb, upstream usually knows better than me which tests
> are important. Tests are quite important for the packager to know if
> they didn't make an obvious mistake when updating a package (e.g new
> dependency missing, something easy to miss by testing manually). Test
> are quite important in a team as you may have to update the package
> while knowing little about it (e.g adding a security patch).

I get why tests are useful. I just think unreliable tests do more harm then
good. (Mostly because this leads to ignoring those tests completely. BTDT.)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Vincent Bernat]

 ❦ 20 février 2017 11:03 GMT, Holger Levsen  :

>> Time is a limited resource and we need to set our priorities. Having
>> test suites that work 100% of the time with constrained resources is not
>> a goal I find worthy of the time I can spend on Debian.
>
> While I agree with Niels that it would be very worthwhile to be able to define
> ressource requirements for a package to build (and thus know I have to life
> with some packages having trouble sometimes) I find it *very* strange to be
> content with test suites which randomly fail.
>
> How do you know an error in a testsuite is a non-critical one which can be
> ignored? *Especialy* if you have flaky tests, how can you be sure (or even
> guesstimate) a test failure is harmless one to ignore and not a critical one
> which needs acting upon???

As a distribution, our primary goal is to deliver software from upstream
to users. It is great if we deliver better software, but we don't have
to fix all bugs. Test suite runs fine in my pbuilder environment. Test
suite runs fine on Travis-CI (or whatever CI system upstream is
using).

Moreover, it's not about ignoring bugs without looking at them, it is
about making them non-RC or discard them if they are bad luck (happened
once) or specific to the build environment.

> I *really* don't get why people advocate keeping unreliable tests enabled in
> releases.

As a rule of thumb, upstream usually knows better than me which tests
are important. Tests are quite important for the packager to know if
they didn't make an obvious mistake when updating a package (e.g new
dependency missing, something easy to miss by testing manually). Test
are quite important in a team as you may have to update the package
while knowing little about it (e.g adding a security patch).
-- 
Document your data layouts.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Mon, Feb 20, 2017 at 11:33:23AM +, Ghislain Vaillant wrote:

> I share the same feelings towards a similar intermittent FTBFS with
> src:python-qtpy (#8544936). I admit I have no clue what is going on,
> neither does upstream, nor does the reporter (Santiago).

That would be #854496. It also fails here:

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/python-qtpy.html

Believe me, this is also frustrating for me. If you absolutely need a
machine to reproduce this, contact me privately.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Ghislain Vaillant]

On Mon, 2017-02-20 at 11:45 +0100, Vincent Bernat wrote:
>  ❦ 20 février 2017 10:05 GMT, Jonathan Dowland  :
> 
> > None of the FTBFS problems I've seen in this thread have been because the
> > tests *required* multiple cores, by the way; more so that they were racy
> > or buggy in some other fashion. If uniprocessor buildds are finding these
> > bugs then that's a virtue, IMHO.
> 
> Time is a limited resource and we need to set our priorities. Having
> test suites that work 100% of the time with constrained resources is not
> a goal I find worthy of the time I can spend on Debian. Unfortunately,
> those bugs are popping up as RC and I have the choice between
> downgrading them and appear as a bad player or fix them.

I share the same feelings towards a similar intermittent FTBFS with
src:python-qtpy (#8544936). I admit I have no clue what is going on,
neither does upstream, nor does the reporter (Santiago).

Ghis

Re: Help requested: Packages which FTBFS randomly

[Christian Seiler]

On 02/20/2017 11:05 AM, Jonathan Dowland wrote:
> On Mon, Feb 20, 2017 at 01:57:52AM +0100, Adam Borowski wrote:
>> * single-CPU machines have gone the way of the dodo.  Even the crummiest
>>   machine I could find while dumpster-diving looking for a non-sse3 one
>>   already has HT and builds your examples successfully.  Same for ARM SoCs
>>   -- my RPi1 is kaputt, and anything newer has multiple cores.  This, I'd
>>   say it's a waste of time to care about _building_ on single CPU.
> 
> /me looks lovingly at the PowerPC G4 Mac Mini propping up his display. I
> wonder if I should consider registering as a porter...
> 
> How uncommon are single CPU virtual machines? Or containers with resource
> constraints to one CPU or something similar? I think that is at least as
> relevant as the availability of (new) uniprocessor hardware.
> 
> None of the FTBFS problems I've seen in this thread have been because the
> tests *required* multiple cores, by the way; more so that they were racy
> or buggy in some other fashion. If uniprocessor buildds are finding these
> bugs then that's a virtue, IMHO.

Also, who's to say that tests failing on uniprocessor machines are buggy?
Maybe the tests are fine and they're exposing a bug in the actual
software or library that might fail in the same way if the circumstances
are just right?

If a failure happens on uniprocessor machines, who's not to say they don't
also happen on multicore machines that are under heavy load? For the same
code to fail on uniprocessor machines but succeed on multicore I would
expect there to be some kind of race condition that is triggered if
program execution is serialized to a single processor. But that could also
happen on multicore if that's under heavy load. Who's not to say that a
bug like this could not also be triggered on a new hardware, because a
new CPU generation behaves slightly differently?

I do believe that packages that FTBFS consistently on unprocessor
machines should be considered RC-buggy. We can argue about whether the
corresponding bugs should be marked as stretch-ignore, but I don't
believe that this something we want to have in the long term.

Regards,
Christian

aren't unreliable tests worse than none? (Re: Help requested: Packages which FTBFS randomly)

[Holger Levsen]

On Mon, Feb 20, 2017 at 11:45:00AM +0100, Vincent Bernat wrote:
> Time is a limited resource and we need to set our priorities. Having
> test suites that work 100% of the time with constrained resources is not
> a goal I find worthy of the time I can spend on Debian.

While I agree with Niels that it would be very worthwhile to be able to define
ressource requirements for a package to build (and thus know I have to life
with some packages having trouble sometimes) I find it *very* strange to be
content with test suites which randomly fail.

How do you know an error in a testsuite is a non-critical one which can be
ignored? *Especialy* if you have flaky tests, how can you be sure (or even
guesstimate) a test failure is harmless one to ignore and not a critical one
which needs acting upon???

I *really* don't get why people advocate keeping unreliable tests enabled in
releases.


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Thomas Goirand]

On 02/15/2017 09:45 PM, Santiago Vila wrote:
> Building a package is like following an algorithm. It may take less or
> more time, but it always has to finish. For this reason, "your CPU is
> too slow to build my package" is always a very poor excuse, and I hope
> you are not trying to set a minimum CPU speed to build packages.

On a few cases, I had to increase some test timeouts because of some
people using *very* slow CPUs to build, and complaining about failures.
While I do agree that in theory, packages should be build-able in most
platforms, there's still a limit to that effort.

Let's say you were attempting to build on a 200 MHz x86 CPU, then yes, I
would say it's not reasonable to open bugs and complain about related
problems. Not because there's no bug (there really is, and it should
really be considered a bug), but because we all have a limited amount of
time to spend on each issue, and it's best to focus on more important
things.

Cheers,

Thomas Goirand (zigo)

Re: Help requested: Packages which FTBFS randomly

[Vincent Bernat]

 ❦ 20 février 2017 10:05 GMT, Jonathan Dowland  :

> None of the FTBFS problems I've seen in this thread have been because the
> tests *required* multiple cores, by the way; more so that they were racy
> or buggy in some other fashion. If uniprocessor buildds are finding these
> bugs then that's a virtue, IMHO.

Time is a limited resource and we need to set our priorities. Having
test suites that work 100% of the time with constrained resources is not
a goal I find worthy of the time I can spend on Debian. Unfortunately,
those bugs are popping up as RC and I have the choice between
downgrading them and appear as a bad player or fix them.
-- 
Make it right before you make it faster.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Jonathan Dowland]

On Mon, Feb 20, 2017 at 01:57:52AM +0100, Adam Borowski wrote:
> * single-CPU machines have gone the way of the dodo.  Even the crummiest
>   machine I could find while dumpster-diving looking for a non-sse3 one
>   already has HT and builds your examples successfully.  Same for ARM SoCs
>   -- my RPi1 is kaputt, and anything newer has multiple cores.  This, I'd
>   say it's a waste of time to care about _building_ on single CPU.

/me looks lovingly at the PowerPC G4 Mac Mini propping up his display. I
wonder if I should consider registering as a porter...

How uncommon are single CPU virtual machines? Or containers with resource
constraints to one CPU or something similar? I think that is at least as
relevant as the availability of (new) uniprocessor hardware.

None of the FTBFS problems I've seen in this thread have been because the
tests *required* multiple cores, by the way; more so that they were racy
or buggy in some other fashion. If uniprocessor buildds are finding these
bugs then that's a virtue, IMHO.


-- 
Jonathan Dowland


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Mon, Feb 20, 2017 at 08:30:04AM +, Simon McVittie wrote:
> Debian is an operating system, not an academic exercise. If a package
> builds successfully reliably enough on buildds, porterboxes, and
> developers' hardware or VMs that we can prepare security updates and
> other urgent patches for it in a reasonable time, then it's close
> enough.
> 
> Conversely, packages that don't work on Debian's buildds are likely to
> be de facto unreleasable, even if they work fine on less problematic
> hardware[1] or in "more realistic" build environments[2].
>
> [...]
>
> [2] 
> https://wiki.debian.org/qa.debian.org/FTBFS#A2017-01-29_tzdata_and_lsb-base_no_longer_installed_in_build_chroots

You are somehow trying to equate RC-ness with "it FTBFS in buildd.debian.org".

While that may be close to the truth, it is not the truth itself.

And the couterexample has been provided by yourself: Bugs about tzdata
and lsb-base were declared RC by a Release Manager well before they
were removed from buildds (thanks to Lucas Nussbaum).

Does this mean we were doing an "academic exercise"?

No, we were following the common rules that we have set for ourselves
regarding build-essential and build-depends, and those rules do not
(and should not) depend on a particular build daemon failing or not.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Mon, Feb 20, 2017 at 01:57:52AM +0100, Adam Borowski wrote:

> * single-CPU machines have gone the way of the dodo.  Even the crummiest
>   machine I could find while dumpster-diving looking for a non-sse3 one
>   already has HT and builds your examples successfully.  Same for ARM SoCs
>   -- my RPi1 is kaputt, and anything newer has multiple cores.  This, I'd
>   say it's a waste of time to care about _building_ on single CPU.

Sure, single-CPU machines are extremely difficult to find these days,
but *only* as physical machines.

If you were to make a derived distro, let's say, using different build
flags, and wanted to build all packages, you would probably not buy a
new computer for that, you would rent virtual machines instead.

And those are single-CPU if they are small enough. Building on
multi-core is anti-economic, because a multi-core machine usually
cost twice or four times the price of a single-CPU machine.

"But they take half or a quarter of the time", you will say. Well,
maybe, or maybe not. Some packages benefit from parallelism and some
do not. I've checked that 97% of all packages in stretch may be built
with only 1 GB of RAM. Using a bigger machine is usually a waste of resources.

So yes, single-CPU machines are still very common these days, except
that they are not whole computers anymore, and lots of people use them
every day. Considering them to be unupported "because all physycal
machines are multi-core these days" would be quite silly.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Simon McVittie]

On Mon, 20 Feb 2017 at 01:00:33 +0100, Santiago Vila wrote:
> Wait a moment. How we do define "common" when applied to a "build
> environment"?

Do we rely on it for Debian to function, or was it set up to determine
what works (e.g. for QA)? The former is common; the latter might not
be, and failing there is evidence of a possible RC bug but not
*necessarily* an RC bug itself.

Debian is an operating system, not an academic exercise. If a package
builds successfully reliably enough on buildds, porterboxes, and
developers' hardware or VMs that we can prepare security updates and
other urgent patches for it in a reasonable time, then it's close
enough.

Conversely, packages that don't work on Debian's buildds are likely to
be de facto unreleasable, even if they work fine on less problematic
hardware[1] or in "more realistic" build environments[2].

For packages that sometimes FTBFS due to intermittent test failures, I
for one would rather have the test coverage than not have it. We could
guarantee that no package will ever fail its build-time tests by not
running them, but having some packages intermittently fail to build
seems an acceptable price to pay for having a trivially easy way to
reject builds that are completely non-functional (for instance because
an upgraded dependency has broken their assumptions, or because they
happen to have hit toolchain bugs that produce broken binaries).

Unfortunately, tests are software too, which means the tests have bugs
just like the software under test does, and making them perfect is a
target to aim for rather than a realistic possibility. They still seem
like a good thing to have.

S

[1] http://apebox.org/wordpress/linux/545
[2] 
https://wiki.debian.org/qa.debian.org/FTBFS#A2017-01-29_tzdata_and_lsb-base_no_longer_installed_in_build_chroots

Re: Help requested: Packages which FTBFS randomly

[Niels Thykier]

Vincent Bernat:
> [...]
> 
> [...] The policy doesn't state that a package
> must build when there is not enough disk space or memory. Maybe it would
> be far simpler to allow packages to fail to build if there is not enough
> CPUs.
> 

On a related note: Having some way to declare minimum requirements for
e.g. disk space and memory (a la "base GB usage + GB usage/core") used
would be great.
  Especially if it is available in metadata, so wanna-build can see
whether it makes sense to assign a given package to a given build-node.

That way we could presumably fix most of the "resource
exhausted"-failures by not over-committing resources on the buildds.

Thanks,
~Niels

Re: Help requested: Packages which FTBFS randomly

[Adam Borowski]

On Mon, Feb 20, 2017 at 12:05:42AM +0100, Vincent Bernat wrote:
> Your chosen build environment is not common and fixing build failures
> for uncommon environment may seem a waste of the "Debian-allocated time"
> for some people (including me).

Yeah.  I think your priorities are somewhat backwards:

* I'd say that failing 10% of the time is way too much, and should be RC for
  stretch -- as someone said in this thread, there'll be expected one FTBFS
  in a release arch per security/stable update, and that's a practical
  problem.

* single-CPU machines have gone the way of the dodo.  Even the crummiest
  machine I could find while dumpster-diving looking for a non-sse3 one
  already has HT and builds your examples successfully.  Same for ARM SoCs
  -- my RPi1 is kaputt, and anything newer has multiple cores.  This, I'd
  say it's a waste of time to care about _building_ on single CPU.


Meow!
-- 
Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
  ./configure --host=zx-spectrum --build=pdp11

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Mon, Feb 20, 2017 at 12:05:42AM +0100, Vincent Bernat wrote:
> More and more packages come with test suites to help developers and
> packagers ensure things are working as expected. It would be great if
> test suites didn't have failures of their own but it's better to have
> them and it's understandable that most packagers don't want to blindly
> disable them.

Well, I don't advocate for doing anything "blindly", I just expect
individual tests to be disabled when it's abundantly clear that they
are wrongly designed.

> Your chosen build environment is not common and fixing build failures
> for uncommon environment may seem a waste of the "Debian-allocated time"
> for some people (including me).

Wait a moment. How we do define "common" when applied to a "build
environment"? Number of packages built? Number of people building with
such build environment? What if somebody has a build environment and
builds packages 100 times as often as it's done on buildd.debian.org?
Does such build environment becomes "common" then? I think it should.

I would say, for example, that the build environment used by Lucas
Nussbaum in his archive rebuilds, or the one in reproducible builds
could easily be more "common" than buildd.debian.org, considering
that the same packages are built over and over again.

My own autobuilders take 7-10 days to build all packages, and they are
running all the time. For Arch:all packages I build them a lot more
often than buildd.debian.org, who only build them once when the
maintainer uploads them, or even zero times when the maintainer does
not upload in source-only form.

> The policy doesn't state that a package
> must build when there is not enough disk space or memory.

It would not be reasonable to have a policy for that. Each package
needs what it needs. I think you are comparing apples and oranges here.

> Maybe it would be far simpler to allow packages to fail to build
> if there is not enough CPUs.

The mere concept of "not enough CPUs" is just flawed and wrong.

This is basic Computer Science: Everything you can do with two or more
CPUs, you can do it as well with only one CPU (even if slower).

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Vincent Bernat]

 ❦ 19 février 2017 22:59 +0100, Santiago Vila  :

> I'm pretty sure that I was already a DD when such thing could have
> happened, but right now I can't remember it.
>
> (See #848063 for why I'm asking this).

IMO, you are also pretty aggressive.

More and more packages come with test suites to help developers and
packagers ensure things are working as expected. It would be great if
test suites didn't have failures of their own but it's better to have
them and it's understandable that most packagers don't want to blindly
disable them.

Your chosen build environment is not common and fixing build failures
for uncommon environment may seem a waste of the "Debian-allocated time"
for some people (including me). The policy doesn't state that a package
must build when there is not enough disk space or memory. Maybe it would
be far simpler to allow packages to fail to build if there is not enough
CPUs.
-- 
Don't diddle code to make it faster - find a better algorithm.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Help requested: Packages which FTBFS randomly

[Holger Levsen]

On Sun, Feb 19, 2017 at 10:59:56PM +0100, Santiago Vila wrote:
> BTW: Could anybody tell me when exactly "FTBFS on a single-CPU machine"
> stopped being serious and RC? Did such thing ever happened?
 
I don't think so. (It never stopped being serious.)

If some people disagree I'd say this is life and move on. (Either by
escalating the bug or ignoring the bug, package & those people.)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Sun, Feb 19, 2017 at 06:27:16PM +0100, Christoph Biedl wrote:
> Ian Jackson wrote...
>
> > If there is to be a failure probability threshold I would set it at
> > 10^-4 or so.  After all, computer time is cheap.
> 
> To determine 10^-4 with some accurance you'd have to rebuild that
> package 2 times. This is an amount where computer time isn't cheap
> any longer.

The figure by Ian is very ambitious indeed, but so much that maybe
it's not very realistic.

I fully agree with the underlying idea, however: If we can measure the
failure rate, then it means it already fails too often to be acceptable.

Anybody who tries to build all Debian source packages will find that
the number of packages that fail to build is a random variable,
sometimes it will be 3, sometimes it will be 5, etc.

A nice goal regarding reliability would be that the expected value for
this random variable (the number of failing packages) should be closer
to 0 than to 1 (when you try to build once every single package).

For that to happen, the around 50 packages which FTBFS randomly should
do so less than 1% of the time (I'm assuming here that all the others
"never" fail to build).

I think this is feasible, but only if we start not allowing
(i.e. making RC) things that we seem to be currently allowing.


BTW: Could anybody tell me when exactly "FTBFS on a single-CPU machine"
stopped being serious and RC? Did such thing ever happened?

I'm pretty sure that I was already a DD when such thing could have
happened, but right now I can't remember it.

(See #848063 for why I'm asking this).

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Christoph Biedl]

Ian Jackson wrote...

> IMO all of these bugs should be RC.  A randomly-reproducible build
> failure with more than negligible probabilty is likely to show up for
> some of Debian's users and downstreams and cause them mysterious
> trouble.  It also causes trouble for stalwarts like Santiago, doing
> much needed and largely-unloved QA work.

Tracking down race conditions in the build that occur every now and then
only isn't joy either.

At least some of the packages have the random failures in the test
suite. If this justifies an RC, the maintainer will rather disable the
particular test, or even the entire test suite. Something I consider
the much worse scenario than having to trigger a rebuild.

Random build failures do happen, for reasons inside or outside the
packages. So downstream (also: security and point release builds) must
be prepared for a failing build anyway. Therefore I'd rather accept
random build failures, grudgingly, with the threshold set to somewhere
between 5% and 10%. Rationale: Build may fail one time, the second one
should very likely succeed then.

> If there is to be a failure probability threshold I would set it at
> 10^-4 or so.  After all, computer time is cheap.

To determine 10^-4 with some accurance you'd have to rebuild that
package 2 times. This is an amount where computer time isn't cheap
any longer.

Christoph


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Steve Cotton]

On Wed, Feb 15, 2017 at 06:26:51PM +0100, Santiago Vila wrote:
> The following packages FTBFS for me randomly. First column is the bug
> number, second column is the estimated probability of failure in my
> build environment, which is described here:
> 
> https://people.debian.org/~sanvila/my-building-environment.txt

Hi Santiago and everyone trying to debug these FTBFSes,

For ri-li, and hopefully many of the other bugs on the list, there's a much
simpler way to replicate the bug in developers' normal dev environments, using
the taskset command from package util-linux to run the build on CPU #0 only:

$ taskset --cpu-list 0 dpkg-buildpackage -A

For ri-li, the failure can be isolated from the build:
ri-li/data-src$ taskset --cpu-list 0 xvfb-run -a --server-args="-screen 0 
1024x768x24" ./MakeDat
Impossible d'initialiser SDL:Couldn't open X11 display

Thanks,
Steve

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Fri, Feb 17, 2017 at 06:59:00PM +, Niels Thykier wrote:
> Santiago Vila:
> > On Fri, Feb 17, 2017 at 06:23:00AM +, Niels Thykier wrote:
> > 
> >> Santiago already brought it up in #844264.  I believe my answer in
> >> comment 70 is still relevant (other than I incorrectly used "after the
> >> freeze" when I meant "after the release").
> > 
> > Well, but when I said "Ok, will do" in Bug #844264, it was a reply to
> > your request to postpone this after the freeze, not after the release.
> > 
> 
> I obviously wrote the wrong thing

Actually, it was not obvious for me at all, this is what you wrote:

>  But I do not think we have capacity for that talk right now in the
>  release team (between an incomplete openssl transition and the BTS
>  breaking causing britney to migrate tons of packages despite RC
>  bugs).

You mentioned here two things: openssl transition and the BTS glitch
that made buggy packages to propagate to testing. Those were the
"issues of the day" at the time, and are now mostly in the past.

This, and the fact that no other Release Manager fixed your mistake,
is the reason why I believed you meant "freeze" when you wrote "freeze".

> and I am sorry for creating that misunderstanding.

Ok, but please let us face the consequences: I downgraded all those
bugs to important because I expected you, Release Managers, to give
some guidelines about how to handle them *before* the release.

If this is not going to happen, I'll have to raise those bugs to
serious again, following your own guideline (see Message #35 from
Julien Cristau) that the bugs are serious as a general rule (because
no other guideline was given so far).

If we really want to skip this issue for stretch, then we should use
stretch-ignore, not severity important.

Do I have your permission to use stretch-ignore here?

[ I would do that on any FTBFS-randomly bug present or future, except
  those packages failing more than 50% of the time, because IMO that
  would be completely absurd ].

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Niels Thykier]

Santiago Vila:
> On Fri, Feb 17, 2017 at 06:23:00AM +, Niels Thykier wrote:
> 
>> Santiago already brought it up in #844264.  I believe my answer in
>> comment 70 is still relevant (other than I incorrectly used "after the
>> freeze" when I meant "after the release").
> 
> Well, but when I said "Ok, will do" in Bug #844264, it was a reply to
> your request to postpone this after the freeze, not after the release.
> 

I obviously wrote the wrong thing and I am sorry for creating that
misunderstanding.

> I definitely don't think we should do absolutely nothing in stretch
> about this, so I'll continue to ask for help here in -devel:
> 
> [...]

Note, I did not intend to stop this thread on -devel.  I did intend to
inform Ian of the bug and the release team's view on it, since he
involved -release.

FTR, I am ready to consider unblock requests for "randomly FTBFS" bugs
for stretch on a case-by-case basis.

Thanks,
~Niels

Re: Help requested: Packages which FTBFS randomly

[Adam Borowski]

On Fri, Feb 17, 2017 at 08:54:54AM -0600, Jeff Epler wrote:
> On Fri, Feb 17, 2017 at 12:16:16PM +0100, Santiago Vila wrote:
> > Can anyone here reproduce any of the following *two* bugs?
> > (Using sbuild in a single-CPU machine)
> > 
> > 837067  1.000   libsecret
> 
> Using a fresh 1CPU / 1GB x86 system on digitalocean's NYC region, upgraded to
> debian testing, I tried sbuilding libsecret.  So far, 2/2 builds failed.

On the other hand, on the crummiest machine I could find, a real i386 (an
early pre-nx Pentium 4), succeeded 5/5 libsecret and 5/5 ri-li.  It's HT
though, so it can (slowly) run two threads at once.


Meow!
-- 
Autotools hint: to do a zx-spectrum build on a pdp11 host, type:
  ./configure --host=zx-spectrum --build=pdp11

Re: Help requested: Packages which FTBFS randomly

[Holger Levsen]

On Fri, Feb 17, 2017 at 03:34:08PM +, Holger Levsen wrote:
> https://tests.reproducible-builds.org/libsecret looks better:

I forgot to mention this (kind of) URL:

https://tests.reproducible-builds.org/debian/history/libsecret.html 
 

-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Holger Levsen]

On Fri, Feb 17, 2017 at 08:54:54AM -0600, Jeff Epler wrote:
> On Fri, Feb 17, 2017 at 12:16:16PM +0100, Santiago Vila wrote:
> > Can anyone here reproduce any of the following *two* bugs?
> > (Using sbuild in a single-CPU machine)
> > 837067  1.000   libsecret
> 
> Using a fresh 1CPU / 1GB x86 system on digitalocean's NYC region, upgraded to
> debian testing, I tried sbuilding libsecret.  So far, 2/2 builds failed.

https://tests.reproducible-builds.org/libsecret looks better:

- 7x success
- 1x ftbfs

(actually, that's 13 successful builds and one failure, as we always build 
twice…)

https://tests.reproducible-builds.org/debian/logs/testing/armhf/libsecret_0.18.5-2.build2.log.gz
failed with:

FAIL: test-session 4 /session/ensure-async-aes
**
ERROR:libsecret/test-session.c:167:test_ensure_async_aes: assertion failed: 
(G_IS_ASYNC_RESULT (result))

(These systems have more cores and more memory too, but also do several builds
at the same time…)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Jeff Epler]

On Fri, Feb 17, 2017 at 12:16:16PM +0100, Santiago Vila wrote:
> Can anyone here reproduce any of the following *two* bugs?
> (Using sbuild in a single-CPU machine)
> 
> 837067  1.000   libsecret

Using a fresh 1CPU / 1GB x86 system on digitalocean's NYC region, upgraded to
debian testing, I tried sbuilding libsecret.  So far, 2/2 builds failed.

# grep ^FAIL libsecret_*.build
libsecret_i386-2017-02-17T14:42:53Z.build:FAIL: test-collection 18 
/collection/delete-sync
libsecret_i386-2017-02-17T14:50:00Z.build:FAIL: test-collection 18 
/collection/delete-sync

Jeff

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Fri, Feb 17, 2017 at 06:23:00AM +, Niels Thykier wrote:

> Santiago already brought it up in #844264.  I believe my answer in
> comment 70 is still relevant (other than I incorrectly used "after the
> freeze" when I meant "after the release").

Well, but when I said "Ok, will do" in Bug #844264, it was a reply to
your request to postpone this after the freeze, not after the release.

I definitely don't think we should do absolutely nothing in stretch
about this, so I'll continue to ask for help here in -devel:

Let's consider only the most blatant cases, the ones at the top of the
list I posted before.

Can anyone here reproduce any of the following *two* bugs?
(Using sbuild in a single-CPU machine)

837067  1.000   libsecret
848063  0.990   ri-li

If we could only fix those two for stretch, we will be proudly able to
say, when released as stable, that no package in stretch does fail to
build more than 50% of the time (in a single-CPU machine).

Then we could think of thresholds lower than 50% for buster, but at
least we would avoid the Schrödinger paradox in stretch.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Niels Thykier]

Ian Jackson:
> Santiago Vila writes ("Help requested: Packages which FTBFS randomly"):
>> The following packages FTBFS for me randomly. First column is the bug
>> number, second column is the estimated probability of failure in my
>> build environment, which is described here:
> 
> [...]
> 
> To the release team: please would you provide a clear answer to
> Santiago's question.  In particular, please provide an answer (or a
> rule which can be used to answer) to each of the 28 bugs mentioned in
> Santiago's mail.  If you think it will take you a while to answer the
> question, please say when you think you will have an answer.
> 
> Santiago: please keep up the good work.
> 
> Thanks,
> Ian.
> 

Hi,

Santiago already brought it up in #844264.  I believe my answer in
comment 70 is still relevant (other than I incorrectly used "after the
freeze" when I meant "after the release").

Thanks,
~Niels

Re: Help requested: Packages which FTBFS randomly

[Jonathan Dowland]

On Thu, Feb 16, 2017 at 10:32:04PM +, Jonathan Dowland wrote:
> Thanks for doing this.

...despite alarming hostility from your fellow developers (as I read
the bugs in more detail, in growing horror... #844264 in particular)



-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Jonathan Dowland]

This is really intriguing. I suspected your build environment must
have been weird, but it looks fine to me. The likely elephant in the
room is the single processor. I'm pretty amazed that this seems to be
a stumbling block, but there we go.

Thanks for doing this.


signature.asc
Description: Digital signature

Re: Help requested: Packages which FTBFS randomly

[Ian Jackson]

Santiago Vila writes ("Help requested: Packages which FTBFS randomly"):
> The following packages FTBFS for me randomly. First column is the bug
> number, second column is the estimated probability of failure in my
> build environment, which is described here:

IMO all of these bugs should be RC.  A randomly-reproducible build
failure with more than negligible probabilty is likely to show up for
some of Debian's users and downstreams and cause them mysterious
trouble.  It also causes trouble for stalwarts like Santiago, doing
much needed and largely-unloved QA work.

If there is to be a failure probability threshold I would set it at
10^-4 or so.  After all, computer time is cheap.

To the release team: please would you provide a clear answer to
Santiago's question.  In particular, please provide an answer (or a
rule which can be used to answer) to each of the 28 bugs mentioned in
Santiago's mail.  If you think it will take you a while to answer the
question, please say when you think you will have an answer.

Santiago: please keep up the good work.

Thanks,
Ian.

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Wed, Feb 15, 2017 at 03:02:23PM -0500, Jeremy Bicha wrote:
> On Wed, Feb 15, 2017 at 2:36 PM, Santiago Vila  wrote:
> > Allowing packages to fail 50% of the time is interpreting Release
> > Policy in a somewhat twisted way.
> 
> Except that your build system is far more limited than the average
> system used to build packages in 2017.

If you know for sure that any of the bugs above happen because of me
not using enough RAM, you are welcome to prove it.
(in the nnn...@bugs.debian.org address, please).

But if you talk about number of CPUs or CPU speed, that's a completely
different issue.

Building a package is like following an algorithm. It may take less or
more time, but it always has to finish. For this reason, "your CPU is
too slow to build my package" is always a very poor excuse, and I hope
you are not trying to set a minimum CPU speed to build packages.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Wed, Feb 15, 2017 at 08:38:17PM +0100, Bernd Zeimetz wrote:
> Hi,
> 
> > The following packages FTBFS for me randomly. First column is the bug
> > number, second column is the estimated probability of failure in my
> > build environment, which is described here:
> > 
> > https://people.debian.org/~sanvila/my-building-environment.txt
> 
> 
> did you try to diff the build output? Did the packages fail with the
> same error running the same command?
> Are there other differences in the output?

I was busy reporting bugs and there were so many of them that I had
little time to debug them myself. That's why I'm asking for help.
I can't do this alone anymore.

The build logs are out there, just take a look at the bugs themselves.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Jeremy Bicha]

On Wed, Feb 15, 2017 at 2:36 PM, Santiago Vila  wrote:
> Allowing packages to fail 50% of the time is interpreting Release
> Policy in a somewhat twisted way.

Except that your build system is far more limited than the average
system used to build packages in 2017.

Thanks,
Jeremy Bicha

Re: Help requested: Packages which FTBFS randomly

[Santiago Vila]

On Wed, Feb 15, 2017 at 08:05:51PM +0100, Thibaut Paumard wrote:
> Dear Santiago,
> 
> Le 15/02/2017 à 18:26, Santiago Vila a écrit :
> > Hello.
> > 
> > The following packages FTBFS for me randomly. First column is the bug
> > number, second column is the estimated probability of failure in my
> > build environment, which is described here:
> > 
> > https://people.debian.org/~sanvila/my-building-environment.txt
> > 
> > Before I ask the Release Managers that they make the bugs at the top
> > of this list serious again
> 
> If you do that, please at the same time ask the release team to mark
> them stretch-ignore,

No, that's not the plan. Please read Bug #844264 for details.

Those bugs are already (currently) being stretch-ignored, because I
downgraded them all to important *temporarily*.

But the idea is to *not* ignore them if they fail more than a certain
threshold (to be determined by the Release Managers).

Upgrading to serious and marking them with stretch-ignore would be
quite pointless. Certainly, I did not bother to report all those FTBFS
bugs so that they are ignored.

The bugs are FTBFS bugs. Ideally, all those bugs would be serious and
we would use the stretch-ignore tag as an exception, precisely when
the package does not fail "often enough".

> unless you are sure the test failures indicate a
> real serious bug in the package.

I am sure that Release Policy says "packages must autobuild from
source and without failure".

Allowing packages to fail 50% of the time is interpreting Release
Policy in a somewhat twisted way.

Thanks.

Re: Help requested: Packages which FTBFS randomly

[Bernd Zeimetz]

Hi,

> The following packages FTBFS for me randomly. First column is the bug
> number, second column is the estimated probability of failure in my
> build environment, which is described here:
> 
> https://people.debian.org/~sanvila/my-building-environment.txt


did you try to diff the build output? Did the packages fail with the
same error running the same command?
Are there other differences in the output?

Cheers,

Bernd


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: Help requested: Packages which FTBFS randomly

[Thibaut Paumard]

Dear Santiago,

Le 15/02/2017 à 18:26, Santiago Vila a écrit :
> Hello.
> 
> The following packages FTBFS for me randomly. First column is the bug
> number, second column is the estimated probability of failure in my
> build environment, which is described here:
> 
> https://people.debian.org/~sanvila/my-building-environment.txt
> 
> Before I ask the Release Managers that they make the bugs at the top
> of this list serious again

If you do that, please at the same time ask the release team to mark
them stretch-ignore, unless you are sure the test failures indicate a
real serious bug in the package.

Regards, Thibaut.



> (using some kind of threshold, as discussed
> in Bug #844264), I would appreciate if somebody else could double check
> that those packages have indeed very flaky tests (or buggy Makefiles,
> in some cases).
> 
> [ Please report any findings to the nnn...@bugs.debian.org address ].
> 
> While I agree that none of those individual bugs is more important
> than a bug of "FTBFS-always" type, some packages have a failure ratio
> so high that if we accepted that they "build from source" with such a
> high failure rate, they would effectively honor policy if we had it
> reversed, as in "packages must *not* build from source". I call this
> "the FTBFS-randomly Schrödinger paradox".
> 
> This is, however, not an invitation for the Release Managers to set
> 50% of failure rate as the threshold, that would be probably the worst
> possible interpretation of "Packages must build from source" (which is
> already Release Policy).
> 
> The current list:
> 
> 837067  1.000   libsecret
> 848063  0.990   ri-li
> 850282  0.820   influxdb (this one is already serious, thanks Adrian)
> 854686  0.556   fabric
> 854494  0.457   execnet
> 834686  0.440   ruby-httpclient
> 843038  0.400   elki
> 842244  0.390   congress
> 850506  0.327   acorn
> 850507  0.300   golang-github-tideland-golib
> 846771  0.292   python-traits
> 851722  0.288   django-pipeline
> 842836  0.250   golang-github-mxk-go-flowrate
> 854496  0.248   python-qtpy
> 844088  0.230   conversant-disruptor
> 848055  0.230   golang-github-go-chef-chef
> 834962  0.210   ruby-diaspora-vines
> 849217  0.185   jruby
> 839444  0.183   libgda5
> 850201  0.160   tendermint-go-flowrate
> 848054  0.150   debci
> 853056  0.145   txfixtures
> 854495  0.143   node-merge-stream
> 843052  0.140   libterm-filter-perl
> 852642  0.125   scoop
> 834959  0.120   golang-goleveldb
> 850414  0.110   ruby-ftw
> 832865  0.100   telepathy-python
> 
> [ The remaining packages that FTBFS randomly for me do so less than
>   10% of the time, but in either case all the figures are approximate ]
> 
> Thanks.
>