Re: Migration despite an RC bug?

[Adam D. Barratt]

On Wed, 2017-01-04 at 20:33 +0100, Michael Biebl wrote:
> Am 04.01.2017 um 19:53 schrieb Margarita Manterola:
> 
> > Can we accelerate the removal of non key packages, please?
> > 
> > One example: https://tracker.debian.org/pkg/libsigc++-1.2 migrated to
> > testing on Dec 29th even though it has an RC bug that was intended to
> > keep it out of it.
> > 
> > For whatever unexplicable reason, this package is marked "Priority:
> > important" which means that it gets installed by debootstrap (and, at
> > least with my preseeds, makes the installation fail).
> 
> Hm, we should remove this package completely (along with the only
> remaining rdep). I'll repurpose #817551 for that.

In the meantime I've added a removal hint for the two packages.

Regards,

Adam

Re: Migration despite an RC bug?

[Michael Biebl]

Am 04.01.2017 um 19:53 schrieb Margarita Manterola:

> Can we accelerate the removal of non key packages, please?
> 
> One example: https://tracker.debian.org/pkg/libsigc++-1.2 migrated to
> testing on Dec 29th even though it has an RC bug that was intended to
> keep it out of it.
> 
> For whatever unexplicable reason, this package is marked "Priority:
> important" which means that it gets installed by debootstrap (and, at
> least with my preseeds, makes the installation fail).

Hm, we should remove this package completely (along with the only
remaining rdep). I'll repurpose #817551 for that.


Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Re: Migration despite an RC bug?

[Margarita Manterola]

Hi,

On Fri, Dec 30, 2016 at 5:25 AM, Emilio Pozuelo Monfort 
wrote:

> We thought about rolling back to the previous state, but that means testing
> users who have already upgraded would need to manually downgrade... which
> is not
> good.
>
> So we may just wait for the auto-removal hinter to do its job for
> non-key-packages, and for the rest to be fixed or manually downgraded, or
> whatever other solution...
>

Can we accelerate the removal of non key packages, please?

One example: https://tracker.debian.org/pkg/libsigc++-1.2 migrated to
testing on Dec 29th even though it has an RC bug that was intended to keep
it out of it.

For whatever unexplicable reason, this package is marked "Priority:
important" which means that it gets installed by debootstrap (and, at least
with my preseeds, makes the installation fail).

-- 
Besos,
Marga

Re: Migration despite an RC bug? [and 1 more messages]

[Ian Jackson]

Niels Thykier writes ("Re: Migration despite an RC bug?"):
> An exception in my experience: In process is cheaper when the
> (de)compressor is available in the PerlIO Layer as native C code.
> Notable example being libperlio-gzip-perl where you use "open(my $fd,
> '<:gzip', $file)".
>   At least that was the case when I benchmarked on Lintian in 2.5.10 (2
> releases ago).

Surely this depends a lot on what you are doing with the data.  If the
consumer is doing any significant computation, you save elapsed time
by being able to do decompression and consumption in parallel.

Russ Allbery writes ("Re: Migration despite an RC bug?"):
> Ah, I didn't try that!  I was only playing around with IO::Uncompress.  I
> may have to go revisit that project, since managing the external process
> was a huge pain.

Maybe there should be a library to do the process management.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Migration despite an RC bug?

[Don Armstrong]

On Tue, 03 Jan 2017, Niels Thykier wrote:
> An exception in my experience: In process is cheaper when the
> (de)compressor is available in the PerlIO Layer as native C code.
> Notable example being libperlio-gzip-perl where you use "open(my $fd,
> '<:gzip', $file)".
>   At least that was the case when I benchmarked on Lintian in 2.5.10 (2
> releases ago).

Neat. I'd love a perlio layer which enabled me to do something like:

  open($fh,'<:anyuncompress:encoding(UTF-8)',$file) [...]

too, as the archive has moved away from gzip to xz.

-- 
Don Armstrong  https://www.donarmstrong.com

life's not a paragraph
And death i think is no parenthesis
 -- e.e. cummings "Four VII" _is 5_

Re: Migration despite an RC bug?

[Russ Allbery]

Niels Thykier  writes:
> Russ Allbery:

>> I've done extensive benchmarking of this in Perl for a different
>> project and yes, fork and exec of an external compresser is *way*
>> faster than using a library.  I suspect the Perl compress libraries are
>> making extraneous data copies or doing something else suboptimal.

> An exception in my experience: In process is cheaper when the
> (de)compressor is available in the PerlIO Layer as native C code.
> Notable example being libperlio-gzip-perl where you use "open(my $fd,
> '<:gzip', $file)".
>   At least that was the case when I benchmarked on Lintian in 2.5.10 (2
> releases ago).

Ah, I didn't try that!  I was only playing around with IO::Uncompress.  I
may have to go revisit that project, since managing the external process
was a huge pain.

-- 
Russ Allbery (r...@debian.org)   

Re: Migration despite an RC bug?

[Niels Thykier]

Russ Allbery:
> [...]
>> I wouldn't have expected that either, but it appeared to be 4-5 times
>> slower than the equivalent code with fork a decompressor, which is why I
>> swapped it out. [I didn't bother to benchmark them, because the
>> differences between them was so stark.]
> 
> I've done extensive benchmarking of this in Perl for a different project
> and yes, fork and exec of an external compresser is *way* faster than
> using a library.  I suspect the Perl compress libraries are making
> extraneous data copies or doing something else suboptimal.
> 

An exception in my experience: In process is cheaper when the
(de)compressor is available in the PerlIO Layer as native C code.
Notable example being libperlio-gzip-perl where you use "open(my $fd,
'<:gzip', $file)".
  At least that was the case when I benchmarked on Lintian in 2.5.10 (2
releases ago).

Thanks,
~Niels

Re: Migration despite an RC bug?

[Don Armstrong]

On Tue, 03 Jan 2017, Russ Allbery wrote:
> If you're using DB_File, I think you have to use the explicit put()
> and get() API instead of the tied magical hash in order to get error
> reporting.

That matches what documentation I've found so far.

Instead of really working on hacking this out, I've been working instead
on making this whole code section use a real database, which it really
should have been doing the entire time... but it still needs to work in
the meantime.


-- 
Don Armstrong  https://www.donarmstrong.com

Science is a way of trying not to fool yourself. The first principle
is that you must not fool yourself, and you are the easiest person to
fool.
 -- Richard Feynman "What is and What Should be the Role of Scientific
Culture in Modern Society"; 1964

Re: Migration despite an RC bug?

[Russ Allbery]

Don Armstrong  writes:
> On Tue, 03 Jan 2017, Ian Jackson wrote:

>> Also, have you checked whether your DB library properly throws errors
>> on writes to a tied hash ?

> I don't know whether it does or not; I went looking to see whether you
> could trap errors on untie(), and untie doesn't return anything useful
> that you can check.

If you're using DB_File, I think you have to use the explicit put() and
get() API instead of the tied magical hash in order to get error
reporting.

>> Also, I'm not sure why it would be "incredibly slow". In a
>> singlethreaded cpubound task (the worst case) I wouldn't expect worse
>> than a 50% slowdown.

> I wouldn't have expected that either, but it appeared to be 4-5 times
> slower than the equivalent code with fork a decompressor, which is why I
> swapped it out. [I didn't bother to benchmark them, because the
> differences between them was so stark.]

I've done extensive benchmarking of this in Perl for a different project
and yes, fork and exec of an external compresser is *way* faster than
using a library.  I suspect the Perl compress libraries are making
extraneous data copies or doing something else suboptimal.

-- 
Russ Allbery (r...@debian.org)   

Re: Migration despite an RC bug?

[Don Armstrong]

On Tue, 03 Jan 2017, Ian Jackson wrote:
> The result is that you ignore nonzero exit status from your
> decompression program. My theory for the incident we are discussing is
> that your decompressor got a SIGTERM, and your script got EOF on the
> pipe.

Yeah, this is likely it. Thanks for catching that; I'll fix this
codepath shortly.

> Also, have you checked whether your DB library properly throws errors
> on writes to a tied hash ?

I don't know whether it does or not; I went looking to see whether you
could trap errors on untie(), and untie doesn't return anything useful
that you can check.

> Also, I'm not sure why it would be "incredibly slow". In a
> singlethreaded cpubound task (the worst case) I wouldn't expect worse
> than a 50% slowdown.

I wouldn't have expected that either, but it appeared to be 4-5 times
slower than the equivalent code with fork a decompressor, which is why I
swapped it out. [I didn't bother to benchmark them, because the
differences between them was so stark.]

> Finally, examples/debian/versions/update-mldbm starts with
>   #! /bin/sh -e
> I would normally use `set -e' instead, because foolish people
> sometimes run scripts by saying `bash /path/to/script' or `sh
> /path/to/script'.  (This doesn't seem to be a problem in debbugs.)

Yeah, the -e was inherited, but it has both as you note.

-- 
Don Armstrong  https://www.donarmstrong.com

"The trouble with you, Ibid" he said, "is that you think you're the
biggest bloody authority on everything"
 -- Terry Pratchet _Pyramids_ p146

Re: Migration despite an RC bug?

[Ian Jackson]

Don Armstrong writes ("Re: Migration despite an RC bug?"):
> On Sat, 31 Dec 2016, Ian Jackson wrote:
> > I've debugged a lot of this kind of thing.  Point me at your
> > (pre-just-fixed) code and I might spot it ?
> 
> These two are how I think I've fixed it:

I don't think so.  I cloned[1] the code and looked at
examples/debian/versions/build-versions-db.

In this area the most obvious error handling bug is that you do not
ever close $PACKAGES, which you earlier opened from a pipe.  Consult
the docs for close in perlfunc(1).

The result is that you ignore nonzero exit status from your
decompression program.  My theory for the incident we are discussing
is that your decompressor got a SIGTERM, and your script got EOF on
the pipe.  Your script thought that the EOF meant "all data has been
read" and exited status zero having written partial output.  If your
script had closed $PACKAGES and checked the return value, it would
have discovered the SIGTERM and died, instead.

Other things I noticed:

You do not check the error from readdir, nor glob.  (glob is
particularly annoying to fix.)  Admittedly, failures of opendir and
readdir are quite rare as they normally result only from very serious
problems, or absurd permissions.

You use perl's `-d' and `-f' operators other than on `_'.  This is
wrong (at least, if you can't safely tolerate false negatives) because
they return false if stat fails with an unexpected error code (eg,
EIO).  The right pattern is to use [l]stat, check for undef and check
$!, and then perhaps say `-d _'.

This code in the suites loop
my $sources = (grep { -f $_ } \
glob "$suitedir/$component/source/Sources.*")[0];
next unless defined $sources;
seems perhaps too forgiving of unexpected situations.

Also, have you checked whether your DB library properly throws errors
on writes to a tied hash ?

Of the above only the process exit status bug seems likely to have
resulted in an empty output database occuring as a consquence of
running this script during system shutdown.  But, see below, because
I'm confused about what version was running when the bug triggered.

The only other possibility I can think of is that the input files it
reads were empty or missing.

> http://git.donarmstrong.com/?p=debbugs.git;a=commitdiff;h=29b55e4d5535a68cc6d2294f5c362d271b53c6d2
> http://git.donarmstrong.com/?p=debbugs.git;a=commitdiff;h=d83ffb68f75ae98ad5005eee9b173d5dac08c343

I can't see how that would fix any such underlying bug.  AFAICT it
just arranges to update an existing database rather than generating a
fresh one.  In general that seems likely to introduce more bugs rather
than fixing bugs, although I haven't thought about it in detail.

Also I was very puzzled by this hunk:

   sub read_packages {
  -my ($packages, $component,$arch,$dist) = @_;
  -my $PACKAGES = open_compressed_file($packages)
  +my ($db,$db2,$packages, $component,$arch,$dist) = @_;
  +my $PACKAGES = open_compressed_file($packages) or
   die "Unable to open $packages for reading: $!";

The "old" code is:

   sub read_packages {
   my ($packages, $component,$arch,$dist) = @_;
   my $PACKAGES = open_compressed_file($packages)
   die "Unable to open $packages for reading: $!";

which is a syntax error.

> [I believe I exposed this bug because I switched to IO::Uncompress,
> which is incredibly slow; I've now switched relevant pieces of code
> back.]

I think this isn't right.  If my theory above is right, the bug was in
the open_compressed_file version.  But that version seems to have this
syntax error, which was only fixed in git on the 2nd of January.  Is
it possible that the switch to open_compressed_file was made directly
in the running working tree, and committed only later ?

FWIW IMO using an in-process library, rather than forking an
(un)compressor, is an antipattern.  Using a library rather than
fork/exec:
 * introduces additional ABI/API coupling
 * makes poorer use of modern multicore computers (or, if you
   are really unlucky, uses threads)
 * makes it more work to support an additional compression formats

On my system IO::Uncompress::AnyUncompress does not seem to fork.  But
maybe it does on the debbugs machine, in which case it could have a
similar error handling bug.

Also, I'm not sure why it would be "incredibly slow".  In a
singlethreaded cpubound task (the worst case) I wouldn't expect worse
than a 50% slowdown.

Finally, examples/debian/versions/update-mldbm starts with
  #! /bin/sh -e
I would normally use `set -e' instead, because foolish people
sometimes run scripts by saying `bash /path/to/script' or `sh
/path/to/script'.  (This doesn't seem to be a problem in debbugs.)

Regards,
Ian.

[1] IWBNI your gitweb had the git clone url.  I guessed.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Migration despite an RC bug?

[Don Armstrong]

On Sat, 31 Dec 2016, Ian Jackson wrote:

> Don Armstrong writes ("Re: Migration despite an RC bug?"):
> > I'm still not quite sure how the script was failing. The outer shell
> > invocation which calls a perl script to do the versioning database
> > update is run with set -e, and the perl script should exit with non-zero
> > if it fails to work properly. However, it appears that when buxtehude is
> > restarted, the perl script is killed before the outer bash script
> > completes but the exit state isn't properly handled?
> 
> I've debugged a lot of this kind of thing.  Point me at your
> (pre-just-fixed) code and I might spot it ?

These two are how I think I've fixed it:

http://git.donarmstrong.com/?p=debbugs.git;a=commitdiff;h=29b55e4d5535a68cc6d2294f5c362d271b53c6d2
http://git.donarmstrong.com/?p=debbugs.git;a=commitdiff;h=d83ffb68f75ae98ad5005eee9b173d5dac08c343

[I believe I exposed this bug because I switched to IO::Uncompress,
which is incredibly slow; I've now switched relevant pieces of code
back.]

> That may well help. Or maybe the bug is in a higher layer somewhere.
> Are HTTP and CGI involved ?

No; it's happening at the updating the versioning database level.


-- 
Don Armstrong  https://www.donarmstrong.com

All bad precedents began as justifiable measures.
 -- Gaius Julius Caesar in "The Conspiracy of Catiline" by Sallust

Re: Migration despite an RC bug?

[Ian Jackson]

Don Armstrong writes ("Re: Migration despite an RC bug?"):
> I'm still not quite sure how the script was failing. The outer shell
> invocation which calls a perl script to do the versioning database
> update is run with set -e, and the perl script should exit with non-zero
> if it fails to work properly. However, it appears that when buxtehude is
> restarted, the perl script is killed before the outer bash script
> completes but the exit state isn't properly handled?

I've debugged a lot of this kind of thing.  Point me at your
(pre-just-fixed) code and I might spot it ?

> I've fixed this by making the perl script do all of the logic which was
> in the bash script originally so if it fails, it should at least do so
> gracefully.

That may well help.  Or maybe the bug is in a higher layer somewhere.
Are HTTP and CGI involved ?

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Migration despite an RC bug?

[Don Armstrong]

On Thu, 29 Dec 2016, Christoph Biedl wrote:
> Emilio Pozuelo Monfort wrote...
> 
> > Unforunately, the BTS exported a broken/incomplete RC bug list, and britney 
> > used
> > that and didn't see that some packages had an RC bug, so it allowed them to 
> > migrate.
> 
> Ouch, that's quite a nightmare. While I'm curious to learn how this
> happened and what is done to prevent this from happening again -

I'm still not quite sure how the script was failing. The outer shell
invocation which calls a perl script to do the versioning database
update is run with set -e, and the perl script should exit with non-zero
if it fails to work properly. However, it appears that when buxtehude is
restarted, the perl script is killed before the outer bash script
completes but the exit state isn't properly handled?

I've fixed this by making the perl script do all of the logic which was
in the bash script originally so if it fails, it should at least do so
gracefully.

-- 
Don Armstrong  https://www.donarmstrong.com

He was wrong. Nature abhors dimensional abnormalities, and seals them
neatly away so that they don't upset people. Nature, in fact, abhors a
lot of things, including vacuums, ships called the Marie Celeste, and
the chuck keys for electric drills.
 -- Terry Pratchet _Pyramids_ p166

Re: Migration despite an RC bug?

[Bálint Réczey]

Hi,

2016-12-30 12:53 GMT+01:00 Guillem Jover :
> Hi!
>
> On Fri, 2016-12-30 at 09:25:20 +0100, Emilio Pozuelo Monfort wrote:
>> On 29/12/16 23:36, Christoph Biedl wrote:
>> > Emilio Pozuelo Monfort wrote...
>> >> Unforunately, the BTS exported a broken/incomplete RC bug list, and 
>> >> britney used
>> >> that and didn't see that some packages had an RC bug, so it allowed them 
>> >> to migrate.
>
>> > please rather focus on restoring the correct state. Installations
>> > running testing already might have got packages they shouldn't see.
>>
>> We thought about rolling back to the previous state, but that means testing
>> users who have already upgraded would need to manually downgrade... which is 
>> not
>> good.
>
> Yeah, any solution at this point will have serious downsides. :/

While this is not a solution but user-tagging the RC bugs which would
have prevented the migration but were ignored could help quickly
reviewing the packages which are affected.

Cheers,
Balint

>
>> So we may just wait for the auto-removal hinter to do its job for
>> non-key-packages, and for the rest to be fixed or manually downgraded, or
>> whatever other solution...
>
> I hope this does not trigger a storm of unnecessarily epoch'ed uploads. :(
>
> Thanks,
> Guillem
>

Re: Migration despite an RC bug?

[Guillem Jover]

Hi!

On Fri, 2016-12-30 at 09:25:20 +0100, Emilio Pozuelo Monfort wrote:
> On 29/12/16 23:36, Christoph Biedl wrote:
> > Emilio Pozuelo Monfort wrote...
> >> Unforunately, the BTS exported a broken/incomplete RC bug list, and 
> >> britney used
> >> that and didn't see that some packages had an RC bug, so it allowed them 
> >> to migrate.

> > please rather focus on restoring the correct state. Installations
> > running testing already might have got packages they shouldn't see.
> 
> We thought about rolling back to the previous state, but that means testing
> users who have already upgraded would need to manually downgrade... which is 
> not
> good.

Yeah, any solution at this point will have serious downsides. :/

> So we may just wait for the auto-removal hinter to do its job for
> non-key-packages, and for the rest to be fixed or manually downgraded, or
> whatever other solution...

I hope this does not trigger a storm of unnecessarily epoch'ed uploads. :(

Thanks,
Guillem

Re: Migration despite an RC bug?

[Emilio Pozuelo Monfort]

On 29/12/16 23:36, Christoph Biedl wrote:
> Emilio Pozuelo Monfort wrote...
> 
>> Unforunately, the BTS exported a broken/incomplete RC bug list, and britney 
>> used
>> that and didn't see that some packages had an RC bug, so it allowed them to 
>> migrate.
> 
> Ouch, that's quite a nightmare. While I'm curious to learn how this
> happened and what is done to prevent this from happening again -

I don't know what will happen in the BTS side to avoid this in the future - but
we'll add a check on the britney side to abort if the RC bug list looks
suspiciously small.

> please rather focus on restoring the correct state. Installations
> running testing already might have got packages they shouldn't see.

We thought about rolling back to the previous state, but that means testing
users who have already upgraded would need to manually downgrade... which is not
good.

So we may just wait for the auto-removal hinter to do its job for
non-key-packages, and for the rest to be fixed or manually downgraded, or
whatever other solution...

Cheers,
Emilio

Re: Migration despite an RC bug?

[Christoph Biedl]

Emilio Pozuelo Monfort wrote...

> Unforunately, the BTS exported a broken/incomplete RC bug list, and britney 
> used
> that and didn't see that some packages had an RC bug, so it allowed them to 
> migrate.

Ouch, that's quite a nightmare. While I'm curious to learn how this
happened and what is done to prevent this from happening again -
please rather focus on restoring the correct state. Installations
running testing already might have got packages they shouldn't see.

Christoph


signature.asc
Description: Digital signature

Re: Migration despite an RC bug?

[Ole Streicher]

Emilio Pozuelo Monfort  writes:
> On 29/12/16 17:24, Ole Streicher wrote:
>> Hi,
>> 
>> the python-numpy package in unstable has an RC bug:
>> 
>> https://bugs.debian.org/849196
>> 
>> however, today it migrated to testing, the migration status still says
>> however "valid candidate".
>> 
>> I thought that RC bugs would prevent packages from migration -- what is
>> the exception here?
>
> Unforunately, the BTS exported a broken/incomplete RC bug list, and britney 
> used
> that and didn't see that some packages had an RC bug, so it allowed
> them to migrate.

Is there a way to get the old version back? At least, I am quite unhappy
with a buggy release candidate in testing.

Best regards

Ole

Re: Migration despite an RC bug?

[Emilio Pozuelo Monfort]

On 29/12/16 17:24, Ole Streicher wrote:
> Hi,
> 
> the python-numpy package in unstable has an RC bug:
> 
> https://bugs.debian.org/849196
> 
> however, today it migrated to testing, the migration status still says
> however "valid candidate".
> 
> I thought that RC bugs would prevent packages from migration -- what is
> the exception here?

Unforunately, the BTS exported a broken/incomplete RC bug list, and britney used
that and didn't see that some packages had an RC bug, so it allowed them to 
migrate.

Cheers,
Emilio

Migration despite an RC bug?

[Ole Streicher]

Hi,

the python-numpy package in unstable has an RC bug:

https://bugs.debian.org/849196

however, today it migrated to testing, the migration status still says
however "valid candidate".

I thought that RC bugs would prevent packages from migration -- what is
the exception here?

Best regards

Ole