Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Hi Niels, > I had a look in the logs and it is not clear to me that the blacklist is > working --- /srv/lindsay.debian.org/config.yaml 2018-04-09 09:57:18.859643045 + +++ /srv/lindsay.debian.org/config.yaml 2018-04-09 09:57:58.336325579 + @@ -68,4 +68,4 @@ LINTIAN_SOURCE: https://anonscm.debian.org/git/lintian/lintian.git blacklist: - - gcc-8-cross-ports (#890873) + - gcc-8-cross-ports # (#890873) "I believe that has some significance for our problem." — Richard Feynman Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Niels Thykier: > Chris Lamb: >> Hi Niels, >> > [...] >>> Thanks, looks good to me with a quick review. >> >> I'll give it a whirl! >> >> >> Best wishes, >> > > Looking forward to seeing updated results on lintian.debian.org. :) > > Thanks, > ~Niels > Hi, I had a look in the logs and it is not clear to me that the blacklist is working. Looking for gcc-8-cross-ports I get the following: > $ grep gcc-8-cross-ports /srv/lintian.debian.org/logs/sync_state.log* > /srv/lintian.debian.org/logs/sync_state.log.0:gcc-8-cross-ports/6 is > out-of-date: New group (triggered by > binary:cpp-8-alpha-linux-gnu/8-20180402-1cross2/i386) > /srv/lintian.debian.org/logs/sync_state.log.0:Group gcc-8-cross-ports/2 > dropped: It is not an active group Looking for "blacklisted" (which should appear in the log if something is blacked listed), I get nothing: > $ grep -r blacklisted /srv/lintian.debian.org/logs > $ I would have expected to at least see: > Ignoring blacklisted package src:gcc-8-cross-ports Thanks, ~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Chris Lamb:
> Hi Niels,
>
>> I do not remember if you can use $foo->{'bar'} in a string
>> interpolation, but I guess are about to find out. :)
>
> $ cat test.pl
> use strict;
> my $foo = { 'bar' => 'Yes.' };
> print "Can we use \$foo->{'bar'} in a string interpolation?
> $foo->{'bar'}\n";
>
> $ perl test.pl
> Can we use $foo->{'bar'} in a string interpolation? Yes.
>
> TIL :)
>
TIWR :)
>> Thanks, looks good to me with a quick review.
>
> I'll give it a whirl!
>
>
> Best wishes,
>
Looking forward to seeing updated results on lintian.debian.org. :)
Thanks,
~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Hi Niels,
> I do not remember if you can use $foo->{'bar'} in a string
> interpolation, but I guess are about to find out. :)
$ cat test.pl
use strict;
my $foo = { 'bar' => 'Yes.' };
print "Can we use \$foo->{'bar'} in a string interpolation? $foo->{'bar'}\n";
$ perl test.pl
Can we use $foo->{'bar'} in a string interpolation? Yes.
TIL :)
> Thanks, looks good to me with a quick review.
I'll give it a whirl!
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Chris Lamb:
> Hi Niels,
>
>> Thanks for having a look at this.
>
> Thanks for the review. I've updated my patch here:
>
>
> https://gist.githubusercontent.com/lamby/996d85ac0e6ba0a18af2758e79912299/raw
>
>> I have one design concern; I think the blacklist should be global
>> not per archive.
>
> ACK, updated.
>
>> On the more minor side of things, I would have liked to see lintian
>> still record the packages that are blacklisted.
>
> In lieu of marking them as such on the reports, I've made it spit out log
> messages when it skips something. (For the offending packages, it will at
> least still show "processed with Lintian $very_old_version" so it should
> not be /too/ magic.)
>
>
>> We will probably want to map the blacklist into a hashref so we can do
>> O(1) testing to see if a source is blacklisted if the blacklist grows
>> more than a handful of packages.
>
> I hope this is unnecessary too (!), but I'm a bit stubborn so have
> implemented it regardless.
>
> Looking forward to your feedback... :)
>
>
> Best wishes,
>
Thanks, looks good to me with a quick review.
I think we should try that variant and see what breaks. :) Honestly, I
do not remember if you can use $foo->{'bar'} in a string interpolation,
but I guess are about to find out. :)
Thanks,
~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Hi Niels, > Thanks for having a look at this. Thanks for the review. I've updated my patch here: https://gist.githubusercontent.com/lamby/996d85ac0e6ba0a18af2758e79912299/raw > I have one design concern; I think the blacklist should be global > not per archive. ACK, updated. > On the more minor side of things, I would have liked to see lintian > still record the packages that are blacklisted. In lieu of marking them as such on the reports, I've made it spit out log messages when it skips something. (For the offending packages, it will at least still show "processed with Lintian $very_old_version" so it should not be /too/ magic.) > We will probably want to map the blacklist into a hashref so we can do > O(1) testing to see if a source is blacklisted if the blacklist grows > more than a handful of packages. I hope this is unnecessary too (!), but I'm a bit stubborn so have implemented it regardless. Looking forward to your feedback... :) Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Chris Lamb:
> Chris Lamb wrote:
>
>>Can we not blacklist gcc-8-cross-ports fornow re. #890873?
> […]
>>At this point I think even a hacky list of exceptions in the
>> code would be preferable to being so behind.
>
> My first stab at a patch for blacklisting in the meantime:
>
> https://gist.github.com/lamby/20bd2c85f77a501fdc7a6db058b06ff4/raw
>
>
> Regards,
>
Thanks for having a look at this.
I have one design concern; I think the blacklist should be global and
not per archive. Note that lintian merges packages across archives into
a single group (i.e. it takes the source and binaries from one archive
and merges it with the binaries from the other archive to support dbgsym
processing).
From that perspective, a per-archive blacklist seems rather awkward
because it is not immediately clear what it means to blacklist a source
in one archive if it happens to exist in another archive as well. Not
to mention, you would still see gcc-8-cross-ports being processed (but
now we would only process the dbgsym packages).
If you change the config handling to make the blacklist was "global"
(i.e. apply to all archives), then every thing will probably "just
work(tm)".
Nits:
=
On the more minor side of things, I would have liked to see lintian
still record the packages that are blacklisted (and mark them as such in
the reports). E.g. to catch now obsolete backlists. However that can
certainly appear later.
We will probably want to map the blacklist into a hashref so we can do
O(1) testing to see if a source is blacklisted if the blacklist grows
more than a handful of packages. I hope it is unnecessary, but noted in
case you find "return if exists($blacklist->{$src});" more readable than
"return if grep { $src eq $_ } @{$blacklist};".
Thanks,
~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Chris Lamb wrote: >Can we not blacklist gcc-8-cross-ports fornow re. #890873? […] >At this point I think even a hacky list of exceptions in the > code would be preferable to being so behind. My first stab at a patch for blacklisting in the meantime: https://gist.github.com/lamby/20bd2c85f77a501fdc7a6db058b06ff4/raw Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Hi,
>From #debian-qa:
Can we not blacklist gcc-8-cross-ports fornow re. #890873? Whilst
ugly, I think it would be better for lintian reports to be more up-to-date.
we could try my patch after getting DSA to install the new
dependencies
that is what I would recommend; but I am not working on lintian
atm., so I am happy to leave the decision to you
Cool. Well, unfortunately your unpacker-io-async branch is outside of
my Perl competency to review alas. So I would not have any idea how to fix it
if it broke :)
Not /quite/ sure how I'd even blacklist, mind you. The test harness
has
always kinda worked before :)
blacklist?
As in, keep HEAD as it is, simply skip gcc-8-cross-ports so at least
we
are processing the rest of the archive.
ah, blacklist in the harness. We never had a need for that so the
code does not exist but it is probably doable to implement
Basically, if find_backlog (L::Util) does not return the package,
it
will never be processed
so we need some control for blacklisting packages (e.g. the config
file) and propogate it to find_backlog. This could be sync-state that sets a
"blacklisted: true" flag and find_backlog that looks for it
and then, we probably want to show which packages have been
blacklisted in the generated reports (but that is an independent step)
At this point I think even a hacky list of exceptions in the code
would
be preferable to being so behind. :( (unsure how the config file foo works,
alas)
The config is just a YAML file - in perl we work with it as a
hashref at the root level and then whatever structure we expect beneath that
/srv/lintian.debian.org/config.yaml ?
yes. so something like: for my $dist
(@{$CONFIG->{'archives'}{'debian-debug'}{'distributions'}}) { ... }"
http://paste.debian.net/1019095/ From $GIT/reporting/config.yaml
(which is an example)
I think /srv/lintian.debian.org/config.yaml also have debian-debug
(that is how we find the -dbgsym packages
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Olly Betts: > On Tue, Feb 20, 2018 at 07:48:31AM +0100, Niels Thykier wrote: >> Seen twice on lindsay.d.o in the last 24 hours. The exact reason is >> unknown, but it is probably during the unpacker (last non-error in the >> log is lintian starting the unpack, plus previously we had a race-condition >> in the unpacker that could trigger a similar fork-bomp situation). >> >> Most likely we never fixed this condition and only made it >> "sufficiently unlikely" until gcc-8-cross-ports showed up. >> >> For now, I have disabled the crontab on lindsay.d.o to avoid it taking >> down our host. > > I tried to build the wxwidgets3.0 3.0.4+dfsg-1 in an unstable chroot a > couple of days ago. I use sbuild and it's configured to run lintian in > the chroot on the built packages. However lintian ran out of disk space > in the chroot, which isn't something I have hit before. And I would > expect the build itself to need more disk space than lintian (assuming > that the debian/rules clean target is run before lintian is). Lintian > appeared to be slowly failing on each binary package in turn, so I just > hit Ctrl-C on it. > > I asked on #debian-devel in case this was a known problem, and lamby > pointed out this ticket and suggested adding a note in case this is the > same underlying issue, so I'm doing so. I didn't attempt to investigate > further yet. > > Cheers, > Olly > Hi Olly, Can you check if you end up with 10+ lintian processes when you experience the "out of disk"-issue? If possible, can you also try lintian with the [io-async branch] to see if that fixes your issue? Note: It will require libio-async-perl to be available when lintian runs (in case you do ad-hoc patching of the installed lintian). Thanks, ~Niels [io-async branch]: https://anonscm.debian.org/git/users/nthykier/lintian.git/log/?h=unpacker-io-async
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
On Tue, Feb 20, 2018 at 07:48:31AM +0100, Niels Thykier wrote: > Seen twice on lindsay.d.o in the last 24 hours. The exact reason is > unknown, but it is probably during the unpacker (last non-error in the > log is lintian starting the unpack, plus previously we had a race-condition > in the unpacker that could trigger a similar fork-bomp situation). > > Most likely we never fixed this condition and only made it > "sufficiently unlikely" until gcc-8-cross-ports showed up. > > For now, I have disabled the crontab on lindsay.d.o to avoid it taking > down our host. I tried to build the wxwidgets3.0 3.0.4+dfsg-1 in an unstable chroot a couple of days ago. I use sbuild and it's configured to run lintian in the chroot on the built packages. However lintian ran out of disk space in the chroot, which isn't something I have hit before. And I would expect the build itself to need more disk space than lintian (assuming that the debian/rules clean target is run before lintian is). Lintian appeared to be slowly failing on each binary package in turn, so I just hit Ctrl-C on it. I asked on #debian-devel in case this was a known problem, and lamby pointed out this ticket and suggested adding a note in case this is the same underlying issue, so I'm doing so. I didn't attempt to investigate further yet. Cheers, Olly
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Niels Thykier: > Package: lintian > Version: 2.5.74 > Severity: important > > Seen twice on lindsay.d.o in the last 24 hours. The exact reason is > unknown, but it is probably during the unpacker (last non-error in the > log is lintian starting the unpack, plus previously we had a race-condition > in the unpacker that could trigger a similar fork-bomp situation). > > Most likely we never fixed this condition and only made it > "sufficiently unlikely" until gcc-8-cross-ports showed up. > > > For now, I have disabled the crontab on lindsay.d.o to avoid it taking > down our host. > > Thanks, > ~Niels > Btw, I got an old prototype rewrite where the custom parallelization handling is replaced by IO::Async in https://anonscm.debian.org/git/users/nthykier/lintian.git/log/?h=unpacker-io-async No clue if this handles this particular problem, but I wanted to mentioned it in case someone else got the same idea of rewriting things to use IO::Async. Thanks, ~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Package: lintian Version: 2.5.74 Severity: important Seen twice on lindsay.d.o in the last 24 hours. The exact reason is unknown, but it is probably during the unpacker (last non-error in the log is lintian starting the unpack, plus previously we had a race-condition in the unpacker that could trigger a similar fork-bomp situation). Most likely we never fixed this condition and only made it "sufficiently unlikely" until gcc-8-cross-ports showed up. For now, I have disabled the crontab on lindsay.d.o to avoid it taking down our host. Thanks, ~Niels
