Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Hi Niels, > I had a look in the logs and it is not clear to me that the blacklist is > working --- /srv/lindsay.debian.org/config.yaml 2018-04-09 09:57:18.859643045 + +++ /srv/lindsay.debian.org/config.yaml 2018-04-09 09:57:58.336325579 + @@ -68,4 +68,4 @@ LINTIAN_SOURCE: https://anonscm.debian.org/git/lintian/lintian.git blacklist: - - gcc-8-cross-ports (#890873) + - gcc-8-cross-ports # (#890873) "I believe that has some significance for our problem." — Richard Feynman Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Niels Thykier: > Chris Lamb: >> Hi Niels, >> > [...] >>> Thanks, looks good to me with a quick review. >> >> I'll give it a whirl! >> >> >> Best wishes, >> > > Looking forward to seeing updated results on lintian.debian.org. :) > > Thanks, > ~Niels > Hi, I had a look in the logs and it is not clear to me that the blacklist is working. Looking for gcc-8-cross-ports I get the following: > $ grep gcc-8-cross-ports /srv/lintian.debian.org/logs/sync_state.log* > /srv/lintian.debian.org/logs/sync_state.log.0:gcc-8-cross-ports/6 is > out-of-date: New group (triggered by > binary:cpp-8-alpha-linux-gnu/8-20180402-1cross2/i386) > /srv/lintian.debian.org/logs/sync_state.log.0:Group gcc-8-cross-ports/2 > dropped: It is not an active group Looking for "blacklisted" (which should appear in the log if something is blacked listed), I get nothing: > $ grep -r blacklisted /srv/lintian.debian.org/logs > $ I would have expected to at least see: > Ignoring blacklisted package src:gcc-8-cross-ports Thanks, ~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Chris Lamb: > Hi Niels, > >> Thanks for having a look at this. > > Thanks for the review. I've updated my patch here: > > > https://gist.githubusercontent.com/lamby/996d85ac0e6ba0a18af2758e79912299/raw > >> I have one design concern; I think the blacklist should be global >> not per archive. > > ACK, updated. > >> On the more minor side of things, I would have liked to see lintian >> still record the packages that are blacklisted. > > In lieu of marking them as such on the reports, I've made it spit out log > messages when it skips something. (For the offending packages, it will at > least still show "processed with Lintian $very_old_version" so it should > not be /too/ magic.) > > >> We will probably want to map the blacklist into a hashref so we can do >> O(1) testing to see if a source is blacklisted if the blacklist grows >> more than a handful of packages. > > I hope this is unnecessary too (!), but I'm a bit stubborn so have > implemented it regardless. > > Looking forward to your feedback... :) > > > Best wishes, > Thanks, looks good to me with a quick review. I think we should try that variant and see what breaks. :) Honestly, I do not remember if you can use $foo->{'bar'} in a string interpolation, but I guess are about to find out. :) Thanks, ~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Hi Niels, > Thanks for having a look at this. Thanks for the review. I've updated my patch here: https://gist.githubusercontent.com/lamby/996d85ac0e6ba0a18af2758e79912299/raw > I have one design concern; I think the blacklist should be global > not per archive. ACK, updated. > On the more minor side of things, I would have liked to see lintian > still record the packages that are blacklisted. In lieu of marking them as such on the reports, I've made it spit out log messages when it skips something. (For the offending packages, it will at least still show "processed with Lintian $very_old_version" so it should not be /too/ magic.) > We will probably want to map the blacklist into a hashref so we can do > O(1) testing to see if a source is blacklisted if the blacklist grows > more than a handful of packages. I hope this is unnecessary too (!), but I'm a bit stubborn so have implemented it regardless. Looking forward to your feedback... :) Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Hi, >From #debian-qa: Can we not blacklist gcc-8-cross-ports fornow re. #890873? Whilst ugly, I think it would be better for lintian reports to be more up-to-date. we could try my patch after getting DSA to install the new dependencies that is what I would recommend; but I am not working on lintian atm., so I am happy to leave the decision to you Cool. Well, unfortunately your unpacker-io-async branch is outside of my Perl competency to review alas. So I would not have any idea how to fix it if it broke :) Not /quite/ sure how I'd even blacklist, mind you. The test harness has always kinda worked before :) blacklist? As in, keep HEAD as it is, simply skip gcc-8-cross-ports so at least we are processing the rest of the archive. ah, blacklist in the harness. We never had a need for that so the code does not exist but it is probably doable to implement Basically, if find_backlog (L::Util) does not return the package, it will never be processed so we need some control for blacklisting packages (e.g. the config file) and propogate it to find_backlog. This could be sync-state that sets a "blacklisted: true" flag and find_backlog that looks for it and then, we probably want to show which packages have been blacklisted in the generated reports (but that is an independent step) At this point I think even a hacky list of exceptions in the code would be preferable to being so behind. :( (unsure how the config file foo works, alas) The config is just a YAML file - in perl we work with it as a hashref at the root level and then whatever structure we expect beneath that /srv/lintian.debian.org/config.yaml ? yes. so something like: for my $dist (@{$CONFIG->{'archives'}{'debian-debug'}{'distributions'}}) { ... }" http://paste.debian.net/1019095/ From $GIT/reporting/config.yaml (which is an example) I think /srv/lintian.debian.org/config.yaml also have debian-debug (that is how we find the -dbgsym packages Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Olly Betts: > On Tue, Feb 20, 2018 at 07:48:31AM +0100, Niels Thykier wrote: >> Seen twice on lindsay.d.o in the last 24 hours. The exact reason is >> unknown, but it is probably during the unpacker (last non-error in the >> log is lintian starting the unpack, plus previously we had a race-condition >> in the unpacker that could trigger a similar fork-bomp situation). >> >> Most likely we never fixed this condition and only made it >> "sufficiently unlikely" until gcc-8-cross-ports showed up. >> >> For now, I have disabled the crontab on lindsay.d.o to avoid it taking >> down our host. > > I tried to build the wxwidgets3.0 3.0.4+dfsg-1 in an unstable chroot a > couple of days ago. I use sbuild and it's configured to run lintian in > the chroot on the built packages. However lintian ran out of disk space > in the chroot, which isn't something I have hit before. And I would > expect the build itself to need more disk space than lintian (assuming > that the debian/rules clean target is run before lintian is). Lintian > appeared to be slowly failing on each binary package in turn, so I just > hit Ctrl-C on it. > > I asked on #debian-devel in case this was a known problem, and lamby > pointed out this ticket and suggested adding a note in case this is the > same underlying issue, so I'm doing so. I didn't attempt to investigate > further yet. > > Cheers, > Olly > Hi Olly, Can you check if you end up with 10+ lintian processes when you experience the "out of disk"-issue? If possible, can you also try lintian with the [io-async branch] to see if that fixes your issue? Note: It will require libio-async-perl to be available when lintian runs (in case you do ad-hoc patching of the installed lintian). Thanks, ~Niels [io-async branch]: https://anonscm.debian.org/git/users/nthykier/lintian.git/log/?h=unpacker-io-async
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
On Tue, Feb 20, 2018 at 07:48:31AM +0100, Niels Thykier wrote: > Seen twice on lindsay.d.o in the last 24 hours. The exact reason is > unknown, but it is probably during the unpacker (last non-error in the > log is lintian starting the unpack, plus previously we had a race-condition > in the unpacker that could trigger a similar fork-bomp situation). > > Most likely we never fixed this condition and only made it > "sufficiently unlikely" until gcc-8-cross-ports showed up. > > For now, I have disabled the crontab on lindsay.d.o to avoid it taking > down our host. I tried to build the wxwidgets3.0 3.0.4+dfsg-1 in an unstable chroot a couple of days ago. I use sbuild and it's configured to run lintian in the chroot on the built packages. However lintian ran out of disk space in the chroot, which isn't something I have hit before. And I would expect the build itself to need more disk space than lintian (assuming that the debian/rules clean target is run before lintian is). Lintian appeared to be slowly failing on each binary package in turn, so I just hit Ctrl-C on it. I asked on #debian-devel in case this was a known problem, and lamby pointed out this ticket and suggested adding a note in case this is the same underlying issue, so I'm doing so. I didn't attempt to investigate further yet. Cheers, Olly
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Niels Thykier: > Package: lintian > Version: 2.5.74 > Severity: important > > Seen twice on lindsay.d.o in the last 24 hours. The exact reason is > unknown, but it is probably during the unpacker (last non-error in the > log is lintian starting the unpack, plus previously we had a race-condition > in the unpacker that could trigger a similar fork-bomp situation). > > Most likely we never fixed this condition and only made it > "sufficiently unlikely" until gcc-8-cross-ports showed up. > > > For now, I have disabled the crontab on lindsay.d.o to avoid it taking > down our host. > > Thanks, > ~Niels > Btw, I got an old prototype rewrite where the custom parallelization handling is replaced by IO::Async in https://anonscm.debian.org/git/users/nthykier/lintian.git/log/?h=unpacker-io-async No clue if this handles this particular problem, but I wanted to mentioned it in case someone else got the same idea of rewriting things to use IO::Async. Thanks, ~Niels
Bug#890873: lintian: gcc-8-cross-ports makes lintian fork-bomb, consume all memory and fill the disk
Package: lintian Version: 2.5.74 Severity: important Seen twice on lindsay.d.o in the last 24 hours. The exact reason is unknown, but it is probably during the unpacker (last non-error in the log is lintian starting the unpack, plus previously we had a race-condition in the unpacker that could trigger a similar fork-bomp situation). Most likely we never fixed this condition and only made it "sufficiently unlikely" until gcc-8-cross-ports showed up. For now, I have disabled the crontab on lindsay.d.o to avoid it taking down our host. Thanks, ~Niels