Bug#932296: qa.debian.org: getwatch filling up /tmp
Re: Lucas Nussbaum > To get the watch file, UDD extracts the source package (once per source > package per version; then the watch file is stored in DB). /tmp on > ullmann only has 5.3GB available, which is too small to extract some > source packages in Debian (such as nvidia-cuda-toolkit). For exactly this use case, I wrote `dscextract` in devscripts. It extracts files from debian/* without having to unpack the tarball. Christoph
Bug#932296: qa.debian.org: getwatch filling up /tmp
On 17/06/20 at 12:09 +0200, Julien Cristau wrote: > On Wed, Jun 17, 2020 at 07:00:43 +0100, Adam D. Barratt wrote: > > > On Wed, 2020-06-17 at 00:14 +0200, Lucas Nussbaum wrote: > > > Apparently the condition where this happens is quite rare > > > occurences on 08/2019, 12/2019, 06/2020), so notifying me after the > > > files were cleaned up from /tmp makes it hard to identify which > > > packages cause this issue. If I could get notified when a warning > > > limit is reached, it would be much easier to debug. > > > > I'm not sure what the usual policy on that is, but I didn't clean up > > /tmp after disabling the importer last night: I could not reproduce this exact issue, but ran into something similar. It boils down to: To get the watch file, UDD extracts the source package (once per source package per version; then the watch file is stored in DB). /tmp on ullmann only has 5.3GB available, which is too small to extract some source packages in Debian (such as nvidia-cuda-toolkit). I've just added a more generic exception handling, so now UDD should clean up /tmp from those extracted packages in all cases (which wasn't the case before, even if I don't understand exactly why it wasn't). However, until the disk space available for /tmp is increased, this importer would continue to fill up / on a regular basis, which will likely break other stuff. So, please increase the disk space available in /tmp (or provide a dedicated temporary space, for example under /srv/udd.debian.org/). In the meantime, I've disabled the importer in rudd.conf. Thanks Lucas
Bug#932296: qa.debian.org: getwatch filling up /tmp
On Wed, Jun 17, 2020 at 07:00:43 +0100, Adam D. Barratt wrote: > On Wed, 2020-06-17 at 00:14 +0200, Lucas Nussbaum wrote: > > Apparently the condition where this happens is quite rare > > occurences on 08/2019, 12/2019, 06/2020), so notifying me after the > > files were cleaned up from /tmp makes it hard to identify which > > packages cause this issue. If I could get notified when a warning > > limit is reached, it would be much easier to debug. > > I'm not sure what the usual policy on that is, but I didn't clean up > /tmp after disabling the importer last night: > > drwx-- 3 udd uddadm 4096 Jun 16 20:20 getwatch.qmapshack.n13QHA > drwx-- 3 udd uddadm 4096 Jun 16 20:20 getwatch.picard-tools.Zg0jud > drwx-- 3 udd uddadm 4096 Jun 16 20:50 getwatch.qmapshack.aH184l > drwx-- 3 udd uddadm 4096 Jun 16 20:50 getwatch.picard-tools.SqIkjD > drwx-- 3 udd uddadm 4096 Jun 16 21:20 getwatch.qmapshack.1pIg10 > drwx-- 3 udd uddadm 4096 Jun 16 21:20 getwatch.picard-tools.g3weib > drwx-- 3 udd uddadm 4096 Jun 16 21:50 getwatch.qmapshack.oklPSa > drwx-- 3 udd uddadm 4096 Jun 16 21:50 getwatch.picard-tools.Lo3UhJ > This is how it looked before reboot yesterday, according to my terminal's scroll buffer: jcristau@ullmann:~$ ls -l /tmp/getwatch.* -d drwx-- 3 udd uddadm 4096 Jun 16 12:50 /tmp/getwatch.deepin-icon-theme.dXa34U drwx-- 3 udd uddadm 4096 Jun 16 12:20 /tmp/getwatch.deepin-icon-theme.EDzB2B drwx-- 3 udd uddadm 4096 Jun 16 11:20 /tmp/getwatch.deepin-icon-theme.fG5L65 drwx-- 3 udd uddadm 4096 Jun 16 13:50 /tmp/getwatch.deepin-icon-theme.GKeDmI drwx-- 3 udd uddadm 4096 Jun 16 10:50 /tmp/getwatch.deepin-icon-theme.JiwELJ drwx-- 3 udd uddadm 4096 Jun 16 14:20 /tmp/getwatch.deepin-icon-theme.kgoDIn drwx-- 3 udd uddadm 4096 Jun 16 09:50 /tmp/getwatch.deepin-icon-theme.kqqITx drwx-- 3 udd uddadm 4096 Jun 16 13:20 /tmp/getwatch.deepin-icon-theme.p0Lknv drwx-- 3 udd uddadm 4096 Jun 16 10:20 /tmp/getwatch.deepin-icon-theme.sMzg7u drwx-- 3 udd uddadm 4096 Jun 16 11:50 /tmp/getwatch.deepin-icon-theme.uSHETI drwx-- 3 udd uddadm 4096 Jun 16 11:20 /tmp/getwatch.htsjdk.eC4uvs drwx-- 3 udd uddadm 4096 Jun 16 11:50 /tmp/getwatch.htsjdk.EU4suU drwx-- 3 udd uddadm 4096 Jun 16 14:20 /tmp/getwatch.htsjdk.kih83R drwx-- 3 udd uddadm 4096 Jun 16 12:20 /tmp/getwatch.htsjdk.L9J9LA drwx-- 3 udd uddadm 4096 Jun 16 10:20 /tmp/getwatch.htsjdk.m2FgS0 drwx-- 3 udd uddadm 4096 Jun 16 10:50 /tmp/getwatch.htsjdk.MwALoV drwx-- 3 udd uddadm 4096 Jun 16 09:50 /tmp/getwatch.htsjdk.N7bIVe drwx-- 3 udd uddadm 4096 Jun 16 13:20 /tmp/getwatch.htsjdk.NRopqF drwx-- 3 udd uddadm 4096 Jun 16 13:50 /tmp/getwatch.htsjdk.wEFDNs drwx-- 3 udd uddadm 4096 Jun 16 12:50 /tmp/getwatch.htsjdk.Wqf6gL drwx-- 3 udd uddadm 4096 Jun 16 09:50 /tmp/getwatch.picard-tools.BfwMyC drwx-- 3 udd uddadm 4096 Jun 16 12:20 /tmp/getwatch.picard-tools.gY2ZQk drwx-- 3 udd uddadm 4096 Jun 16 12:50 /tmp/getwatch.picard-tools.I1wZDY drwx-- 3 udd uddadm 4096 Jun 16 11:50 /tmp/getwatch.picard-tools.JG01pg drwx-- 3 udd uddadm 4096 Jun 16 14:20 /tmp/getwatch.picard-tools.KawVh5 drwx-- 3 udd uddadm 4096 Jun 16 11:20 /tmp/getwatch.picard-tools.l0wUag drwx-- 3 udd uddadm 4096 Jun 16 13:20 /tmp/getwatch.picard-tools.oVJAT9 drwx-- 3 udd uddadm 4096 Jun 16 13:50 /tmp/getwatch.picard-tools.SdFotX drwx-- 3 udd uddadm 4096 Jun 16 10:50 /tmp/getwatch.picard-tools.Tq98F0 drwx-- 3 udd uddadm 4096 Jun 16 10:20 /tmp/getwatch.picard-tools.zPqqVr drwx-- 3 udd uddadm 4096 Jun 16 12:20 /tmp/getwatch.qmapshack.B3SMHo drwx-- 3 udd uddadm 4096 Jun 16 10:20 /tmp/getwatch.qmapshack.hADG4I drwx-- 3 udd uddadm 4096 Jun 16 13:20 /tmp/getwatch.qmapshack.I1X2xV drwx-- 3 udd uddadm 4096 Jun 16 09:50 /tmp/getwatch.qmapshack.i8ooLp drwx-- 3 udd uddadm 4096 Jun 16 11:20 /tmp/getwatch.qmapshack.JRgmcP drwx-- 3 udd uddadm 4096 Jun 16 13:50 /tmp/getwatch.qmapshack.k7ujsc drwx-- 3 udd uddadm 4096 Jun 16 10:50 /tmp/getwatch.qmapshack.muqRD1 drwx-- 3 udd uddadm 4096 Jun 16 11:50 /tmp/getwatch.qmapshack.VkgQed drwx-- 3 udd uddadm 4096 Jun 16 14:20 /tmp/getwatch.qmapshack.W00S3T drwx-- 3 udd uddadm 4096 Jun 16 12:50 /tmp/getwatch.qmapshack.zPrnz7 Cheers, Julien
Bug#932296: qa.debian.org: getwatch filling up /tmp
On Wed, 2020-06-17 at 00:14 +0200, Lucas Nussbaum wrote:
> On 16/06/20 at 21:07 +0100, Adam D. Barratt wrote:
> > On Tue, 2020-06-16 at 16:40 +0200, Julien Cristau wrote:
> > > On Wed, Dec 18, 2019 at 02:03:13PM +0100, Julien Cristau wrote:
> > > > Control: severity -1 serious
> > > >
> > > > On Thu, Aug 08, 2019 at 01:45:27PM +0200, Julien Cristau wrote:
> > > > > On Wed, Jul 17, 2019 at 10:11:39PM +0200, Lucas Nussbaum
> > > > > wrote:
> > > > > > On 17/07/19 at 14:01 +0200, Julien Cristau wrote:
> > > > > > > something in udd seems to extract entire source packages
> > > > > > > to
> > > > > > > /tmp/getwatch.*. This fills up the disk. Please make it
> > > > > > > not
> > > > > > > do that.
> > [...]
> > > > This happened again. If it won't get fixed I'll go ahead and
> > > > disable that job.
> > > >
> > > Done now, removed the "upstream" importer from the config file.
> > >
> >
> > It looks like that wasn't enough, as ullmann filled its disk again.
> >
> > I've now also updated rudd.conf to disable the importer there.
As a quick note on that, the "disable" key in the configuration doesn't
appear to actually disable anything; from
/srv/udd.debian.org/udd/rlibs/udd-daemon.rb:
def run_importer(imp)
raise 'bugs importer is special' if imp == 'bugs'
if imp.has_key?('disabled')
puts "Not running #{imp['name']}: disabled"
end
init_log if not defined?($log)
$log.debug "Running #{imp['name']}"
So RUDD seems to log that the importer was marked as disabled, and then
run it anyway.
> I emptied the 'upstream' UDD table (no data is better than wrong
> data).
>
> In a previous message, it was proposed to use temporary space under
> /srv, but /srv only has 3.1 GB left. Could you maybe create a
> /srv/udd.debian.org/tmp with maybe 10G ?
>
> Also, does DSA offer the service to send icinga notifications to
> service
> owners? Apparently the condition where this happens is quite rare
> occurences on 08/2019, 12/2019, 06/2020), so notifying me after the
> files were cleaned up from /tmp makes it hard to identify which
> packages cause this issue. If I could get notified when a warning
> limit is reached, it would be much easier to debug.
I'm not sure what the usual policy on that is, but I didn't clean up
/tmp after disabling the importer last night:
drwx-- 3 udd uddadm 4096 Jun 16 20:20 getwatch.qmapshack.n13QHA
drwx-- 3 udd uddadm 4096 Jun 16 20:20 getwatch.picard-tools.Zg0jud
drwx-- 3 udd uddadm 4096 Jun 16 20:50 getwatch.qmapshack.aH184l
drwx-- 3 udd uddadm 4096 Jun 16 20:50 getwatch.picard-tools.SqIkjD
drwx-- 3 udd uddadm 4096 Jun 16 21:20 getwatch.qmapshack.1pIg10
drwx-- 3 udd uddadm 4096 Jun 16 21:20 getwatch.picard-tools.g3weib
drwx-- 3 udd uddadm 4096 Jun 16 21:50 getwatch.qmapshack.oklPSa
drwx-- 3 udd uddadm 4096 Jun 16 21:50 getwatch.picard-tools.Lo3UhJ
So it looks like it's the same couple of packages over and over.
Regards,
Adam
Bug#932296: qa.debian.org: getwatch filling up /tmp
On 16/06/20 at 21:07 +0100, Adam D. Barratt wrote: > On Tue, 2020-06-16 at 16:40 +0200, Julien Cristau wrote: > > On Wed, Dec 18, 2019 at 02:03:13PM +0100, Julien Cristau wrote: > > > Control: severity -1 serious > > > > > > On Thu, Aug 08, 2019 at 01:45:27PM +0200, Julien Cristau wrote: > > > > On Wed, Jul 17, 2019 at 10:11:39PM +0200, Lucas Nussbaum wrote: > > > > > On 17/07/19 at 14:01 +0200, Julien Cristau wrote: > > > > > > something in udd seems to extract entire source packages to > > > > > > /tmp/getwatch.*. This fills up the disk. Please make it not > > > > > > do that. > [...] > > > This happened again. If it won't get fixed I'll go ahead and > > > disable that job. > > > > > Done now, removed the "upstream" importer from the config file. > > > > It looks like that wasn't enough, as ullmann filled its disk again. > > I've now also updated rudd.conf to disable the importer there. I emptied the 'upstream' UDD table (no data is better than wrong data). In a previous message, it was proposed to use temporary space under /srv, but /srv only has 3.1 GB left. Could you maybe create a /srv/udd.debian.org/tmp with maybe 10G ? Also, does DSA offer the service to send icinga notifications to service owners? Apparently the condition where this happens is quite rare occurences on 08/2019, 12/2019, 06/2020), so notifying me after the files were cleaned up from /tmp makes it hard to identify which packages cause this issue. If I could get notified when a warning limit is reached, it would be much easier to debug. Lucas
Bug#932296: qa.debian.org: getwatch filling up /tmp
On Tue, 2020-06-16 at 16:40 +0200, Julien Cristau wrote: > On Wed, Dec 18, 2019 at 02:03:13PM +0100, Julien Cristau wrote: > > Control: severity -1 serious > > > > On Thu, Aug 08, 2019 at 01:45:27PM +0200, Julien Cristau wrote: > > > On Wed, Jul 17, 2019 at 10:11:39PM +0200, Lucas Nussbaum wrote: > > > > On 17/07/19 at 14:01 +0200, Julien Cristau wrote: > > > > > something in udd seems to extract entire source packages to > > > > > /tmp/getwatch.*. This fills up the disk. Please make it not > > > > > do that. [...] > > This happened again. If it won't get fixed I'll go ahead and > > disable that job. > > > Done now, removed the "upstream" importer from the config file. > It looks like that wasn't enough, as ullmann filled its disk again. I've now also updated rudd.conf to disable the importer there. Regards, Adam
Bug#932296: qa.debian.org: getwatch filling up /tmp
On Wed, Dec 18, 2019 at 02:03:13PM +0100, Julien Cristau wrote: > Control: severity -1 serious > > On Thu, Aug 08, 2019 at 01:45:27PM +0200, Julien Cristau wrote: > > On Wed, Jul 17, 2019 at 10:11:39PM +0200, Lucas Nussbaum wrote: > > > On 17/07/19 at 14:01 +0200, Julien Cristau wrote: > > > > something in udd seems to extract entire source packages to > > > > /tmp/getwatch.*. This fills up the disk. Please make it not do that. > > > > > > Hi, > > > > > > Thanks for reporting. > > > > > > It needs to extract the source packages to get the watch file. I don't > > > think there's a way to ask dpkg-source to only extract a single file, > > > and I don't want to re-implement dpkg-source. > > > > > It would be a single call to tar or patch though, which doesn't seem > > like a huge amount of effort. > > > > > Reviewing the code, there was a path where the tmp dir was not removed. > > > I've fixed that. I'm not 100% sure this fixes everything, but it should > > > clearly help. > > > > > There were quite a few getwatch temp dirs before I rebooted ullmann just > > now because it was out of space. > > > > > However, I also note that /tmp is on /, and / is quite small (only 5.3 > > > GB remaining). Would it be possible to add some disk space for /tmp or / > > > on ullmann? > > > > > I'd dispute the "quite small" bit, extracting watch files shouldn't need > > more than 5g. But you could also put your temp files somewhere under > > /srv? > > > This happened again. If it won't get fixed I'll go ahead and disable that > job. > Done now, removed the "upstream" importer from the config file. Cheers, Julien
Bug#932296: qa.debian.org: getwatch filling up /tmp
Control: severity -1 serious On Thu, Aug 08, 2019 at 01:45:27PM +0200, Julien Cristau wrote: > On Wed, Jul 17, 2019 at 10:11:39PM +0200, Lucas Nussbaum wrote: > > On 17/07/19 at 14:01 +0200, Julien Cristau wrote: > > > something in udd seems to extract entire source packages to > > > /tmp/getwatch.*. This fills up the disk. Please make it not do that. > > > > Hi, > > > > Thanks for reporting. > > > > It needs to extract the source packages to get the watch file. I don't > > think there's a way to ask dpkg-source to only extract a single file, > > and I don't want to re-implement dpkg-source. > > > It would be a single call to tar or patch though, which doesn't seem > like a huge amount of effort. > > > Reviewing the code, there was a path where the tmp dir was not removed. > > I've fixed that. I'm not 100% sure this fixes everything, but it should > > clearly help. > > > There were quite a few getwatch temp dirs before I rebooted ullmann just > now because it was out of space. > > > However, I also note that /tmp is on /, and / is quite small (only 5.3 > > GB remaining). Would it be possible to add some disk space for /tmp or / > > on ullmann? > > > I'd dispute the "quite small" bit, extracting watch files shouldn't need > more than 5g. But you could also put your temp files somewhere under > /srv? > This happened again. If it won't get fixed I'll go ahead and disable that job. Cheers, Julien
Bug#932296: qa.debian.org: getwatch filling up /tmp
On Wed, Jul 17, 2019 at 10:11:39PM +0200, Lucas Nussbaum wrote: > On 17/07/19 at 14:01 +0200, Julien Cristau wrote: > > something in udd seems to extract entire source packages to > > /tmp/getwatch.*. This fills up the disk. Please make it not do that. > > Hi, > > Thanks for reporting. > > It needs to extract the source packages to get the watch file. I don't > think there's a way to ask dpkg-source to only extract a single file, > and I don't want to re-implement dpkg-source. > It would be a single call to tar or patch though, which doesn't seem like a huge amount of effort. > Reviewing the code, there was a path where the tmp dir was not removed. > I've fixed that. I'm not 100% sure this fixes everything, but it should > clearly help. > There were quite a few getwatch temp dirs before I rebooted ullmann just now because it was out of space. > However, I also note that /tmp is on /, and / is quite small (only 5.3 > GB remaining). Would it be possible to add some disk space for /tmp or / > on ullmann? > I'd dispute the "quite small" bit, extracting watch files shouldn't need more than 5g. But you could also put your temp files somewhere under /srv? Cheers, Julien
Bug#932296: qa.debian.org: getwatch filling up /tmp
On 17/07/19 at 14:01 +0200, Julien Cristau wrote: > Package: qa.debian.org > Severity: important > User: qa.debian@packages.debian.org > Usertags: udd > X-Debbugs-Cc: lu...@debian.org, debian-ad...@lists.debian.org > > Hi, > > something in udd seems to extract entire source packages to > /tmp/getwatch.*. This fills up the disk. Please make it not do that. Hi, Thanks for reporting. It needs to extract the source packages to get the watch file. I don't think there's a way to ask dpkg-source to only extract a single file, and I don't want to re-implement dpkg-source. Reviewing the code, there was a path where the tmp dir was not removed. I've fixed that. I'm not 100% sure this fixes everything, but it should clearly help. However, I also note that /tmp is on /, and / is quite small (only 5.3 GB remaining). Would it be possible to add some disk space for /tmp or / on ullmann? Thanks Lucas
Bug#932296: qa.debian.org: getwatch filling up /tmp
Package: qa.debian.org Severity: important User: qa.debian@packages.debian.org Usertags: udd X-Debbugs-Cc: lu...@debian.org, debian-ad...@lists.debian.org Hi, something in udd seems to extract entire source packages to /tmp/getwatch.*. This fills up the disk. Please make it not do that. Thanks, Julien
