Hello all
Thanks to Nicholas hints, I have managed to make the script skip the
packages that are producing errors; in particular, with these commits:
https://salsa.debian.org/l10n-team/dl10n/commit/ded6ee5874700ddeb5faec86cc16c52ff8f1e4f7
https://salsa.debian.org/l10n-team/dl10n/commit/1af15de8b55d0058d7904d91e7c3f3a14488539c
and now the gen-material script go to the end, but fails when trying to
write the data:
Unable to write to
/srv/i18n.debian.org//dl10n/git/../data/gen-material/data/unstable.gz:
Cannot allocate memory
Same happens with the gen-material call for testing:
Unable to write to
/srv/i18n.debian.org//dl10n/git/../data/gen-material/data/testing.gz:
Cannot allocate memory
I've had a look at the code and I cannot find any place where the
scripts are called with a limitation for memory.
If I am not wrong, this is the stack of calls:
1.- cron:
25 14 * * * run-parts --report /srv/i18n.debian.org/etc/cron.d/
2.- cron.d/10gen-material-unstable
#!/bin/bash
/srv/i18n.debian.org/dl10n/git/cron/gen-material unstable
3.- gen-material unstable
gzip -dc $MIRRORDIR/dists/$d/main/source/Sources.gz \
$MIRRORDIR/dists/$d/contrib/source/Sources.gz \
$MIRRORDIR/dists/$d/non-free/source/Sources.gz \
| grep-dctrl -n -s Directory,Files '' \
| sed -n -e '/^pool\//{N;N;s/\n.* /\//g;p;}' \
| sed -e "s,^,$MIRRORDIR/," \
| grep -Ev "/${IGNMATERIAL}/" \
| $DL10NDIR/dl10n-check \
--remove-unused \
--files-from=- \
--tmp=$TMPDIR \
--db=$OUTDIR/data/$d.gz \
--po=$OUTDIR/po/$d \
--templates=$OUTDIR/templates/$d \
--menu=$OUTDIR/menu/$d $* >> $LOGPREFIX.log 2>> $LOGPREFIX.err
4.- dl10n-check
L326:
$data->write($DB_FILE);
5.- lib/Debian/L10n/Db.pm
L259 (in sub write):
if ($file =~ m/\.gz$/) {
open (DB,"| gzip -c > $file")
|| die "Unable to write to $file: $!\n";
And this is when the program stops with the error "Cannot allocate memory".
What can we do? I've looked at tye.debian.org
$ free -h
totalusedfree shared buff/cache
available
Mem: 2,0G695M 91M5,9M1,2G
1,1G
Swap: 511M151M360M
(But I don't know how to monitor memory while the script is running).
The resultant (written) files shouldn't be very big (the current,
outdated ones are):
[ ] testing.gz 2018-04-09 14:3317K
[ ] unstable.gz 2017-11-23 14:333.9M
So I don't know if there is a problem in the resources in tye, or a
problem in our scripts.
I have manually run the gen-material unstable with --verbose and
--debug, and those produced huge log/err files, but couldn't find any
more detail about the current problem.
Any hint? Should I file an RT ticket for DSA to help us?
--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona
