Bug#887107: Bug 887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Hello all Thanks to Nicholas hints, I have managed to make the script skip the packages that are producing errors; in particular, with these commits: https://salsa.debian.org/l10n-team/dl10n/commit/ded6ee5874700ddeb5faec86cc16c52ff8f1e4f7 https://salsa.debian.org/l10n-team/dl10n/commit/1af15de8b55d0058d7904d91e7c3f3a14488539c and now the gen-material script go to the end, but fails when trying to write the data: Unable to write to /srv/i18n.debian.org//dl10n/git/../data/gen-material/data/unstable.gz: Cannot allocate memory Same happens with the gen-material call for testing: Unable to write to /srv/i18n.debian.org//dl10n/git/../data/gen-material/data/testing.gz: Cannot allocate memory I've had a look at the code and I cannot find any place where the scripts are called with a limitation for memory. If I am not wrong, this is the stack of calls: 1.- cron: 25 14 * * * run-parts --report /srv/i18n.debian.org/etc/cron.d/ 2.- cron.d/10gen-material-unstable #!/bin/bash /srv/i18n.debian.org/dl10n/git/cron/gen-material unstable 3.- gen-material unstable gzip -dc $MIRRORDIR/dists/$d/main/source/Sources.gz \ $MIRRORDIR/dists/$d/contrib/source/Sources.gz \ $MIRRORDIR/dists/$d/non-free/source/Sources.gz \ | grep-dctrl -n -s Directory,Files '' \ | sed -n -e '/^pool\//{N;N;s/\n.* /\//g;p;}' \ | sed -e "s,^,$MIRRORDIR/," \ | grep -Ev "/${IGNMATERIAL}/" \ | $DL10NDIR/dl10n-check \ --remove-unused \ --files-from=- \ --tmp=$TMPDIR \ --db=$OUTDIR/data/$d.gz \ --po=$OUTDIR/po/$d \ --templates=$OUTDIR/templates/$d \ --menu=$OUTDIR/menu/$d $* >> $LOGPREFIX.log 2>> $LOGPREFIX.err 4.- dl10n-check L326: $data->write($DB_FILE); 5.- lib/Debian/L10n/Db.pm L259 (in sub write): if ($file =~ m/\.gz$/) { open (DB,"| gzip -c > $file") || die "Unable to write to $file: $!\n"; And this is when the program stops with the error "Cannot allocate memory". What can we do? I've looked at tye.debian.org $ free -h totalusedfree shared buff/cache available Mem: 2,0G695M 91M5,9M1,2G 1,1G Swap: 511M151M360M (But I don't know how to monitor memory while the script is running). The resultant (written) files shouldn't be very big (the current, outdated ones are): [ ] testing.gz 2018-04-09 14:3317K [ ] unstable.gz 2017-11-23 14:333.9M So I don't know if there is a problem in the resources in tye, or a problem in our scripts. I have manually run the gen-material unstable with --verbose and --debug, and those produced huge log/err files, but couldn't find any more detail about the current problem. Any hint? Should I file an RT ticket for DSA to help us? -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23 - too much packages to add to the ignore_list
Hello again El 18/04/18 a las 01:00, Laura Arjona Reina escribió: > Hello > After running the gen-material script several times adding as exceptions > the packages that were producing errors, I don't know if it makes sense > to go on this way. It seems that en each run, one or two new packages > produce errors. Indeed, this is the last error log: --- cat /srv/i18n.debian.org/log/gen-material/gen-material.20180417-2241.err Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/fr.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_fr.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/ru.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_ru.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/oc.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_oc.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/da.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_da.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/lt.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_lt.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/s...@latin.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2...@latin.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/gl.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_gl.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/sr.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_sr.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/nb.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_nb.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/zh_CN.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_zh_CN.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/hu.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_hu.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/eu.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_eu.po.gz": No such file or directory Cannot run: gzip -9f -c "/srv/i18n.debian.org//tmp/gen-material/gcab/po/cs.po" > "/srv/i18n.debian.org//dl10n/git/../data/gen-material/po/unstable/main/g/gcab/po/gcab_1.1-2_cs.po.gz": No such file or directory Unable to open /srv/mirrors/debian//pool/main/g/gcc-3.3/gcc-3.3_3.3.6ds1.orig.tar.gz at /srv/i18n.debian.org//dl10n/git/dl10n-check line 463. Unable to open /srv/mirrors/debian//pool/main/g/gcc-5/gcc-5_5.5.0.orig.tar.gz at /srv/i18n.debian.org//dl10n/git/dl10n-check line 463. Unable to open /srv/mirrors/debian//pool/main/g/gcc-5/gcc-5_5.5.0.orig.tar.gz at /srv/i18n.debian.org//dl10n/git/dl10n-check line 463. read() on closed filehandle GEN3287 at /srv/i18n.debian.org//dl10n/git/lib/Debian/Pkg/Tar.pm line 176. Failed to read `/srv/mirrors/debian//pool/main/g/gcc-5/gcc-5_5.5.0.orig.tar.gz': Bad file descriptor at /srv/i18n.debian.org//dl10n/git/dl10n-check line 463. 'x' outside of string in unpack at /srv/i18n.debian.org//dl10n/git/lib/Debian/Pkg/Tar.pm line 368. --- I am sorry but I'm giving up adding packages as exception, we have a long list already and we're only in letter "g". I've had a look at the code but frankly I have no idea about how to solve this problem :( Regards, -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23 - too much packages to add to the ignore_list
Hello After running the gen-material script several times adding as exceptions the packages that were producing errors, I don't know if it makes sense to go on this way. It seems that en each run, one or two new packages produce errors. I believe some errors are being produced due to this bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748716 l10n scripts should not look at upstream debian/ of source/format 3.0 (quilt) packages And others in problems in this script: https://salsa.debian.org/l10n-team/dl10n/blob/master/lib/Debian/Pkg/Tar.pm If anybody with Perl skills can give a hand, it's appreciated. OTOH, I don't know if this bug should be reasigned to the dl10n package. I see there are other bugs there and even some patches, but I don't know if the dl10n package uses a different repo as source. The version uploaded to sid and the version in https://salsa.debian.org/l10n-team/dl10n are not the same; I don't know if this is just because the contents of the dl10n repo have not been uploaded to sid recently. Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Hello I've had a look at today's log and there is only the warning that is reported in bug #890083 (dpkg --compare-versions using obsolete operator). Guillem Jover ACK'ed my patch for #890083 and thus I have applied it in the repo. Christian, do you mind to update again the repo in tye? Let's hope tomorrow we have our unstable.gz updated :) Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Hi again El 14/04/18 a las 07:31, Christian PERRIER escribió: > And I launched thte > /srv/i18n.debian.org/etc/cron.d/10gen-material-unstable script > manually, just to see what happens. I've had a look at the log and added one more package to the exception list (gcc-7). I think after that the script should update the material (until some other package is updated in unstable and produces error :s ). I've requested to join the debian-i18n unix group (RT#7216), but in the meanwhile (or for the case it's not appropriate), do you mind to git pull again? > Indeed, we get a daily cron > message which I (sadly) ignored for ages.that's quite probably the > problem. > > debian-i18n crontab on tye has: > MAILTO="brot...@debian.org,bubu...@debian.org,f...@debian.org,nek...@debian.org,taf...@debian.org" > > but it seems that all of us are ignoring these mails nowadays. > > I don't remember how one can be added to the right group on Debian > machines and be allowed to "sudo" to debian-i18n. I suspect this > should be done with a ticket to the Debian admin team. > > At minimum, I can add your mail address to this crontab, Laura, tthat > may help Thanks, I'll try to have a look at those mails and see if there is something I can do. Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Quoting Laura Arjona Reina (larj...@debian.org): > Hello > The material data has not been updated, there are more packages > producing errors. > I've added them as exceptions to the config file, to workaround the > issue (as with my previous patch). > Christian, do you mind to update again the repo in tye, to see if > tomorrow the material data is produced? > I wouldn't mind to join the debian-i18n unix group so I can do it > myself, if that's ok with you too, but I'm not sure about the procedure. > Thanks! I just "git pulled" again. And I launched thte /srv/i18n.debian.org/etc/cron.d/10gen-material-unstable script manually, just to see what happens. Indeed, we get a daily cron message which I (sadly) ignored for ages.that's quite probably the problem. debian-i18n crontab on tye has: MAILTO="brot...@debian.org,bubu...@debian.org,f...@debian.org,nek...@debian.org,taf...@debian.org" but it seems that all of us are ignoring these mails nowadays. I don't remember how one can be added to the right group on Debian machines and be allowed to "sudo" to debian-i18n. I suspect this should be done with a ticket to the Debian admin team. At minimum, I can add your mail address to this crontab, Laura, tthat may help signature.asc Description: PGP signature
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Hello The material data has not been updated, there are more packages producing errors. I've added them as exceptions to the config file, to workaround the issue (as with my previous patch). Christian, do you mind to update again the repo in tye, to see if tomorrow the material data is produced? I wouldn't mind to join the debian-i18n unix group so I can do it myself, if that's ok with you too, but I'm not sure about the procedure. Thanks! El 12/04/18 a las 18:52, Christian PERRIER escribió: > Quoting Laura Arjona Reina (larj...@debian.org): >> Hi again >> >> El 12/04/18 a las 07:12, Christian PERRIER escribió: >> >>> Doh, I'm now in the salsa mess: >>> debian-i18n@tye:/srv/i18n.debian.org/dl10n/git$ git pull >>> fatal: unable to access 'https://salsa.debian.org/l10n-team/dl10n.git/': >>> server certificate verification failed. CAfile: >>> /etc/ssl/certs/ca-certificates.crt CRLfile: none >> Sorry, my fault, I gave incomplete directions. >> There is also needed to do the following (in tye.debian.org): >> >> dir=/etc/ssl/ca-debian >> test -d $dir && git config --local --add http.sslCAInfo >> $dir/ca-certificates.crt >> >> (This is documented in https://wiki.debian.org/ServicesSSL#git ) > > Gracias! > > It just worked now and the local copy on tye is now synced again with > the git repo on salsa. > > -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
El 12 de abril de 2018 18:52:56 CEST, Christian PERRIERescribió: >Quoting Laura Arjona Reina (larj...@debian.org): >> Hi again >> >> El 12/04/18 a las 07:12, Christian PERRIER escribió: >> >> > Doh, I'm now in the salsa mess: >> > debian-i18n@tye:/srv/i18n.debian.org/dl10n/git$ git pull >> > fatal: unable to access >'https://salsa.debian.org/l10n-team/dl10n.git/': server certificate >verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt >CRLfile: none >> Sorry, my fault, I gave incomplete directions. >> There is also needed to do the following (in tye.debian.org): >> >> dir=/etc/ssl/ca-debian >> test -d $dir && git config --local --add http.sslCAInfo >$dir/ca-certificates.crt >> >> (This is documented in https://wiki.debian.org/ServicesSSL#git ) > > >Gracias! > >It just worked now and the local copy on tye is now synced again with >the git repo on salsa. Merci! Let's see what happens tomorrow in the next run. Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona Sent with K-9 mail
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Quoting Laura Arjona Reina (larj...@debian.org): > Hi again > > El 12/04/18 a las 07:12, Christian PERRIER escribió: > > > Doh, I'm now in the salsa mess: > > debian-i18n@tye:/srv/i18n.debian.org/dl10n/git$ git pull > > fatal: unable to access 'https://salsa.debian.org/l10n-team/dl10n.git/': > > server certificate verification failed. CAfile: > > /etc/ssl/certs/ca-certificates.crt CRLfile: none > Sorry, my fault, I gave incomplete directions. > There is also needed to do the following (in tye.debian.org): > > dir=/etc/ssl/ca-debian > test -d $dir && git config --local --add http.sslCAInfo > $dir/ca-certificates.crt > > (This is documented in https://wiki.debian.org/ServicesSSL#git ) Gracias! It just worked now and the local copy on tye is now synced again with the git repo on salsa. signature.asc Description: PGP signature
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Hi again El 12/04/18 a las 07:12, Christian PERRIER escribió: > Doh, I'm now in the salsa mess: > debian-i18n@tye:/srv/i18n.debian.org/dl10n/git$ git pull > fatal: unable to access 'https://salsa.debian.org/l10n-team/dl10n.git/': > server certificate verification failed. CAfile: > /etc/ssl/certs/ca-certificates.crt CRLfile: none Sorry, my fault, I gave incomplete directions. There is also needed to do the following (in tye.debian.org): dir=/etc/ssl/ca-debian test -d $dir && git config --local --add http.sslCAInfo $dir/ca-certificates.crt (This is documented in https://wiki.debian.org/ServicesSSL#git ) Thanks! -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Quoting Laura Arjona Reina (larj...@debian.org): > > > El 12 de abril de 2018 6:52:28 CEST, Christian PERRIER> escribió: > >Quoting Laura Arjona Reina (larj...@debian.org): > >> Hello all > >> I applied a patch to the dl10n repo in salsa, with a workaround for > >this > >> bug (adding the problematic packages to an exclusion list). > >> However, the material is not being updated, I think because the repo > >in > >> tye still uses alioth as origin. > >> > >> Can somebody in the debian-i18n group update the config in > >> /srv/i18n.debian.org/dl10n/git/ so it uses the salsa repo? > >> > >> (I'm not sure if manual git pull is needed too, or if there is some > >cron > >> script that auto-updates the repo at some time. I'm not in the > >> debian-i18n group so I think I cannot check... > > > > > >Hi Laura, > > > >Thanks for your work. > > > >I logged in to i18n.d.o, then sudo'ed to debian-i18n, then moved to > >/srv/i18n.debian.org/dl10n/git > > > >"git pull" mentions that the local copy is up-to-date and "git log" > >says : > > > >commit d17144ddc047f48bc4609202bb6e48d04fe92f33 > >Author: David Prévot > >Date: Wed Jan 7 12:12:43 2015 -0400 > > > >Workaround recent SSL breakage from DSA > > > ><1420638006.22794.18.ca...@debian.org>. > > > > > >So, it looks like I can't have your patch make its way to the > >server:-( > > > > Please modify the .git/config file to change the origin to salsa instead of > alioth: > > https://salsa.debian.org/l10n-team/dl10n.git > > Then save and git pull. Doh, I'm now in the salsa mess: debian-i18n@tye:/srv/i18n.debian.org/dl10n/git$ git pull fatal: unable to access 'https://salsa.debian.org/l10n-team/dl10n.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none (for the record, I didn't move anything of what I personnally maintain, to salsa: I decided that It's no longer time for me to learn how to make things work with new stuff..call that lazyness, it's prrobably more a lack of motivation) So, well, help is welcomed again, here, sorry for being a n00b. signature.asc Description: PGP signature
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
El 12 de abril de 2018 6:52:28 CEST, Christian PERRIERescribió: >Quoting Laura Arjona Reina (larj...@debian.org): >> Hello all >> I applied a patch to the dl10n repo in salsa, with a workaround for >this >> bug (adding the problematic packages to an exclusion list). >> However, the material is not being updated, I think because the repo >in >> tye still uses alioth as origin. >> >> Can somebody in the debian-i18n group update the config in >> /srv/i18n.debian.org/dl10n/git/ so it uses the salsa repo? >> >> (I'm not sure if manual git pull is needed too, or if there is some >cron >> script that auto-updates the repo at some time. I'm not in the >> debian-i18n group so I think I cannot check... > > >Hi Laura, > >Thanks for your work. > >I logged in to i18n.d.o, then sudo'ed to debian-i18n, then moved to >/srv/i18n.debian.org/dl10n/git > >"git pull" mentions that the local copy is up-to-date and "git log" >says : > >commit d17144ddc047f48bc4609202bb6e48d04fe92f33 >Author: David Prévot >Date: Wed Jan 7 12:12:43 2015 -0400 > >Workaround recent SSL breakage from DSA > ><1420638006.22794.18.ca...@debian.org>. > > >So, it looks like I can't have your patch make its way to the >server:-( > Please modify the .git/config file to change the origin to salsa instead of alioth: https://salsa.debian.org/l10n-team/dl10n.git Then save and git pull. Thanks! >From what I remember, there is no auto update of the local git >repository copy, it has to be pulled manually. > >debian-i18n@tye:/srv/i18n.debian.org/dl10n/git$ git remote -v >origin git://anonscm.debian.org/debian-l10n/dl10n.git (fetch) >origin git://anonscm.debian.org/debian-l10n/dl10n.git (push) > >I think that, indeed, it would be good if someone more active than me >would apply to be included in the debian-i18n group -- Laura Arjona Reina https://wiki.debian.org/LauraArjona Sent with K-9 mail
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Quoting Laura Arjona Reina (larj...@debian.org): > Hello all > I applied a patch to the dl10n repo in salsa, with a workaround for this > bug (adding the problematic packages to an exclusion list). > However, the material is not being updated, I think because the repo in > tye still uses alioth as origin. > > Can somebody in the debian-i18n group update the config in > /srv/i18n.debian.org/dl10n/git/ so it uses the salsa repo? > > (I'm not sure if manual git pull is needed too, or if there is some cron > script that auto-updates the repo at some time. I'm not in the > debian-i18n group so I think I cannot check... Hi Laura, Thanks for your work. I logged in to i18n.d.o, then sudo'ed to debian-i18n, then moved to /srv/i18n.debian.org/dl10n/git "git pull" mentions that the local copy is up-to-date and "git log" says : commit d17144ddc047f48bc4609202bb6e48d04fe92f33 Author: David PrévotDate: Wed Jan 7 12:12:43 2015 -0400 Workaround recent SSL breakage from DSA <1420638006.22794.18.ca...@debian.org>. So, it looks like I can't have your patch make its way to the server:-( From what I remember, there is no auto update of the local git repository copy, it has to be pulled manually. debian-i18n@tye:/srv/i18n.debian.org/dl10n/git$ git remote -v origin git://anonscm.debian.org/debian-l10n/dl10n.git (fetch) origin git://anonscm.debian.org/debian-l10n/dl10n.git (push) I think that, indeed, it would be good if someone more active than me would apply to be included in the debian-i18n group signature.asc Description: PGP signature
Bug#887107: https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Hello all I applied a patch to the dl10n repo in salsa, with a workaround for this bug (adding the problematic packages to an exclusion list). However, the material is not being updated, I think because the repo in tye still uses alioth as origin. Can somebody in the debian-i18n group update the config in /srv/i18n.debian.org/dl10n/git/ so it uses the salsa repo? (I'm not sure if manual git pull is needed too, or if there is some cron script that auto-updates the repo at some time. I'm not in the debian-i18n group so I think I cannot check... Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#887107: Bug #887107 https://i18n.debian.org/material/data/unstable.gz not updated since 2017-11-23
Hi all I see there are some packages excludes from the gen-material script, due to different issues: https://anonscm.debian.org/cgit/debian-l10n/dl10n.git/tree/etc/dl10n.conf Maybe we can workaround the problem adding the packages that are problematic now to the exclusion list. I'll try to put some time on this later. Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona