Re: build of installation-guide on web server broken [Was: Re: debian-doc files on the Debian web server]
Hi, Holger Wansing wrote (Tue, 1 Feb 2022 19:56:19 +0100): > I fixed that, and now it says (here locally): > > ned@t520:~/deb_holgerw/website/on-salsa/cron/cron$ ./lessoften-parts/1ftpfiles > I: Download http://ftp.de.debian.org/debian/dists/sid/main/source/Sources.xz > I: Skip: pool/main/i/installation-guide/installation-guide_20220129.dsc > (already downloaded) > I: Skip: pool/main/i/installation-guide/installation-guide_20220129.tar.xz > (already downloaded) > I: Download > http://ftp.de.debian.org/debian/dists/stable/main/source/Sources.xz > I: Skip: pool/main/i/installation-guide/installation-guide_20210730.dsc > (already downloaded) > I: Skip: pool/main/i/installation-guide/installation-guide_20210730.tar.xz > (already downloaded) > I: Download > http://ftp.de.debian.org/debian/dists/oldstable/main/source/Sources.xz > I: Skip: pool/main/i/installation-guide/installation-guide_20190622.dsc > (already downloaded) > I: Skip: pool/main/i/installation-guide/installation-guide_20190622.tar.xz > (already downloaded) > > and now *.tar.xz and *.dsc files are both there in the target dir. > > So I guess it will work tomorrow as it should. Hmm, it still did not work on wolkenstein. However, it worked on my laptop yesterday, hr. My best guess for this is, that this might be caused by a different version of wget (or some other tool involved), since I'm running Bullseye here on my laptop, and wolkenstein still has Buster. I implemented one more change, should work now (fingers crossed). Holger -- Holger Wansing PGP-Fingerprint: 496A C6E8 1442 4B34 8508 3529 59F1 87CA 156E B076
Re: build of installation-guide on web server broken [Was: Re: debian-doc files on the Debian web server]
Hi,
Osamu Aoki wrote (Wed, 02 Feb 2022 00:29:40 +0900):
> Hi,
>
> I am going to sleep :-)
>
> I will look into this on my week end.
>
> I suppose path needs some adjustment. I don't know where should it be.
>
> Since I have no direct access to the build environment, I need to run cron
> job to
> debug...
Ok.
I set up a test environment here locally and debugged the situation.
Found the error:
The log had:
starting the lessoften cron job at Mon Jan 31 14:49:01 UTC 2022
Already up to date.
run-parts: executing /srv/www.debian.org/cron/lessoften-parts/1ftpfiles
I: Download http://ftp.de.debian.org/debian/dists/sid/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20220129.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20220129.tar.xz
I: Download http://ftp.de.debian.org/debian/dists/stable/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20210730.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20210730.tar.xz
I: Download
http://ftp.de.debian.org/debian/dists/oldstable/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20190622.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20190622.tar.xz
So, it downloads the *.dsc and *.tar.xz files everytime.
That's because the check for the existence of the files used the wrong file path
(it searched for
"pool/main/i/installation-guide/installation-guide_20190622.dsc"
and not for
"installation-guide_20190622.dsc"
That caused the line
rm -f ${srcmask}_*
to be executed everytime, and wiped out the just downloaded *.dsc file, when
going through the for loop the second time (during processing the *.tar.xz
file).
I fixed that, and now it says (here locally):
ned@t520:~/deb_holgerw/website/on-salsa/cron/cron$ ./lessoften-parts/1ftpfiles
I: Download http://ftp.de.debian.org/debian/dists/sid/main/source/Sources.xz
I: Skip: pool/main/i/installation-guide/installation-guide_20220129.dsc
(already downloaded)
I: Skip: pool/main/i/installation-guide/installation-guide_20220129.tar.xz
(already downloaded)
I: Download http://ftp.de.debian.org/debian/dists/stable/main/source/Sources.xz
I: Skip: pool/main/i/installation-guide/installation-guide_20210730.dsc
(already downloaded)
I: Skip: pool/main/i/installation-guide/installation-guide_20210730.tar.xz
(already downloaded)
I: Download
http://ftp.de.debian.org/debian/dists/oldstable/main/source/Sources.xz
I: Skip: pool/main/i/installation-guide/installation-guide_20190622.dsc
(already downloaded)
I: Skip: pool/main/i/installation-guide/installation-guide_20190622.tar.xz
(already downloaded)
and now *.tar.xz and *.dsc files are both there in the target dir.
So I guess it will work tomorrow as it should.
Best regards
Holger
> Osamu
> -Original Message-
> From: Holger Wansing
> To: Osamu Aoki
> Cc: debian-www@lists.debian.org
> Subject: build of installation-guide on web server broken [Was: Re:
> debian-doc files
> on the Debian web server]
> Date: Mon, 31 Jan 2022 19:06:29 +0100
>
> Hi Osamu,
>
>
> Osamu Aoki wrote (Thu, 04 Nov 2021 11:34:06 +0900):
> > Hi, (Sorry)
> >
> > On Wed, 2021-11-03 at 18:14 +0100, Holger Wansing wrote:
> > > Hi Osamu,
> > >
> > > Am 3. November 2021 15:45:44 MEZ schrieb Osamu Aoki :
> > > > Hi,
> > > >
> > > > I finally made update to the cron script so we don't rely on FTP
> > > > protocol.
> > > > url = g...@salsa.debian.org:webmaster-team/cron.git
> > > >
> > > > 526ebd0 ("Download deb via HTTP", 2021-11-01)
> > > >
> > > > No more use of FTP protocol
> > > >
> > > > Ensure to avoid unnecessary downloads of non-sid packages and already
> > > > downloaded packages.
> > > >
> > > > unpack once and do not reprocess
> > > >
> > > > Signed-off-by: Osamu Aoki
> > > >
> > > >
> > > > I stop making a copy of doc. I don't see the following data are used
> > > > any more.
> > > > > $crondir/ftpfiles/doc
> > > >
> > > > Although it was tested on my machine, since this is rather big change,
> > > > I am
> > > > afraid
> > > > something may break.
> > >
> > > As far as I can see, the installation-guide source
> > > packages are
> > > no longer downloaded by 1ftpfiles.
> > > This was handled by this line:
> > >
> > > wgetfiles installation-guide installation-guide_*
> >
> > I thought this was some kind of left-over.
build of installation-guide on web server broken [Was: Re: debian-doc files on the Debian web server]
Hi Osamu,
Osamu Aoki wrote (Thu, 04 Nov 2021 11:34:06 +0900):
> Hi, (Sorry)
>
> On Wed, 2021-11-03 at 18:14 +0100, Holger Wansing wrote:
> > Hi Osamu,
> >
> > Am 3. November 2021 15:45:44 MEZ schrieb Osamu Aoki :
> > > Hi,
> > >
> > > I finally made update to the cron script so we don't rely on FTP protocol.
> > > url = g...@salsa.debian.org:webmaster-team/cron.git
> > >
> > > 526ebd0 ("Download deb via HTTP", 2021-11-01)
> > >
> > >No more use of FTP protocol
> > >
> > >Ensure to avoid unnecessary downloads of non-sid packages and already
> > >downloaded packages.
> > >
> > >unpack once and do not reprocess
> > >
> > >Signed-off-by: Osamu Aoki
> > >
> > >
> > > I stop making a copy of doc. I don't see the following data are used any
> > > more.
> > > > $crondir/ftpfiles/doc
> > >
> > > Although it was tested on my machine, since this is rather big change, I
> > > am afraid
> > > something may break.
> >
> > As far as I can see, the installation-guide source
> > packages are
> > no longer downloaded by 1ftpfiles.
> > This was handled by this line:
> >
> > wgetfiles installation-guide installation-guide_*
>
> I thought this was some kind of left-over. Sorry, you are getting *.dsc file
> and
> tarball here. When I wrote wgetfiles, I didn't anticipate such uses.
>
> > Now, building of installation-guide can no longer work.
> > (the lessoften-parts/1installation-guide script relies on the
> > source files to be existing in $ftpdir/pool dir.)
> >
> > Or did I miss something?
> >
>
> Give me sometime (2 days), I have some idea. Instead of Packages, we need to
> download Sources there. Then it is trivial to download correct version. No
> rmadison
> complication needed.
Two days ago, we had the first upload of installation-guide package since
the above changings.
Now I see, that the build is broken.
At https://www-master.debian.org/build-logs/webwml/lessoften.log I see:
-
starting the lessoften cron job at Sun Jan 30 14:49:01 UTC 2022
Already up to date.
run-parts: executing /srv/www.debian.org/cron/lessoften-parts/1ftpfiles
I: Download http://ftp.de.debian.org/debian/dists/sid/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20220129.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20220129.tar.xz
I: Download http://ftp.de.debian.org/debian/dists/stable/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20210730.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20210730.tar.xz
I: Download
http://ftp.de.debian.org/debian/dists/oldstable/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20190622.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20190622.tar.xz
1ftpfiles finished (at Sun Jan 30 14:49:08 UTC 2022)
run-parts: executing
/srv/www.debian.org/cron/lessoften-parts/1installation-guide
no new installation-guide for stable (bullseye), skipping build
extracting installation-guide source for testing (bookworm) ...
dpkg-source: error: cannot read
/srv/www.debian.org/cron/ftpfiles/sid/installation-guide_20220129.dsc: No such
file or directory
building installation-guide for testing (bookworm) ...
/srv/www.debian.org/cron/lessoften-parts/1installation-guide: line 116: cd:
installation-guide-20220129/build: No such file or directory
FAILED to build the installmanual for testing (bookworm) ...
no new installation-guide for oldstable (buster), skipping build
-
So, even if 1ftpfiles says, it downloads the *.dsc file, it is not there.
The 1installation-guide script however needs it.
Could you take a look?
Thanks
Holger
--
Holger Wansing
PGP-Fingerprint: 496A C6E8 1442 4B34 8508 3529 59F1 87CA 156E B076
Re: debian-doc files on the Debian web server
Hi, Osamu Aoki wrote (Wed, 03 Nov 2021 23:45:44 +0900): > Holger: You are building installation-guide from source. Even though this is > on > 'lessoften' side, it is quite heavy process. Since new 7doc can handle > stable and > oldstable, using 7doc may be an idea. Such build is only triggered, when a new version of installation-guide is uploaded, so only happens approx. 1-3 times a year (wild guess). So I guess it's not worse any effort. Holger -- Holger Wansing PGP-Fingerprint: 496A C6E8 1442 4B34 8508 3529 59F1 87CA 156E B076
Re: debian-doc files on the Debian web server
Hi, Osamu Aoki wrote (Thu, 04 Nov 2021 23:29:46 +0900): > Hi, > > On Thu, 2021-11-04 at 06:28 +0100, Joost van Baal-Ilić wrote: > > > > > > Once this is good, splitting and moving code to lessoften side may be good > > > idea to further reducing CPU load. > > > > Sounds good to me, thanks for your work! > > I just split > > 1ftpfiles > > (I fixed minor bug as I updated) > > Let's see in 12 hours. Looks good now, I think. Thanks Holger -- Holger Wansing PGP-Fingerprint: 496A C6E8 1442 4B34 8508 3529 59F1 87CA 156E B076
Re: debian-doc files on the Debian web server
Hi, On Thu, 2021-11-04 at 06:28 +0100, Joost van Baal-Ilić wrote: > > > > Once this is good, splitting and moving code to lessoften side may be good > > idea to further reducing CPU load. > > Sounds good to me, thanks for your work! I just split > 1ftpfiles (I fixed minor bug as I updated) Let's see in 12 hours. Osamu
Re: debian-doc files on the Debian web server
On Thu, 2021-11-04 at 15:21 +0900, Osamu Aoki wrote: > Hi, > > On Thu, 2021-11-04 at 06:28 +0100, Joost van Baal-Ilić wrote: > > > > Sounds good to me, thanks for your work! > > As I tried to check build recent log, I couldn't initially locate current log > location. > > https://www.debian.org/doc/cvs (OUTDATED) which links to > https://www-master.debian.org/build-logs/ddp/ (OLD dead content) I realize this is only a problem in Japanese. Updated translation.
Re: debian-doc files on the Debian web server
Hi, On Thu, 2021-11-04 at 06:28 +0100, Joost van Baal-Ilić wrote: > > Sounds good to me, thanks for your work! As I tried to check build recent log, I couldn't initially locate current log location. https://www.debian.org/doc/cvs (OUTDATED) which links to https://www-master.debian.org/build-logs/ddp/ (OLD dead content) https://www.debian.org/devel/website/desc Looks OK and https://www-master.debian.org/build-logs/webwml/ (CURRENT) I will check next lessoften run for source package download with the updated code. Osamu
Re: debian-doc files on the Debian web server
Hi, On Thu, Nov 04, 2021 at 12:52:16PM +0900, Osamu Aoki wrote: > On Thu, 2021-11-04 at 11:34 +0900, Osamu Aoki wrote: > > > > Give me sometime (2 days), I have some idea. Instead of Packages, we need > > to download Sources there. Then it is trivial to download correct version. > > No rmadison complication needed. > > Done. > > I renamed some variables and added source package support. > > Once this is good, splitting and moving code to lessoften side may be good > idea to further reducing CPU load. Sounds good to me, thanks for your work! Bye, Joost
Re: debian-doc files on the Debian web server
Done, On Thu, 2021-11-04 at 11:34 +0900, Osamu Aoki wrote: > Hi, (Sorry) > > > Give me sometime (2 days), I have some idea. Instead of Packages, we need to > download Sources there. Then it is trivial to download correct version. No > rmadison > complication needed. Done. I renamed some variables and added source package support. Once this is good, splitting and moving code to lessoften side may be good idea to further reducing CPU load. Osamu
Re: debian-doc files on the Debian web server
Hi, (Sorry)
On Wed, 2021-11-03 at 18:14 +0100, Holger Wansing wrote:
> Hi Osamu,
>
> Am 3. November 2021 15:45:44 MEZ schrieb Osamu Aoki :
> > Hi,
> >
> > I finally made update to the cron script so we don't rely on FTP protocol.
> > url = g...@salsa.debian.org:webmaster-team/cron.git
> >
> > 526ebd0 ("Download deb via HTTP", 2021-11-01)
> >
> >No more use of FTP protocol
> >
> >Ensure to avoid unnecessary downloads of non-sid packages and already
> >downloaded packages.
> >
> >unpack once and do not reprocess
> >
> >Signed-off-by: Osamu Aoki
> >
> >
> > I stop making a copy of doc. I don't see the following data are used any
> > more.
> > > $crondir/ftpfiles/doc
> >
> > Although it was tested on my machine, since this is rather big change, I am
> > afraid
> > something may break.
>
> As far as I can see, the installation-guide source
> packages are
> no longer downloaded by 1ftpfiles.
> This was handled by this line:
>
> wgetfiles installation-guide installation-guide_*
I thought this was some kind of left-over. Sorry, you are getting *.dsc file
and
tarball here. When I wrote wgetfiles, I didn't anticipate such uses.
> Now, building of installation-guide can no longer work.
> (the lessoften-parts/1installation-guide script relies on the
> source files to be existing in $ftpdir/pool dir.)
>
> Or did I miss something?
>
Give me sometime (2 days), I have some idea. Instead of Packages, we need to
download Sources there. Then it is trivial to download correct version. No
rmadison
complication needed.
> > Holger: You are building installation-guide from source. Even though this
> > is on
> > 'lessoften' side, it is quite heavy process. Since new 7doc can handle
> > stable and
> > oldstable, using 7doc may be an idea.
>
> Will see, what can be done Herr.
This is longer term ... no rush.
Re: debian-doc files on the Debian web server
Hi Osamu,
Am 3. November 2021 15:45:44 MEZ schrieb Osamu Aoki :
>Hi,
>
>I finally made update to the cron script so we don't rely on FTP protocol.
>url = g...@salsa.debian.org:webmaster-team/cron.git
>
> 526ebd0 ("Download deb via HTTP", 2021-11-01)
>
>No more use of FTP protocol
>
>Ensure to avoid unnecessary downloads of non-sid packages and already
>downloaded packages.
>
>unpack once and do not reprocess
>
>Signed-off-by: Osamu Aoki
>
>
>I stop making a copy of doc. I don't see the following data are used any more.
>> $crondir/ftpfiles/doc
>
>Although it was tested on my machine, since this is rather big change, I am
>afraid
>something may break.
As far as I can see, the installation-guide source
packages are
no longer downloaded by 1ftpfiles.
This was handled by this line:
wgetfiles installation-guide installation-guide_*
Now, building of installation-guide can no longer work.
(the lessoften-parts/1installation-guide script relies on the
source files to be existing in $ftpdir/pool dir.)
Or did I miss something?
>
>Holger: You are building installation-guide from source. Even though this is
>on
>'lessoften' side, it is quite heavy process. Since new 7doc can handle stable
>and
>oldstable, using 7doc may be an idea.
Will see, what can be done Herr.
Holger
--
Sent from /e/ OS on Fairphone3
debian-doc files on the Debian web server
Hi,
I finally made update to the cron script so we don't rely on FTP protocol.
url = g...@salsa.debian.org:webmaster-team/cron.git
526ebd0 ("Download deb via HTTP", 2021-11-01)
No more use of FTP protocol
Ensure to avoid unnecessary downloads of non-sid packages and already
downloaded packages.
unpack once and do not reprocess
Signed-off-by: Osamu Aoki
I stop making a copy of doc. I don't see the following data are used any more.
> $crondir/ftpfiles/doc
Although it was tested on my machine, since this is rather big change, I am
afraid
something may break.
Holger: You are building installation-guide from source. Even though this is on
'lessoften' side, it is quite heavy process. Since new 7doc can handle stable
and
oldstable, using 7doc may be an idea.
Regards,
Osamu
