Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)

2018-06-07 Thread Alex ARNAUD 

On Wed, 24 Jan 2018 17:32:45 +0100 Cyril Brulebois  wrote:

Control: tag -1 patch

Hi Laura,

Thanks for mentioning this on #-www today!


Hello all,

I'm also experiencing such issue on this page: 
https://packages.debian.org/experimental/firefox-esr


Best regards,
Alex.

Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)

2018-01-25 Thread Cyril Brulebois 
Hi Olly,

Olly Betts  (2018-01-26):
> Yes.  The directory on pkgmirror-csail actually had two different
> versions of the database using different database backends, so the
> files which aren't in the directory on picconi are essentially an
> old version of the same database.
> 
> That alone shouldn't cause the search to fail though - it should just
> open one or the other (it looks like both Xapian 1.2.x and 1.4.x will
> open the old one in this rather odd situation).
> 
> Xapian's replication should cleanly handle the database being replicated
> switching backends so I'm guessing the replication here is using rsync
> (without --delete-after) or similar?  That might also explain why the
> old database is broken as the databases aren't safe to rsync if an
> update is in progress - if the last rsync of the old database happened
> while an update was in progress, it could have been left broken, which
> would typically result in particular searches failing.
> 
> Updating with rsync can also break searches on the mirror while the
> rsync is in progress even if the master isn't updating, so perhaps we
> should switch to using Xapian's replication?  That's already being
> used for the main website search (I set it up with weasel at Debconf
> 15).
> 
> Happy to assist with that, though I'll probably need ssh access to
> pkgmirror-csail (seems I currently only have it for picconi).  My
> Debian username is "olly".

Looking around without knowing much about pkg stuff:
 - picconi has: conf/pushpdo.mirror → csail pkgmirror-csail.debian.org pkg_user
 - those bits are read by bin/pushpdo, which calls rsync this way:
 rsync -e "ssh -i ${MKEYFILE} -${MPROTO} ${SSH_OPTS}" -av --stats  
"${MIRRORPATH}"  ${MUSER}@${MHOSTNAME}:/does/not/matter 
>"${LOGDIR}/pushpdo.${MLNAME}.log"

I suppose help is welcome to possibly fix/improve that, but I'm no pkg
expert (I just happened to have helped one or twice in the past, and to
still be in the pkg_maint group…)


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)

2018-01-25 Thread Olly Betts 
On Wed, Jan 24, 2018 at 05:32:45PM +0100, Cyril Brulebois wrote:
> > mod_fcgid: stderr: Can't call method "get_document" on an undefined value 
> > at ../lib/Packages/Search.pm line 264.
> > End of script output before headers: dispatcher.fcgi
> 
> Looking at the code, that's a method call on a xapian query result.
> 
> Looking at the xapian directory on picconi:
> > pkg_user@picconi:/srv/packages.debian.org$ ls -l files/db/xapian/
> > total 302384
> > -rw-r--r-- 1 pkg_user pkg_maint   1974272 Jan 24 15:56 docdata.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 15:53 flintlock
> > -rw-r--r-- 1 pkg_user pkg_maint   138 Jan 24 15:56 iamglass
> > -rw-r--r-- 1 pkg_user pkg_maint 181952512 Jan 24 15:56 position.glass
> > -rw-r--r-- 1 pkg_user pkg_maint  80699392 Jan 24 15:56 postlist.glass
> > -rw-r--r-- 1 pkg_user pkg_maint  44998656 Jan 24 15:56 termlist.glass
> 
> Looking at the xapian directory on the mirror:
> > pkg_user@pkgmirror-csail:/srv/packages.debian.org$ ls -l files/db/xapian/
> > total 537584
> > -rw-r--r-- 1 pkg_user pkg_maint   1974272 Jan 24 09:48 docdata.glass
> > -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 09:45 flintlock
> > -rw-r--r-- 1 pkg_user pkg_maint28 Nov  8 09:26 iamchert
> > -rw-r--r-- 1 pkg_user pkg_maint   138 Jan 24 09:48 iamglass
> > -rw-r--r-- 1 pkg_user pkg_maint  1699 Nov  8 09:30 position.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint  1725 Nov  8 09:30 position.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint 111788032 Nov  8 09:30 position.DB
> > -rw-r--r-- 1 pkg_user pkg_maint 181944320 Jan 24 09:48 position.glass
> > -rw-r--r-- 1 pkg_user pkg_maint  1254 Nov  8 09:30 postlist.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint  1263 Nov  8 09:30 postlist.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint  81616896 Nov  8 09:30 postlist.DB
> > -rw-r--r-- 1 pkg_user pkg_maint  80707584 Jan 24 09:48 postlist.glass
> > -rw-r--r-- 1 pkg_user pkg_maint54 Nov  8 09:30 record.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint56 Nov  8 09:30 record.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint   2523136 Nov  8 09:30 record.DB
> > -rw-r--r-- 1 pkg_user pkg_maint   691 Nov  8 09:30 termlist.baseA
> > -rw-r--r-- 1 pkg_user pkg_maint   703 Nov  8 09:30 termlist.baseB
> > -rw-r--r-- 1 pkg_user pkg_maint  44883968 Nov  8 09:30 termlist.DB
> > -rw-r--r-- 1 pkg_user pkg_maint  45006848 Jan 24 09:48 termlist.glass
> 
> For files dated Jan 24, the timestamps don't match, but that's probably
> just a sync waiting to happen, and that doesn't explain the issues which
> have been happening for so long. I suspected the stray files instead,
> dating back to November. I've created an “old-files” directory and moved
> them there, and the mirror seems to be behaving properly now.
> 
> I'm tagging this bug report with “pending” for the time being, to give
> people a chance to comment. But I suppose it should be safe to remove
> those old files entirely?

Yes.  The directory on pkgmirror-csail actually had two different
versions of the database using different database backends, so the
files which aren't in the directory on picconi are essentially an
old version of the same database.

That alone shouldn't cause the search to fail though - it should just
open one or the other (it looks like both Xapian 1.2.x and 1.4.x will
open the old one in this rather odd situation).

Xapian's replication should cleanly handle the database being replicated
switching backends so I'm guessing the replication here is using rsync
(without --delete-after) or similar?  That might also explain why the
old database is broken as the databases aren't safe to rsync if an
update is in progress - if the last rsync of the old database happened
while an update was in progress, it could have been left broken, which
would typically result in particular searches failing.

Updating with rsync can also break searches on the mirror while the
rsync is in progress even if the master isn't updating, so perhaps we
should switch to using Xapian's replication?  That's already being
used for the main website search (I set it up with weasel at Debconf
15).

Happy to assist with that, though I'll probably need ssh access to
pkgmirror-csail (seems I currently only have it for picconi).  My
Debian username is "olly".

Cheers,
Olly

Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)

2018-01-24 Thread Cyril Brulebois 
Control: tag -1 patch

Hi Laura,

Thanks for mentioning this on #-www today!

Laura Arjona Reina  (2018-01-24):
> I can reproduce the problem with some other packages (I tried several from the
> list https://packages.debian.org/unstable/main/newpkg and got the error in the
> links to buster and sid).
> 
> I also tried
> https://packages.debian.org/stretch-backports/kernel-image-4.14.0-0.bpo.2-arm64-di
> with the same results (error 500).
> 
> Thanks to some help in the IRC channel we've found that only one of our wo
> machines serving packages.debian.org fails:
> 
> https://packages-pkgmirror-csail.debian.org/sid/elpa-find-file-in-project
> gives the 500 Internal Server Error
> but
> https://packages-picconi.debian.org/sid/elpa-find-file-in-project
> works.
> 
> If I can be of any help to solve this, just tell. I'll try to look at the logs
> later, but I'm not sure I have the corresponding permissions (will try).

From the apache error log on the mirror:
> mod_fcgid: stderr: Can't call method "get_document" on an undefined value at 
> ../lib/Packages/Search.pm line 264.
> End of script output before headers: dispatcher.fcgi

Looking at the code, that's a method call on a xapian query result.

Looking at the xapian directory on picconi:
> pkg_user@picconi:/srv/packages.debian.org$ ls -l files/db/xapian/
> total 302384
> -rw-r--r-- 1 pkg_user pkg_maint   1974272 Jan 24 15:56 docdata.glass
> -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 15:53 flintlock
> -rw-r--r-- 1 pkg_user pkg_maint   138 Jan 24 15:56 iamglass
> -rw-r--r-- 1 pkg_user pkg_maint 181952512 Jan 24 15:56 position.glass
> -rw-r--r-- 1 pkg_user pkg_maint  80699392 Jan 24 15:56 postlist.glass
> -rw-r--r-- 1 pkg_user pkg_maint  44998656 Jan 24 15:56 termlist.glass

Looking at the xapian directory on the mirror:
> pkg_user@pkgmirror-csail:/srv/packages.debian.org$ ls -l files/db/xapian/
> total 537584
> -rw-r--r-- 1 pkg_user pkg_maint   1974272 Jan 24 09:48 docdata.glass
> -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 09:45 flintlock
> -rw-r--r-- 1 pkg_user pkg_maint28 Nov  8 09:26 iamchert
> -rw-r--r-- 1 pkg_user pkg_maint   138 Jan 24 09:48 iamglass
> -rw-r--r-- 1 pkg_user pkg_maint  1699 Nov  8 09:30 position.baseA
> -rw-r--r-- 1 pkg_user pkg_maint  1725 Nov  8 09:30 position.baseB
> -rw-r--r-- 1 pkg_user pkg_maint 111788032 Nov  8 09:30 position.DB
> -rw-r--r-- 1 pkg_user pkg_maint 181944320 Jan 24 09:48 position.glass
> -rw-r--r-- 1 pkg_user pkg_maint  1254 Nov  8 09:30 postlist.baseA
> -rw-r--r-- 1 pkg_user pkg_maint  1263 Nov  8 09:30 postlist.baseB
> -rw-r--r-- 1 pkg_user pkg_maint  81616896 Nov  8 09:30 postlist.DB
> -rw-r--r-- 1 pkg_user pkg_maint  80707584 Jan 24 09:48 postlist.glass
> -rw-r--r-- 1 pkg_user pkg_maint54 Nov  8 09:30 record.baseA
> -rw-r--r-- 1 pkg_user pkg_maint56 Nov  8 09:30 record.baseB
> -rw-r--r-- 1 pkg_user pkg_maint   2523136 Nov  8 09:30 record.DB
> -rw-r--r-- 1 pkg_user pkg_maint   691 Nov  8 09:30 termlist.baseA
> -rw-r--r-- 1 pkg_user pkg_maint   703 Nov  8 09:30 termlist.baseB
> -rw-r--r-- 1 pkg_user pkg_maint  44883968 Nov  8 09:30 termlist.DB
> -rw-r--r-- 1 pkg_user pkg_maint  45006848 Jan 24 09:48 termlist.glass

For files dated Jan 24, the timestamps don't match, but that's probably
just a sync waiting to happen, and that doesn't explain the issues which
have been happening for so long. I suspected the stray files instead,
dating back to November. I've created an “old-files” directory and moved
them there, and the mirror seems to be behaving properly now.

I'm tagging this bug report with “pending” for the time being, to give
people a chance to comment. But I suppose it should be safe to remove
those old files entirely?

FTR, that would mean cleaning this:

230M /srv/packages.debian.org/files/db/xapian/old-files/


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)

2018-01-24 Thread Laura Arjona Reina 
Hello

I can reproduce the problem with some other packages (I tried several from the
list https://packages.debian.org/unstable/main/newpkg and got the error in the
links to buster and sid).

I also tried
https://packages.debian.org/stretch-backports/kernel-image-4.14.0-0.bpo.2-arm64-di
with the same results (error 500).

Thanks to some help in the IRC channel we've found that only one of our wo
machines serving packages.debian.org fails:

https://packages-pkgmirror-csail.debian.org/sid/elpa-find-file-in-project
gives the 500 Internal Server Error
but
https://packages-picconi.debian.org/sid/elpa-find-file-in-project
works.

If I can be of any help to solve this, just tell. I'll try to look at the logs
later, but I'm not sure I have the corresponding permissions (will try).

Thanks
-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona