Bug#658937: testing.pl: needs rewriting
On Mon, Feb 13, 2012 at 01:23:49PM +, Adam D. Barratt wrote: On 06.02.2012 20:58, Adam D. Barratt wrote: migration/testing.pl needs to be updated to make it react better when called repeatedly, or to avoid the need to do so. Right now it's sufficiently resource hungry to make a 16-core machine unhappy. An initial idea might be to statically generate package pages, although this would need to be benchmarked for feasibility. testing.pl could then simply become a redirect to /package/blah or similar. Stupid question: is it running under mod_perl? If so, surely there's some way to cache the database between invocations so it doesn't have to be regenerated on each call? Please ignore me if I'm talking nonsense Julian -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#658937: testing.pl: needs rewriting
On 10.05.2012 17:42, Julian Gilbey wrote: On Mon, Feb 13, 2012 at 01:23:49PM +, Adam D. Barratt wrote: On 06.02.2012 20:58, Adam D. Barratt wrote: migration/testing.pl needs to be updated to make it react better when called repeatedly, or to avoid the need to do so. Right now it's sufficiently resource hungry to make a 16-core machine unhappy. An initial idea might be to statically generate package pages, although this would need to be benchmarked for feasibility. testing.pl could then simply become a redirect to /package/blah or similar. Stupid question: is it running under mod_perl? The server in question doesn't appear to have libapache2-mod-perl2 installed, so I'm going to say no. ;-) If so, surely there's some way to cache the database between invocations so it doesn't have to be regenerated on each call? The per-package pages were already cached and it's that cache that we're currently using, via mod_rewrite, to serve the pages that are linked from services such as the PTS. There's no support for any of the other querystring parameters right now, because we'd end up with n-combinatorial versions of each file around. In theory, yes, it could probably be rewritten to store the interesting data somewhere and cross-reference that, rather than re-reading the packages files all the time. As I mentioned in my mail opening this report though, whatever is available needs to be sufficiently sane not to make a machine of the size of ftp-master/release.d.o fall over when comment spammers spider several thousand of the pages in a short time and thus make DSA frown a good deal. Regards, Adam -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#658937: testing.pl: needs rewriting
On 06.02.2012 20:58, Adam D. Barratt wrote: migration/testing.pl needs to be updated to make it react better when called repeatedly, or to avoid the need to do so. Right now it's sufficiently resource hungry to make a 16-core machine unhappy. An initial idea might be to statically generate package pages, although this would need to be benchmarked for feasibility. testing.pl could then simply become a redirect to /package/blah or similar. It turns out that the cron job already generated cache files for the packages pages, which were then returned by the CGI (except presumably for the brief window while they were being re-generated). testing.pl?package=$foo now serves up the cache page via mod_rewrite. This does mean that there's a brief window once an hour where the pages aren't available, but that could be resolved by having the cron job use a temporary data store which is then swapped to live. The extra query string parameters aren't currently supported. We should work out which are actually in use, and whether we can re-introduce support for them in a sane way (without either killing franck or exploding the file system use). Adam -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#658937: testing.pl: needs rewriting
Package: release.debian.org Severity: important User: release.debian@packages.debian.org Usertag: tools migration/testing.pl needs to be updated to make it react better when called repeatedly, or to avoid the need to do so. Right now it's sufficiently resource hungry to make a 16-core machine unhappy. An initial idea might be to statically generate package pages, although this would need to be benchmarked for feasibility. testing.pl could then simply become a redirect to /package/blah or similar. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
