Hi Chris, in between things I've applied a few changes to Wikibrowse that make it quite a bit faster. Feel free to review/comment/pull from http://dev.laptop.org/git/users/martin/wikiserver/
= Batch inline templates = Turns out most templates are only used once or twice. If we blindly inline all templates, page load times drop dramatically and the bzip file shrinks. So there's a mergetemplates.py in tools. It also sorts all pages (which tends to put redirects next to their target too). With those things combined, on XO-1.5 - es_PE 'processed' file shrinks from 84MB to 81MB - Argentina goes from 31s to 12s (it's a 315KB page -- tons of parsing) - Andorra goes from 22s to 2.6 (71KB page -- a bit less parsing effort, but had lots of nested templates) - Lógica goes from 6s to 3s I tried various other strategies with templates. Given that most templates are single use, and the frequent-use templates have a tiny 'payload', inlining them all makes the most sense. The batch inliner takes 7hs on my (slowish) laptop(!). =Various fixes that improve resiliency= -- we couldn't process quite a few pages =pyrlu cache= A bit less important now that we're using inline templates, but still a big win re-visiting pages. On its own, it'd give a good 20% improvement "cold cache" because many pages use tiny templates a zillion times. = What's next= I'll probably prepare a releae of Wikipedia Spanish and English at least with these and a few content edits -- perhaps with your help? cheers, m -- [email protected] [email protected] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff _______________________________________________ Devel mailing list [email protected] http://lists.laptop.org/listinfo/devel
