The proposal to merge lp:~jtv/launchpad/bug-994650-scrub-faster into
lp:launchpad has been updated.
Description changed to:
The garbo job I'm building to update POFileTranslator records for us was
painfully slow. William pointed out the main reason: the query that fetched
the POFiles that the outer loop iterates over. Each time the outer loop takes
little slices from that query's result, Storm re-executes the query.
So we needed to keep that list in memory, but again as William pointed out,
there are rather too many POFiles to keep comfortably in memory. This branch
loads the POFiles' ids up front, and then postpones the loading of further
details for as long as possible. It does mean handing ids around for some
things that were objects before
In fact, you see three levels of POFile retrieval here:
1. Setup fetches only ids, so that only the ids have to be kept in memory.
2. Per batch, the loop loads just the details needed to find out if the POFile
needs updating: its POTemplate id, its Language id, and its POTMsgSet ids.
3. Any POFile that actually needs fixing (hopefully a minority) gets loaded as
a proper object.
This kind of thing is never fast enough, so there are obvious optimizations
that I may pursue next: cache the POTMsgSet ids as we iterate over POFiles of
the same POTemplate, to help the no-change case even more. And to speed up the
case where there are changes, such as on the initial run, first gather those
POFiles in the batch that need fixing, and batch-load them. The batch-loading
can also cover templates, languages, distroseries, distributions,
productseries, and products. Most of these are needed only for the purpose of
logging which POFile is being updated.
To test:
{{{
# Get database schema set up for this branch.
bzr branch lp:~jtv/launchpad/db-994410
cd db-994410
make schema
cd ..
# Get the prerequisite branch.
bzr branch lp:~jtv/launchpad/bug-994650-scrub-pofiletranslator jtv-scrubber
cd jtv-scrubber
# Merge in the branch under review here.
bzr merge lp:~jtv/launchpad/bug-994650-scrub-faster
make
# Test.
./bin/test -vvc lp.translations.scripts.tests.test_scrub_pofiletranslator
}}}
Alternatively, get the branch that combines all these changes:
{{{
bzr branch lp:~jtv/launchpad/combined-async-pofiletranslator
cd combined-async-pofiletranslator
make
make schema
./bin/test -vvc lp.translations.scripts.tests.test_scrub_pofiletranslator
}}}
Jeroen
For more details, see:
https://code.launchpad.net/~jtv/launchpad/bug-994650-scrub-faster/+merge/105169
--
https://code.launchpad.net/~jtv/launchpad/bug-994650-scrub-faster/+merge/105169
Your team Launchpad code reviewers is requested to review the proposed merge of
lp:~jtv/launchpad/bug-994650-scrub-faster into lp:launchpad.
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-reviewers
Post to : [email protected]
Unsubscribe : https://launchpad.net/~launchpad-reviewers
More help : https://help.launchpad.net/ListHelp
