Jeroen T. Vermeulen has proposed merging
lp:~jtv/launchpad/import-queue-cleanup-blocked into lp:launchpad/devel.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers): code
Related bugs:
#609129 Clean up old blocked Ubuntu PO files
https://bugs.launchpad.net/bugs/609129
= Bug 609129 =
The Translations import queue is big: well over 100K entries. It's been
larger; we're slowly getting it under control while actually importing more and
more files.
This branch whittles down the queue by a further 40% or so. It drops off the
queue any uploads for Ubuntu that contain translations (not templates) in the
gettext format and that haven't seen any changes in the past year. These
uploads were generally blocked automatically because their corresponding
template uploads were manually blocked first. Since I'm not touching the
templates, any future uploads of the same translations are likely to get
blocked again—hence there is little or no value to keeping these entries around.
That's for Ubuntu. I discussed this change with the stakeholders. The
considerations may be different for project uploads, so I'm not touching those.
Similarly, there can be other non-template uploads (only XPI at the moment)
but those are few enough that we probably don't have any blocked entries for
them. Even if we did, the extra complexity of blocking those as well would
only be worth it if there were a noticeable number of them.
To test,
{{{
./bin/test -vvc -m lp.translations.tests.test_autoapproval -t cleanUp
}}}
You'll notice a few drive-by lint cleanups as well.
Jeroen
--
https://code.launchpad.net/~jtv/launchpad/import-queue-cleanup-blocked/+merge/30774
Your team Launchpad code reviewers is requested to review the proposed merge of
lp:~jtv/launchpad/import-queue-cleanup-blocked into lp:launchpad/devel.
=== modified file 'lib/lp/translations/model/translationimportqueue.py'
--- lib/lp/translations/model/translationimportqueue.py 2010-07-02 14:23:47 +0000
+++ lib/lp/translations/model/translationimportqueue.py 2010-07-23 13:41:16 +0000
@@ -23,7 +23,7 @@
from zope.interface import implements
from zope.component import getUtility
from sqlobject import SQLObjectNotFound, StringCol, ForeignKey, BoolCol
-from storm.expr import And, Or
+from storm.expr import And, Like, Or
from storm.locals import Int, Reference
from canonical.database.sqlbase import (
@@ -1307,6 +1307,15 @@
TranslationImportQueueEntry.status == status,
TranslationImportQueueEntry.date_status_changed < cutoff))
+ # Also clean out Blocked PO files for Ubuntu that haven't been
+ # touched for a year. Keep blocked templates because they may
+ # determine the blocking of future translation uploads.
+ blocked_cutoff = now - datetime.timedelta(days=365)
+ deletion_clauses.append(And(
+ TranslationImportQueueEntry.distroseries_id != None,
+ TranslationImportQueueEntry.date_status_changed < blocked_cutoff,
+ Like(TranslationImportQueueEntry.path, '%.po')))
+
entries = store.find(
TranslationImportQueueEntry, Or(*deletion_clauses))
=== modified file 'lib/lp/translations/tests/test_autoapproval.py'
--- lib/lp/translations/tests/test_autoapproval.py 2010-07-21 14:02:28 +0000
+++ lib/lp/translations/tests/test_autoapproval.py 2010-07-23 13:41:16 +0000
@@ -824,6 +824,65 @@
self._exists(entry_id),
"Queue entry in state '%s' was not removed." % status)
+ def test_cleanUpObsoleteEntries_blocked_ubuntu_po(self):
+ # _cleanUpObsoleteEntries deletes Ubuntu entries for gettext
+ # translations that are Blocked if they haven't been touched in
+ # a year. These entries once made up about half the queue. As
+ # far as we can tell all these PO files have been auto-blocked
+ # after their template uploads were blocked, so even if they
+ # were ever re-uploaded, they'd just get blocked again.
+ entry = self._makeDistroEntry()
+ entry.path = 'fo.po'
+ entry_id = entry.id
+ self._setStatus(entry, RosettaImportStatus.BLOCKED)
+ now = datetime.now(UTC)
+ entry.dateimported = now - timedelta(days=700)
+
+ # It hasn't been a year yet since the last status chang; the
+ # entry stays in place.
+ entry.date_status_changed = now - timedelta(days=300)
+ with self.beingTheGardener():
+ self.queue._cleanUpObsoleteEntries(self.store)
+ self.assertTrue(self._exists(entry_id))
+
+ # A year has passed; the entry gets cleaned up.
+ entry.date_status_changed = now - timedelta(days=400)
+ with self.beingTheGardener():
+ self.queue._cleanUpObsoleteEntries(self.store)
+ self.assertFalse(self._exists(entry_id))
+
+ def test_cleanUpObsoleteEntries_blocked_product_po(self):
+ # _cleanUpObsoleteEntries leaves blocked project uploads in
+ # place.
+ entry = self._makeProductEntry()
+ entry.path = 'fo.po'
+ entry_id = entry.id
+ self._setStatus(entry, RosettaImportStatus.BLOCKED)
+ now = datetime.now(UTC)
+ entry.dateimported = now - timedelta(days=700)
+ entry.date_status_changed = now - timedelta(days=400)
+
+ with self.beingTheGardener():
+ self.queue._cleanUpObsoleteEntries(self.store)
+
+ self.assertTrue(self._exists(entry_id))
+
+ def test_cleanUpObsoleteEntries_blocked_ubuntu_pot(self):
+ # _cleanUpObsoleteEntries leaves blocked Ubuntu templates in
+ # place.
+ entry = self._makeDistroEntry()
+ entry.path = 'foo.pot'
+ entry_id = entry.id
+ self._setStatus(entry, RosettaImportStatus.BLOCKED)
+ now = datetime.now(UTC)
+ entry.dateimported = now - timedelta(days=700)
+ entry.date_status_changed = now - timedelta(days=400)
+
+ with self.beingTheGardener():
+ self.queue._cleanUpObsoleteEntries(self.store)
+
+ self.assertTrue(self._exists(entry_id))
+
def test_cleanUpInactiveProductEntries(self):
# After a product is deactivated, _cleanUpInactiveProductEntries
# will clean up any entries it may have on the queue.
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-reviewers
Post to : [email protected]
Unsubscribe : https://launchpad.net/~launchpad-reviewers
More help : https://help.launchpad.net/ListHelp
