Данило Шеган has proposed merging lp:~danilo/launchpad/bug-817398 into lp:launchpad.
Requested reviews: Launchpad code reviewers (launchpad-reviewers) Related bugs: Bug #817398 in Launchpad itself: "Launchpad doesn't generate translation templates" https://bugs.launchpad.net/launchpad/+bug/817398 For more details, see: https://code.launchpad.net/~danilo/launchpad/bug-817398/+merge/79571 = Bug 817398: template generation (WIP) = Template generation has stopped working when buildd default image has switched to Ubuntu Natty (as the development version). That included Python 2.7 by default and generate-translation-templates hard-coded Python 2.6 module path. This is easy to fix, but this has been broken for quite some time (maybe a full year even) because there is no proper logging for TranslationTemplatesBuild jobs. I decided to introduce that to avoid further breakage going unnoticed for this long. == Tests == == Demo and Q/A == * Set up translation import for a project (translations.l.n/<project/+configure-translations and translations.l.n/<project>/trunk/+translations-settings * Push a branch which should generate translation template (eg. lp:eog) to staging, link it to the trunk series (what was set-up above) * Run cronscripts/scan-branches.py to scan the branch and generate translation templates build job * Wait for build-manager to pick it up and build it * When the job finishes, it should keep the log in the builder history available from the builder page on launchpad.net/builders = Launchpad lint = Checking for conflicts and issues in changed files. Linting changed files: utilities/make-lp-user lib/lp/translations/model/translationtemplatesbuildbehavior.py lib/canonical/config/schema-lazr.conf lib/lp/translations/model/translationtemplatesbuild.py lib/canonical/buildd/debian/changelog lib/lp/translations/configure.zcml lib/canonical/buildd/generate-translation-templates lib/lp/code/model/recipebuilder.py ./utilities/make-lp-user 124: Line has trailing whitespace. ./lib/lp/translations/model/translationtemplatesbuildbehavior.py 18: 'transaction' imported but unused 210: W391 blank line at end of file ./lib/canonical/config/schema-lazr.conf 592: Line exceeds 78 characters. 1216: Line exceeds 78 characters. 1223: Line exceeds 78 characters. ./lib/lp/translations/model/translationtemplatesbuild.py 21: 'ProxyFactory' imported but unused 16: 'Store' imported but unused 24: 'ProxiedLibraryFileAlias' imported but unused ./lib/canonical/buildd/debian/changelog 232: Line exceeds 78 characters. 298: Line exceeds 78 characters. 488: Line exceeds 78 characters. 495: Line exceeds 78 characters. 516: Line exceeds 78 characters. 522: Line exceeds 78 characters. 528: Line exceeds 78 characters. 534: Line exceeds 78 characters. 541: Line exceeds 78 characters. 549: Line exceeds 78 characters. 556: Line exceeds 78 characters. 570: Line exceeds 78 characters. 576: Line exceeds 78 characters. 582: Line exceeds 78 characters. 590: Line exceeds 78 characters. 596: Line exceeds 78 characters. 603: Line exceeds 78 characters. 614: Line exceeds 78 characters. 623: Line exceeds 78 characters. 630: Line exceeds 78 characters. 636: Line exceeds 78 characters. 643: Line exceeds 78 characters. 649: Line exceeds 78 characters. 655: Line exceeds 78 characters. 662: Line exceeds 78 characters. 669: Line exceeds 78 characters. 676: Line exceeds 78 characters. 683: Line exceeds 78 characters. 689: Line exceeds 78 characters. ./lib/canonical/buildd/generate-translation-templates 59: Line exceeds 78 characters. ./lib/lp/code/model/recipebuilder.py 131: Line exceeds 78 characters. 132: Line exceeds 78 characters. 132: E501 line too long (81 characters) -- https://code.launchpad.net/~danilo/launchpad/bug-817398/+merge/79571 Your team Launchpad code reviewers is requested to review the proposed merge of lp:~danilo/launchpad/bug-817398 into lp:launchpad.
=== modified file 'lib/canonical/buildd/debian/changelog' --- lib/canonical/buildd/debian/changelog 2011-09-26 06:30:07 +0000 +++ lib/canonical/buildd/debian/changelog 2011-10-17 15:21:59 +0000 @@ -1,3 +1,9 @@ +launchpad-buildd (81) hardy-cat; urgency=low + + * generate-translation-templates: switch to Python 2.7. + + -- Danilo Šegan <[email protected]> Mon, 17 Oct 2011 14:46:13 +0200 + launchpad-buildd (80) hardy-cat; urgency=low * binfmt-support demonstrated umount ordering issues for us. LP: #851934 === modified file 'lib/canonical/buildd/generate-translation-templates' --- lib/canonical/buildd/generate-translation-templates 2010-12-21 17:52:32 +0000 +++ lib/canonical/buildd/generate-translation-templates 2011-10-17 15:21:59 +0000 @@ -41,7 +41,7 @@ BUILDD_PACKAGE=canonical/buildd POTTERY=$BUILDD_PACKAGE/pottery # The script should be smarter about detecting the python version. -PYMODULES=/usr/lib/pymodules/python2.6 +PYMODULES=/usr/lib/pymodules/python2.7 echo -n "Default Python in the chroot is: " $BUILD_CHROOT/usr/bin/python --version @@ -62,4 +62,5 @@ # Enter chroot, switch back to unprivileged user, execute the generate script. $SUDO $CHROOT $BUILD_CHROOT \ - $SU - $USER -c "$GENERATE_SCRIPT $BRANCH_URL $RESULT_NAME" + $SU - $USER \ + -c "PYTHONPATH=$PYMODULES $GENERATE_SCRIPT $BRANCH_URL $RESULT_NAME" === modified file 'lib/canonical/config/schema-lazr.conf' --- lib/canonical/config/schema-lazr.conf 2011-10-05 18:02:45 +0000 +++ lib/canonical/config/schema-lazr.conf 2011-10-17 15:21:59 +0000 @@ -1661,7 +1661,7 @@ banner: none # datatype: string -port: tcp:5022 +port: tcp:5023 # datatype: string ftp_port: 2121 === modified file 'lib/lp/code/model/recipebuilder.py' --- lib/lp/code/model/recipebuilder.py 2010-12-07 00:58:46 +0000 +++ lib/lp/code/model/recipebuilder.py 2011-10-17 15:21:59 +0000 @@ -39,10 +39,6 @@ status = None @property - def build(self): - return self.buildfarmjob.build - - @property def display_name(self): ret = "%s, %s, %s" % ( self.build.distroseries.displayname, self.build.recipe.name, === modified file 'lib/lp/translations/configure.zcml' --- lib/lp/translations/configure.zcml 2011-08-29 00:13:22 +0000 +++ lib/lp/translations/configure.zcml 2011-10-17 15:21:59 +0000 @@ -659,7 +659,11 @@ <!-- TranslationTemplatesBuild --> <class class="lp.translations.model.translationtemplatesbuild.TranslationTemplatesBuild"> - <allow interface="lp.translations.interfaces.translationtemplatesbuild.ITranslationTemplatesBuild"/> + <require permission="launchpad.View" interface="lp.translations.interfaces.translationtemplatesbuild.ITranslationTemplatesBuild"/> + <!-- This is needed for UploadProcessor to run. The permission isn't + important; launchpad.Edit isn't actually held by anybody. --> + <require permission="launchpad.Edit" set_attributes="status" /> + </class> <securedutility component="lp.translations.model.translationtemplatesbuild.TranslationTemplatesBuild" === modified file 'lib/lp/translations/model/translationtemplatesbuild.py' --- lib/lp/translations/model/translationtemplatesbuild.py 2011-05-04 04:10:58 +0000 +++ lib/lp/translations/model/translationtemplatesbuild.py 2011-10-17 15:21:59 +0000 @@ -21,6 +21,7 @@ from zope.security.proxy import ProxyFactory from canonical.launchpad.interfaces.lpstorm import IStore +from canonical.launchpad.browser.librarian import ProxiedLibraryFileAlias from lp.buildmaster.model.buildfarmjob import BuildFarmJobDerived from lp.code.model.branchjob import ( BranchJob, @@ -49,6 +50,11 @@ branch_id = Int(name='branch', allow_none=False) branch = Reference(branch_id, 'Branch.id') + @property + def title(self): + return u'%s for %s' % ( + self.__class__.__name__, self.branch.displayname,) + def __init__(self, build_farm_job, branch): super(TranslationTemplatesBuild, self).__init__() self.build_farm_job = build_farm_job @@ -110,3 +116,10 @@ return store.find( TranslationTemplatesBuild, TranslationTemplatesBuild.branch == branch) + + @property + def log_url(self): + """See `IBuildFarmJob`.""" + if self.log is None: + return None + return self.log.http_url === modified file 'lib/lp/translations/model/translationtemplatesbuildbehavior.py' --- lib/lp/translations/model/translationtemplatesbuildbehavior.py 2011-09-07 15:10:38 +0000 +++ lib/lp/translations/model/translationtemplatesbuildbehavior.py 2011-10-17 15:21:59 +0000 @@ -11,8 +11,11 @@ 'TranslationTemplatesBuildBehavior', ] +import datetime import os +import pytz import tempfile +import transaction from twisted.internet import defer from zope.component import getUtility @@ -20,6 +23,7 @@ from zope.security.proxy import removeSecurityProxy from lp.app.interfaces.launchpad import ILaunchpadCelebrities +from lp.buildmaster.enums import BuildStatus from lp.buildmaster.interfaces.buildfarmjobbehavior import ( IBuildFarmJobBehavior, ) @@ -111,6 +115,38 @@ if len(raw_slave_status) >= 4: status['filemap'] = raw_slave_status[3] + def setBuildStatus(self, status): + self.build.status = status + + @staticmethod + def getLogFromSlave(templates_build, queue_item): + """See `IPackageBuild`.""" + SLAVE_LOG_FILENAME = 'buildlog' + builder = queue_item.builder + d = builder.transferSlaveFileToLibrarian( + SLAVE_LOG_FILENAME, + templates_build.buildfarmjob.getLogFileName(), + False) + return d + + @staticmethod + def storeBuildInfo(build, queue_item, build_status): + """See `IPackageBuild`.""" + def got_log(lfa_id): + # log, builder and date_finished are read-only, so we must + # currently remove the security proxy to set them. + naked_build = removeSecurityProxy(build.build) + naked_build.log = lfa_id + naked_build.builder = queue_item.builder + naked_build.date_started = queue_item.date_started + # XXX cprov 20060615 bug=120584: Currently buildduration includes + # the scanner latency, it should really be asking the slave for + # the duration spent building locally. + naked_build.date_finished = datetime.datetime.now(pytz.UTC) + + d = build.getLogFromSlave(build, queue_item) + return d.addCallback(got_log) + def updateBuild_WAITING(self, queue_item, slave_status, logtail, logger): """Deal with a finished ("WAITING" state, perversely) build job. @@ -121,7 +157,6 @@ retry it. """ build_status = self.extractBuildStatus(slave_status) - logger.info( "Templates generation job %s for %s finished with status %s." % ( queue_item.specific_job.getName(), @@ -152,15 +187,24 @@ queue_item.specific_job.branch, tarball, logger) logger.debug("Upload complete.") finally: + self.setBuildStatus(BuildStatus.FULLYBUILT) tarball_file.close() os.remove(filename) - if build_status == 'OK': - logger.debug("Processing successful templates build.") - filemap = slave_status.get('filemap') - d = self._readTarball(queue_item, filemap, logger) - d.addCallback(got_tarball) - d.addCallback(clean_slave) - return d - - return clean_slave(None) + def build_info_stored(ignored): + if build_status == 'OK': + self.setBuildStatus(BuildStatus.UPLOADING) + logger.debug("Processing successful templates build.") + filemap = slave_status.get('filemap') + d = self._readTarball(queue_item, filemap, logger) + d.addCallback(got_tarball) + d.addCallback(clean_slave) + return d + + self.setBuildStatus(BuildStatus.FAILEDTOBUILD) + return clean_slave(None) + + d = self.storeBuildInfo(self, queue_item, build_status) + d.addCallback(build_info_stored) + return d + === modified file 'utilities/make-lp-user' --- utilities/make-lp-user 2011-07-11 03:49:12 +0000 +++ utilities/make-lp-user 2011-10-17 15:21:59 +0000 @@ -44,6 +44,7 @@ from canonical.launchpad.interfaces.gpghandler import IGPGHandler from canonical.launchpad.scripts import execute_zcml_for_scripts +from canonical.lazr.timeout import set_default_timeout_function from lp.registry.interfaces.gpg import GPGKeyAlgorithm, IGPGKeySet from lp.registry.interfaces.person import IPersonSet from lp.registry.interfaces.ssh import ISSHKeySet @@ -57,6 +58,7 @@ DEFAULT_PASSWORD = 'test' factory = LaunchpadObjectFactory() +set_default_timeout_function(lambda: 100) def make_person(username, email): """Create and return a person with the given username.
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
