Colin Watson has proposed merging
lp:~cjwatson/launchpad-buildd/search-partial-log into lp:launchpad-buildd.
Commit message:
Stop searching sbuild logs when we see "Toolchain package versions:".
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~cjwatson/launchpad-buildd/search-partial-log/+merge/187700
When QAing
https://code.launchpad.net/~cjwatson/launchpad-buildd/large-build-artifacts/+merge/186503,
I discovered that in the very case at hand - a large build log -
launchpad-buildd takes about three minutes to process the log during which time
it isn't responding to RPC requests from the master, so the master times it out
and cancels the build.
The log file searches we're doing here only need to care about the phase of the
build up to the end of build-dependency installation. Therefore, a reasonable
fix is to stop searching when we see "Toolchain package versions:". (I thought
about doing this when preparing my previous branch, but the problem of failing
to respond to RPC requests hadn't occurred to me, so I postponed it.) On my
system, this takes the search time down to 0.1 seconds in the worst case where
there's no match for any of the relevant regexes.
--
https://code.launchpad.net/~cjwatson/launchpad-buildd/search-partial-log/+merge/187700
Your team Launchpad code reviewers is requested to review the proposed merge of
lp:~cjwatson/launchpad-buildd/search-partial-log into lp:launchpad-buildd.
=== modified file 'lpbuildd/binarypackage.py'
--- lpbuildd/binarypackage.py 2013-09-24 08:58:57 +0000
+++ lpbuildd/binarypackage.py 2013-09-26 09:44:30 +0000
@@ -90,6 +90,7 @@
"""Finished the sbuild run."""
if success != SBuildExitCodes.OK:
log_patterns = []
+ stop_patterns = [["^Toolchain package versions:", re.M]]
if (success == SBuildExitCodes.DEPFAIL or
success == SBuildExitCodes.PACKAGEFAIL):
@@ -101,7 +102,7 @@
log_patterns.append([rx, re.M])
if log_patterns:
- rx, mo = self.searchLogContents(log_patterns)
+ rx, mo = self.searchLogContents(log_patterns, stop_patterns)
if mo:
if rx in BuildLogRegexes.GIVENBACK:
success = SBuildExitCodes.GIVENBACK
=== modified file 'lpbuildd/debian.py'
--- lpbuildd/debian.py 2013-09-19 11:01:10 +0000
+++ lpbuildd/debian.py 2013-09-26 09:44:30 +0000
@@ -173,13 +173,17 @@
self._state = DebianBuildState.UPDATE
self.doUpdateChroot()
- def searchLogContents(self, patterns_and_flags):
+ def searchLogContents(self, patterns_and_flags,
+ stop_patterns_and_flags=[]):
"""Search for any of a list of regex patterns in the build log.
The build log is matched using a sliding window, which avoids having
to read the whole file into memory at once but requires that matches
be no longer than the chunk size (currently 256KiB).
+ If any of the stop patterns are matched, the search stops
+ immediately without reading the rest of the file.
+
:return: A tuple of the regex pattern that matched and the match
object, or (None, None).
"""
@@ -187,6 +191,9 @@
regexes = [
re.compile(pattern, flags)
for pattern, flags in patterns_and_flags]
+ stop_regexes = [
+ re.compile(pattern, flags)
+ for pattern, flags in stop_patterns_and_flags]
log = open(os.path.join(self._cachepath, "buildlog"))
try:
window = ""
@@ -197,6 +204,9 @@
match = regex.search(window)
if match is not None:
return regex.pattern, match
+ for regex in stop_regexes:
+ if regex.search(window) is not None:
+ return None, None
if len(window) > chunk_size:
window = window[chunk_size:]
chunk = log.read(chunk_size)
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-reviewers
Post to : [email protected]
Unsubscribe : https://launchpad.net/~launchpad-reviewers
More help : https://help.launchpad.net/ListHelp
