#2395: test_master fails randomly
-------------------+---------------------
Reporter: dustin | Owner: dustin
Type: defect | Status: closed
Priority: major | Milestone: 0.8.8
Version: master | Resolution: fixed
Keywords: |
-------------------+---------------------
Comment (by dustin):
So, to summarize for others who might run into a very twist(y|ed) problem
like this in the future:
The `maybeStartBuild` method returned a Deferred, because it operated as a
chain of asynchronous operations, but none of its callers really cared
about that Deferred. They did the usual `d.addErrback(log.err)`, just to
be safe.
Depending on unpredictable timing issues, sometimes a test would end while
an invocation of this method was still processing. In particular, it was
waiting for a DB operation that was never going to occur, because the DB
pool had been shut down. This evaded the notice of trial's checks for a
dirty reactor, because the frozen state was in a queue for the DB pool,
rather than in an outstanding `defer.callLater` or an open TCP port.
Ordinarily, an unfinished Deferred wouldn't be a problem -- it will
eventually get gc'd and forgotten. However, due to heavy use of
`inlineCallbacks`, this Deferred has a generator attached to it. When a
generator is gc'd, the gc calls `g.close()`, which raises a
`GeneratorExit` exception in the generator. Generators aren't supposed to
catch these, but the two methods in question here -- `_sortBuilders` and
`maybeStartBuildsOn` -- use bare try/except blocks and thus do catch and
log the `GeneratorExit` exception.
The Python gc notices this and prints the warning message about the
generator ignoring `GeneratorExit`. But the `GeneratorExit` was logged as
part of handling it, and trial treats logged errors as `[ERROR]` in
whatever test is currently running.
So, the fix was not to leave those Deferred operations running, but now
that I've written this, I should also change the bare try/except's to use
'except Exception'.
--
Ticket URL: <http://trac.buildbot.net/ticket/2395#comment:10>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Buildbot-commits mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/buildbot-commits
