Francis gave a good answer, even if it did not work for him, of how to look at
this interactively.
Unfortunately, in your branch, the yuixhr_fixture tests pass interactively, and
fail within the test suite, so the intended interactive diagnostic path that he
described does not work. I bet it is because run-testapp begins with a
different config than the test machinery does before it switches to the local
appserver-layer config, and the earlier config leaks over somehow; I saw an
instance of leaking earlier, and that might explain the difference. I'll
investigate that later.
All that said, the logs I mentioned namelessly last time
(logs/test-appserver-layer.log) still give a pretty good hint at the problem.
It showed that the error was in fact within the publication machinery as I had
guessed--in endRequest specifically. Here's one example of the same repeated
error that I saw.
2011-10-30T01:10:58 ERROR root Exception during task
Traceback (most recent call last):
File
"/home/gary/launchpad/deps/eggs/zope.server-3.6.1-py2.6.egg/zope/server/taskthreads
.py", line 48, in handlerThread
task.service()
File
"/home/gary/launchpad/deps/eggs/zope.server-3.6.1-py2.6.egg/zope/server/serverchann
elbase.py", line 207, in service
task.service()
File
"/home/gary/launchpad/deps/eggs/zope.server-3.6.1-py2.6.egg/zope/server/http/httpta
sk.py", line 71, in service
self.channel.server.executeRequest(self)
File
"/home/gary/launchpad/deps/eggs/zc.zservertracelog-1.1.5-py2.6.egg/zc/zservertracel
og/tracelog.py", line 114, in executeRequest
response = self.application(env, start_response)
File
"/home/gary/launchpad/deps/eggs/zope.app.wsgi-3.6.0-py2.6.egg/zope/app/wsgi/__init_
_.py", line 58, in __call__
request = publish(request, handle_errors=handle_errors)
File
"/home/gary/launchpad/deps/eggs/zope.publisher-3.12.0-py2.6.egg/zope/publisher/publ
ish.py", line 155, in publish
publication.endRequest(request, obj)
File
"/home/gary/launchpad/lp/workspace/lib/canonical/launchpad/webapp/publication.py",
line 769, in endRequest
sys.exc_info(), request)
File
"/home/gary/launchpad/lp/workspace/lib/canonical/launchpad/webapp/errorlog.py",
lin
e 363, in raising
if self._oops_config.publish(report) is None:
File "/home/gary/launchpad/deps/eggs/oops-0.0.9-py2.6.egg/oops/config.py",
line 184, in
publish
id = publisher(report)
File
"/home/gary/launchpad/deps/eggs/oops_amqp-0.0.3-py2.6.egg/oops_amqp/publisher.py",
line 88, in __call__
channel = self.get_channel()
File
"/home/gary/launchpad/deps/eggs/oops_amqp-0.0.3-py2.6.egg/oops_amqp/publisher.py",
line 65, in get_channel
self.channels.channel = self.connection_factory().channel()
File "/home/gary/launchpad/lp/workspace/lib/lp/services/messaging/rabbit.py",
line 75, i
n connect
raise MessagingUnavailable("Incomplete configuration")
MessagingUnavailable: Incomplete configuration
So...maybe error_queue_key needs to be set somewhere else? Or maybe we need to
do a dance like the one in TestCaptureOopsNoRabbit to make sure that rabbit is
not used in yuixhr? I bet changes to yuixhr might need to be mirrored for the
appserver layer.
I'm not sure what would be better, making sure rabbit is set up correctly, or
making sure that rabbit is not used at all. What do you think?
In any case, you can see where yuixhr sets itself up in runlaunchpad.py's
start_testapp. Please only make surgical changes here--I have three pending
branches with changes to yuixhr test machinery, and I'd prefer not to have to
resolve big conflicts. This section is one of them that has big changes.
I'm going to stop the weekend exploration now. I'm happy to help more later,
though this coming week is jam packed with interviews for me so time is tight
(which is why I took the time to do this exploration now).
(I attempted a quick hack fix for the heck of it, without really looking
closely at your branch. FWIW, this did not help:
=== modified file 'lib/canonical/launchpad/scripts/runlaunchpad.py'
--- lib/canonical/launchpad/scripts/runlaunchpad.py 2011-10-14 02:59:48
+0000
+++ lib/canonical/launchpad/scripts/runlaunchpad.py 2011-10-29 20:23:34
+0000
@@ -355,6 +355,7 @@
installFakeConnect,
uninstallFakeConnect,
)
+ from lp.testing.fixture import CaptureOops
assert config.instance_name.startswith('testrunner-appserver'), (
'%r does not start with "testrunner-appserver"' %
config.instance_name)
@@ -383,9 +384,12 @@
# Interactive tests always need this. We let functional tests use
# a local one too because of simplicity.
LayerProcessController.startSMTPServer()
+ oops_fixture = CaptureOops()
+ oops_fixture.setUp()
try:
start_launchpad(argv, setup)
finally:
+ oops_fixture.tearDown()
LayerProcessController.stopSMTPServer()
LibrarianLayer.tearDown()
DatabaseLayer.tearDown()
)
Gary
On Oct 28, 2011, at 6:04 AM, Robert Collins wrote:
> On Fri, Oct 28, 2011 at 2:05 AM, Gary Poster <gary.pos...@canonical.com>
> wrote:
>> I'm not here--I'm typing this on my phone--but my guess is that your branch
>> messes with the publication machinery in such a way as to break the yuixhr
>> stuff. The "unknown error" suggests something lowlevel like that. My first
>> guess would be that something is not set up that needs to be set up in the
>> associated runlaunchpad variant for yuixhr.
>
> RabbitMQ would be the thing then I guess, but I thought that was on
> for most things now (and if its not, the amqp oops code is meant to
> degrade gracefully - is there some way t get debug output on it?).
> Note that all the other yuixhr tests work ok so I would be surprised
> if yuihr is fundamentally borked.
>
>> I'd be happy to help look at this when I return. That could conceivably be
>> tomorrow but is more likely to be Monday. I'd look at the logs when I ran
>> the tests interactively as my first step. Maybe that would let you tackle it?
>
> What do you mean by that? {Sorry for noob question, but I'm allowed
> one a week :P}
>
> -Rob
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
