Re: [ZODB-Dev] ZEO client hangs when combined with other asyncore code
Tim Peters wrote: ... [Tim Peters] asyncore gives me a headache. [Paul Boots] Same here Then it's time to admit that ZEO's attempts to mix threads with asyncore give me migraine headaches 0.5 wink. I wonder whether this could be the problem: Paul said he's calling ZEO from within the proxy code, but it sounds like the proxy code itself runs as a side effect of asyncore callbacks. If the flow is like this: asyncore mainloop invokes POP3 proxy code POP3 proxy code makes a synchronous ZEO call then I figure the app may well hang then: the thread running the asyncore mainloop is still running a POP3 proxy callback, waiting for a response that can never happen until the asyncore mainloop gets control back (in order to send receive ZEO messages). I think that's exactly how the Proxy runs, we use asynchat and the 'line_terminator' to trigger a callback, so it appears the code runs 'magically' at first glance. I never used asynchat ( ZEO doesn't either), so can't guess whether it's contributing new complications. ZEO's control flow is murky to me too. I _think_ (but may well be wrong) that ZEO expects asyncore to be running in a different thread than the thread(s) application code using ZEO clients is(are) running in. Maybe someone who understands this better than I will jump in with a revelation. ZEO has 2 modes, synchronous and asynchronous. In asynchronous mode, ZEO expects a asyncore main loop to be running in it's own thread. A basic constraint of asyncore (and Twisted) programming is that I/O handlers must perform their tasks very quickly. Generally, they are expected to do short tasks, moving small amounts of data around at a time. Generally, this means that they should not be calling application code directly. This is why Zope executes application logic in separate application threads and limits work done by asyncore handlers to simple I/O. ... IMO/IME, asyncore is a poor fit for applications where the callbacks are fancy, or even where they may just take a long time to complete (because the asyncore mainloop is unresponsive for the duration). So if I had to use asyncore (I've never done so on my own initiative wink), I'd gravitate toward a work-queue model anyway, where threads unfettered by asyncore worries do all the real work-- especially on Windows, which loves to run threads --and where asyncore callbacks do as little as possible. Agreed. This is exactly the model that Zope uses. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO client hangs when combined with other asyncore code
On 6/22/05, Jim Fulton [EMAIL PROTECTED] wrote: Tim Peters wrote: IMO/IME, asyncore is a poor fit for applications where the callbacks are fancy, or even where they may just take a long time to complete (because the asyncore mainloop is unresponsive for the duration). So if I had to use asyncore (I've never done so on my own initiative wink), I'd gravitate toward a work-queue model anyway, where threads unfettered by asyncore worries do all the real work-- especially on Windows, which loves to run threads --and where asyncore callbacks do as little as possible. Agreed. This is exactly the model that Zope uses. ZEO also runs several potentially slow operations in separate threads. I think we've wondered in the past whether the tpc vote should be another of those operations as the disk IO for a large transaction is non-trivial. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] ZEO client hangs when combined with other asyncore code
Tim Peters wrote at 2005-6-21 14:56 -0400:
...
[Dieter Maurer]
If you happen to run your application on Linux (and use the GDB), I
can provide detailed instructions on how to find out where your code
hangs...
That would be helpful!
Also you already solved the main problem, I will share
how I analyse hanging problems with GDB under Linux.
I attach the hanging progess with GDB
gdb python process id
info threads tells me about the process' threads,
thread i allows me to switch into the context
of the various threads. bt shows me the call trace in current
thread. That's all C level information.
To relate that to the Python level, I use a .gdbinit
with the following definitions.
ps (for Print String) outputs the value of a Python
string variable.
pfr (for Print FRame) can be called in eval_frame
C level stack frames and tells filename, function name and line number
of the corresponding Python call.
With these commands I can reconstruct the Python call stack
for the given thread (although it is a bit cumbersome).
Would I know more about how Python stores its interpreter
state per thread, this reconstruction would probably be
even easier...
def ps
x/s ({PyStringObject}$arg0)-ob_sval
end
def pfr
ps f-f_code-co_filename
ps f-f_code-co_name
#p f-f_lineno
lineno
end
define lineno
set $__co = f-f_code
set $__lasti = f-f_lasti
set $__sz = ((PyStringObject *)$__co-co_lnotab)-ob_size/2
set $__p = (unsigned char *)((PyStringObject *)$__co-co_lnotab)-ob_sval
set $__li = $__co-co_firstlineno
set $__ad = 0
while ($__sz-1 = 0)
set $__sz = $__sz - 1
set $__ad = $__ad + *$__p
set $__p = $__p + 1
if ($__ad $__lasti)
# break -- interpreted as breakpoint
set $__sz = -1
end
if ($__sz = 0)
set $__li = $__li + *$__p
set $__p = $__p + 1
end
end
printf %d\n, $__li
end
--
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
