Like I said in the original post this is in a multiprocessing.Process() object which uses the subprocess module to create a subprocess and runs the run() function in that subprocess. You can see the documentation here: http://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process
I'm not using threads or the threading module. Thanks for the reminder about multiple sessions. I had forgotten about that. This means I will have to share the same Connection object between multiple processes. Does the Connection object serialize fine when transferring via the pipe? -----Original Message----- From: Bill Freeman [mailto:[email protected]] Sent: Wednesday, August 07, 2013 5:51 PM To: users Subject: Re: Python Connections Hang from Subprocesses By the way, I presume that you know that you can have multiple sessions on one connection? On Wed, Aug 7, 2013 at 5:49 PM, Bill Freeman <[email protected]> wrote: > > > > On Wed, Aug 7, 2013 at 5:35 PM, Eagy, Taylor <[email protected]>wrote: > >> What O/S are you running this on? >> >> >> >> Ubuntu 12.04 LTS >> >> What does the code that spawns the subprocess look like? >> >> >> >> So my Process objects have a run() method that is called as the >> subprocess. >> > > Ok. Unless I miss my guess, this is not a subprocess. This is a thread. > Your object is probably a subclass or threading.Thread, and you never > import subprocess anywhere. If you ran "ps -alxww" in another > terminal, there wouldn't be a line for more than one process. > >> >> >> >> import qpid.messaging # at the top of the Python module >> >> >> >> def run(): >> >> // some other non-qpid setup >> >> >> >> conn = qpid.messaging.Connection(qpid_endpoint) >> >> conn.open() <-- hangs here >> >> self.sess = conn.session() >> >> I have done this when there was already a connection to the broker, >> but > that other connection was a qmf.console connection, which does not use > qpid.messaging under the hood. > > Someone else will have to speak to the thread safety of > qpid.messaging, because we have now exceeded my experience in that area. > > >> >> Can you, even when the subprocess is hung, run something like spout, >> drain, or qpid-config, e.g.; >> >> qpid-config queues >> >> from another console window on the same box? >> >> >> >> Yes, I can connect to the queue if I try to connect from another >> module (which runs a new copy of the Python interpreter). >> >> ________________________________ >> From: Bill Freeman [[email protected]] >> Sent: Wednesday, August 07, 2013 4:21 PM >> To: users >> Subject: Re: Python Connections Hang from Subprocesses >> >> Just in case it makes me (or someone else) think of something else: >> >> What O/S are you running this on? >> >> What does the code that spawns the subprocess look like? >> >> Can you, even when the subprocess is hung, run something like >> spout, drain, or qpid-config, e.g.; >> >> qpid-config queues >> >> from another console window on the same box? >> >> Bill >> >> >> On Wed, Aug 7, 2013 at 3:13 PM, Eagy, Taylor <[email protected] >> >wrote: >> >> > Oops. Let me clean it up a bit. >> > >> > def attach(self): >> > >> > """ >> > Attach to the remote endpoint. >> > """ >> > if not self._connected: >> > self._connected = True >> > self._driver.start() >> > self._wakeup() >> > self._ewait(lambda: self._transport_connected and not >> > self._unlinked()) >> > >> > def _ewait(self, predicate, timeout=None): >> > >> > >> > result = self._wait(lambda: self.error or predicate(), timeout) >> > <-- >> It >> > times out here. >> > self.check_error() >> > return result >> > >> > >> > ________________________________ >> > From: Eagy, Taylor [[email protected]] >> > Sent: Wednesday, August 07, 2013 2:50 PM >> > To: [email protected] >> > Subject: RE: Python Connections Hang from Subprocesses >> > >> > Bill thanks for your response. The subprocess definitely runs when >> > I >> don't >> > try to connect to the broker. Running a HTTPServer in it works too. >> > It >> just >> > doesn't connect to the qpid broker. I did a little digging and I >> > believe this is where it is hanging: >> > >> > (In Connection class) >> > >> > 270 @synchronized >> > >> > 271 -< >> > >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp >> id.messaging.endpoints-pysrc.html# >> > >> > def attach< >> > >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp >> id.messaging.endpoints.Connection-class.html#attach >> > >(self): >> > >> > 272 """ >> > 273 Attach to the remote endpoint. >> > 274 """ >> > 275 if not self._connected: >> > 276 self._connected = True >> > 277 self._driver.start< >> > >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp >> id.messaging.endpoints-pysrc.html# >> > >() >> > 278 self._wakeup() >> > 279 self._ewait(lambda: self._transport_connected and not >> > self._unlinked()) >> > >> > The parent Python process can connect successfully, but the >> > self._transport_connected never gets set to True for all of the new >> > Connection objects created in the subprocesses that are trying to >> connect >> > to the same broker. Where does this get set to True? >> > Could this have something to do with the predicate because it >> > doesn't return an error it just times out? >> > >> > >> > 212 -< >> > >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp >> id.messaging.endpoints-pysrc.html# >> > >> > def _ewait< >> > >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp >> id.messaging.endpoints.Connection-class.html#_ewait >> >(self, >> > predicate, timeout=None): >> > >> > 213 result = self._wait(lambda: self.error or predicate(), >> timeout< >> > >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp >> id.messaging.endpoints-pysrc.html# >> > >) >> > 214 self.check_error< >> > >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp >> id.messaging.endpoints-pysrc.html# >> > >() >> > 215 return result >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > Thanks, >> > Taylor >> > ________________________________ >> > From: Bill Freeman [[email protected]] >> > Sent: Wednesday, August 07, 2013 2:00 PM >> > To: users >> > Subject: Re: Python Connections Hang from Subprocesses >> > >> > Subprocesses (using the subprocess module, or even the older exec >> stuff, as >> > opposed to threads, or even forked clones) are relatively trouble >> > free >> in >> > python (except maybe on Windows, whose process model has that >> > Microsoft difference). I've certainly made multiple connections to >> > a broker from >> one >> > python process, as well as using tools like spout and drain, which >> > are >> both >> > written in python, while my main development project is running >> connected. >> > >> > Maybe there are broker configuration items which can affect this. >> > If >> so, I >> > hope that someone knowledgeable will speak up. But I doubt that >> > this is the problem. >> > >> > Are you sure that your subprocess runs? It might be trying to >> > report an error to you. Of, if you have pipes configured for >> > interaction with the invoking processor, it might be waiting on one of >> > those. >> > >> > You could, temporarily, instead of your intended code, have the >> subprocess >> > invoke something like BasicHTTPServer, and see if you can interact >> > with >> it >> > using your browser. If that also fails, it leaves the broker >> > connection out as the source of your problems. >> > >> > Possibly easier is to have it log its arrival at various points, so >> > you >> can >> > be sure where it is getting stuck. If you've already confirmed >> > that >> it's >> > in the broker connect, forgive me, and wait for a better answer. >> > >> > Bill >> > >> > >> > On Wed, Aug 7, 2013 at 12:55 PM, Eagy, Taylor >> > <[email protected] >> > >wrote: >> > >> > > Hello, >> > > >> > > >> > > >> > > I'm having an issue connecting multiple Python subprocesses to >> > > the >> qpid >> > > C++ broker. I have a main Python process that imports >> > > C++ qpid.messaging >> at >> > the >> > > top and spawns a few multiprocessing.Process objects that try to >> create a >> > > Qpid connection to the broker. Before I spawn the processes >> > > though, >> the >> > > main process creates a connection to the Qpid broker just fine, >> > > but >> all >> > of >> > > the subprocess code hangs when it trys to connect to the qpid broker. >> I >> > > know qpid is supposed to be multi-threaded, but what about >> multiprocess >> > > with Python? What could be causing the connections to hang from >> > > the subprocesses? Is there some singleton object that isn't >> > > getting >> released >> > > that's blocking the other subprocesses from connecting? >> > > >> > > >> > > >> > > Thanks, >> > > >> > > TJ >> > > >> > >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
