There is one issue with multiprocessing and database connections: Since you're using a new process, any existing database connections will not carry over to the new process, and you must re-establish your connection to the database.
To try to explain a little bit better: Assume your main process is process id 100. When you do the "proc=multip.Process(...)" call, you create a new process with id 101. Those processes have to have their own connection to the database. Furthermore, since they are all their own individual processes, when they finish, they *close* their connection to the database as part of Python's process exit/cleanup. The end result is that the parent process, even if the child processes worked just fine, would find itself unable to use the parent database. In the end, for each child process, get it the information to connect to the database fresh, and work only with a fresh connection that is private to that child process. You'll get the results you're expecting then. On Sat, Sep 17, 2011 at 2:23 AM, Sam <[email protected]> wrote: > Diez was nice enough to give me some advice 17 months ago: > > http://groups.google.com/group/turbogears/browse_thread/thread/85214fab89ff39cb/46c066cd3d535337?lnk=gst&q=multiprocessing#46c066cd3d535337 > > Unfortunately, other priorities came up, but I finally implemented his > advice. > > Multiprocessing was pretty easy to get working - I do: > > queue = multip.Queue() > proc = multip.Process(target=get_extracts, args=(my_pages, company, > store_num, queue)) > proc.start() > bottles = queue.get() > proc.join(30) > > and that part works! > > But immediately after the join, when I go to use the database again... > > I get an error: > > Traceback (most recent call last): > File "/home/sam/dev_branch/ve/lib/python2.6/site-packages/cherrypy/ > _cprequest.py", line 659, in respond > self.hooks.run('on_end_resource') > File "/home/sam/dev_branch/ve/lib/python2.6/site-packages/cherrypy/ > _cprequest.py", line 97, in run > hook() > File "/home/sam/dev_branch/ve/lib/python2.6/site-packages/cherrypy/ > _cprequest.py", line 57, in __call__ > return self.callback(**self.kwargs) > File "/home/sam/dev_branch/ve/lib/python2.6/site-packages/turbogears/ > database.py", line 602, in EndTransactions > end_all() > File "/home/sam/dev_branch/ve/lib/python2.6/site-packages/turbogears/ > database.py", line 419, in end_all > hub.end() > File "/home/sam/dev_branch/ve/lib/python2.6/site-packages/turbogears/ > database.py", line 322, in end > conn.rollback() > File "/home/sam/dev_branch/ve/lib/python2.6/site-packages/sqlobject/ > dbconnection.py", line 754, in rollback > self._connection.rollback() > InterfaceError: connection already closed > > Now in the target function get_extracts I had to open a connection to > the database, of course. I tried opening a connection to the database > immediately after the join, but that didn't help any. > > Any thoughts? > > I'm using TG 1.5, with SQLObject and Postgres running under Linux. > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "TurboGears" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/turbogears?hl=en. > > -- Michael J. Pedersen My IM IDs: Jabber/[email protected], AIM/pedermj022171 Yahoo/pedermj2002, MSN/[email protected] My Online Resume: http://www.icelus.org/ Twitter: pedersentg -- You received this message because you are subscribed to the Google Groups "TurboGears" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.
