Re: [Zope] Re: zope fork in external method - mysql connection dropped
Daniel Lopez wrote at 2006-10-16 13:31 -0700: I made one tweak to the double-fork procedure, adding a waitpid call in the grandparent process (the original zope thread) before it returns out of the external method... the code then looked something like: [...prefork code up to here...] pid1 = os.fork() if pid1 0: #grandparent waits for its child before returning os.waitpid(pid1, 0) return RESPONSE.redirect(wait_page) pid2 = os.fork() os.setsid() if pid2 0: #child quits, orphaning grandchild sys.exit(0) [...grandchild-only code after here...] The waitpid call seems to be preventing the zombies... this is good! (though if you find something bad about this approach, please do speak up) But in the process, a new bug was created, having to do with the MySQL connection. I now receive a Lost connection to MySQL server during query error in what appears to be a final db flush from the grandparent's publish function (ZPublisher.Publish, line 104). The good news is that the grandchild continues to do it's work, but the bad news is that the user receives an error page instead of the redirect to the wait_page. Maybe, the MySQL client library installed an atexit hook that closes the database connection. You may try sys._exit(0) (instead of sys.exit(0)) to finish the child. _exit is for purposes when you do not want standard exit handling (such as flushing buffers and calling atexit hooks). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: zope fork in external method - mysql connection dropped
I made one tweak to the double-fork procedure, adding a waitpid call in the grandparent process (the original zope thread) before it returns out of the external method... the code then looked something like: [...prefork code up to here...] pid1 = os.fork() if pid1 0: #grandparent waits for its child before returning os.waitpid(pid1, 0) return RESPONSE.redirect(wait_page) pid2 = os.fork() os.setsid() if pid2 0: #child quits, orphaning grandchild sys.exit(0) [...grandchild-only code after here...] The waitpid call seems to be preventing the zombies... this is good! (though if you find something bad about this approach, please do speak up) But in the process, a new bug was created, having to do with the MySQL connection. I now receive a Lost connection to MySQL server during query error in what appears to be a final db flush from the grandparent's publish function (ZPublisher.Publish, line 104). The good news is that the grandchild continues to do it's work, but the bad news is that the user receives an error page instead of the redirect to the wait_page. My guess is that this has something to do with the various forked processes sharing the MySQL connection, and in particular, I'm guessing the child process that exit(0)'s is closing the connection, which is then unavailable when the grandparent tries to use it. Here's the interesting part: if I put a sleep(1) line in between the waitpid() line and the return line, I get no error. It would appear that the 1 second pause provides enough time for a new MySQL connection to be made after the previous one is killed (less than 1 second is not enough). This is a very ugly fix, though... I can't guarantee that 1 second will always be enough... can anyone suggest a cleaner solution? Perhaps I shouldn't kill the child with sys.exit(0)? -Daniel On Fri, 13 Oct 2006, Jonathan wrote: Pls keep your posts on the list - so that others can help and so that others can search for problems/solutions! - Original Message - Sent: Friday, October 13, 2006 12:13 PM Subject: zope fork in external method Jonathan- I'm emailing you because I saw your post a few months back at: http://mail.zope.org/pipermail/zope/2006-May/166574.html I've been doing my best to find answers on existing posts, but to no avail. Perhaps you can send further pointers? On my system, a user can hit a page that initiates a long database query. If the user hits the stop button and tries to refresh, it messes things up (by interrupting the code upon return of the query), so I came up with a solution that almost works: I put all the code into an external method, which forks. The parent returns a redirect to a waiting throbber page that uses Ajax to poll whether the query is done. The child then goes on to do the hard work. This is working as desired, with a single exception: zombie child processes. Where can I learn more about forking processes in zope external methods? I've read about the double-fork method, but that hasn't resolved the problem... A few ideas for reaping dead child processes: 1) implement a SIGCHLD handler 2) when a spawned child process is finished its 'zope processing' have it write its process id to a file (be careful when accessing files, you will need locking to eliminate problems that may be caused by zope's multiple threads) and then have a clean up routine which kills all of the processes listed in the file (this could be an independant clean-up routine which wakes up on a regular basis, or it could be built into the 'spawning' process which does the clean up before it spawns a child). 3) as the last thing it does, have the child process issue a kill -9 system command to kill its own process (I haven't tried this myself, but it may work and it would be simple to implement) Good luck! Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: zope fork in external method - mysql connection dropped
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Daniel Lopez wrote: I made one tweak to the double-fork procedure, adding a waitpid call in the grandparent process (the original zope thread) before it returns out of the external method... the code then looked something like: [...prefork code up to here...] pid1 = os.fork() if pid1 0: #grandparent waits for its child before returning os.waitpid(pid1, 0) return RESPONSE.redirect(wait_page) pid2 = os.fork() os.setsid() if pid2 0: #child quits, orphaning grandchild sys.exit(0) [...grandchild-only code after here...] The waitpid call seems to be preventing the zombies... this is good! (though if you find something bad about this approach, please do speak up) But in the process, a new bug was created, having to do with the MySQL connection. I now receive a Lost connection to MySQL server during query error in what appears to be a final db flush from the grandparent's publish function (ZPublisher.Publish, line 104). The good news is that the grandchild continues to do it's work, but the bad news is that the user receives an error page instead of the redirect to the wait_page. My guess is that this has something to do with the various forked processes sharing the MySQL connection, and in particular, I'm guessing the child process that exit(0)'s is closing the connection, which is then unavailable when the grandparent tries to use it. Here's the interesting part: if I put a sleep(1) line in between the waitpid() line and the return line, I get no error. It would appear that the 1 second pause provides enough time for a new MySQL connection to be made after the previous one is killed (less than 1 second is not enough). This is a very ugly fix, though... I can't guarantee that 1 second will always be enough... can anyone suggest a cleaner solution? Perhaps I shouldn't kill the child with sys.exit(0)? -Daniel On Fri, 13 Oct 2006, Jonathan wrote: Pls keep your posts on the list - so that others can help and so that others can search for problems/solutions! - Original Message - Sent: Friday, October 13, 2006 12:13 PM Subject: zope fork in external method Jonathan- I'm emailing you because I saw your post a few months back at: http://mail.zope.org/pipermail/zope/2006-May/166574.html I've been doing my best to find answers on existing posts, but to no avail. Perhaps you can send further pointers? On my system, a user can hit a page that initiates a long database query. If the user hits the stop button and tries to refresh, it messes things up (by interrupting the code upon return of the query), so I came up with a solution that almost works: I put all the code into an external method, which forks. The parent returns a redirect to a waiting throbber page that uses Ajax to poll whether the query is done. The child then goes on to do the hard work. This is working as desired, with a single exception: zombie child processes. Where can I learn more about forking processes in zope external methods? I've read about the double-fork method, but that hasn't resolved the problem... A few ideas for reaping dead child processes: 1) implement a SIGCHLD handler 2) when a spawned child process is finished its 'zope processing' have it write its process id to a file (be careful when accessing files, you will need locking to eliminate problems that may be caused by zope's multiple threads) and then have a clean up routine which kills all of the processes listed in the file (this could be an independant clean-up routine which wakes up on a regular basis, or it could be built into the 'spawning' process which does the clean up before it spawns a child). 3) as the last thing it does, have the child process issue a kill -9 system command to kill its own process (I haven't tried this myself, but it may work and it would be simple to implement) I'd recommend using the 'subprocess' module for such stuff. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFM/sM+gerLs4ltQ4RArk4AJ9BjRO44YwZYPu2Mc4wo2PVEN504gCbBIw8 ja7CChvp288LUobko5cZHqY= =izB3 -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: zope fork in external method - mysql connection dropped
You might also try running zope under runzope instead of under zopectl. IIRC zopectl itself sets signal handlers and performs a fork before starting zope, so the inherited environment may not be what you think it is. - C On Oct 16, 2006, at 4:31 PM, Daniel Lopez wrote: I made one tweak to the double-fork procedure, adding a waitpid call in the grandparent process (the original zope thread) before it returns out of the external method... the code then looked something like: [...prefork code up to here...] pid1 = os.fork() if pid1 0: #grandparent waits for its child before returning os.waitpid(pid1, 0) return RESPONSE.redirect(wait_page) pid2 = os.fork() os.setsid() if pid2 0: #child quits, orphaning grandchild sys.exit(0) [...grandchild-only code after here...] The waitpid call seems to be preventing the zombies... this is good! (though if you find something bad about this approach, please do speak up) But in the process, a new bug was created, having to do with the MySQL connection. I now receive a Lost connection to MySQL server during query error in what appears to be a final db flush from the grandparent's publish function (ZPublisher.Publish, line 104). The good news is that the grandchild continues to do it's work, but the bad news is that the user receives an error page instead of the redirect to the wait_page. My guess is that this has something to do with the various forked processes sharing the MySQL connection, and in particular, I'm guessing the child process that exit(0)'s is closing the connection, which is then unavailable when the grandparent tries to use it. Here's the interesting part: if I put a sleep(1) line in between the waitpid() line and the return line, I get no error. It would appear that the 1 second pause provides enough time for a new MySQL connection to be made after the previous one is killed (less than 1 second is not enough). This is a very ugly fix, though... I can't guarantee that 1 second will always be enough... can anyone suggest a cleaner solution? Perhaps I shouldn't kill the child with sys.exit(0)? -Daniel On Fri, 13 Oct 2006, Jonathan wrote: Pls keep your posts on the list - so that others can help and so that others can search for problems/solutions! - Original Message - Sent: Friday, October 13, 2006 12:13 PM Subject: zope fork in external method Jonathan- I'm emailing you because I saw your post a few months back at: http://mail.zope.org/pipermail/zope/2006-May/166574.html I've been doing my best to find answers on existing posts, but to no avail. Perhaps you can send further pointers? On my system, a user can hit a page that initiates a long database query. If the user hits the stop button and tries to refresh, it messes things up (by interrupting the code upon return of the query), so I came up with a solution that almost works: I put all the code into an external method, which forks. The parent returns a redirect to a waiting throbber page that uses Ajax to poll whether the query is done. The child then goes on to do the hard work. This is working as desired, with a single exception: zombie child processes. Where can I learn more about forking processes in zope external methods? I've read about the double-fork method, but that hasn't resolved the problem... A few ideas for reaping dead child processes: 1) implement a SIGCHLD handler 2) when a spawned child process is finished its 'zope processing' have it write its process id to a file (be careful when accessing files, you will need locking to eliminate problems that may be caused by zope's multiple threads) and then have a clean up routine which kills all of the processes listed in the file (this could be an independant clean-up routine which wakes up on a regular basis, or it could be built into the 'spawning' process which does the clean up before it spawns a child). 3) as the last thing it does, have the child process issue a kill -9 system command to kill its own process (I haven't tried this myself, but it may work and it would be simple to implement) Good luck! Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
