Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Op dinsdag 11-12-2007 om 17:03 uur [tijdzone -0800], schreef Guido van Rossum: On Dec 11, 2007 4:54 PM, Jan Claeys [EMAIL PROTECTED] wrote: Op vrijdag 07-12-2007 om 07:26 uur [tijdzone -0700], schreef Sean Reifschneider: I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... Do you have data to support this claim? Not PyGTK-specific data, but you can see power-saving by reducing wakeups by playing with Intel's 'powertop' utility on linux (also see the lesswatts site mentioned earlier). One example was the interaction between PulseAudio ALSA that caused a lot of wakeups, and thus prevented laptops from going into the more agressive power saving modes; if PyGTK has a similar issue, that would make it less useful on laptops... -- Jan Claeys ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 14, 2007 9:38 AM, Isaac Morland [EMAIL PROTECTED] wrote: I think we successfully resolved this. Adam Olsen will produce a patch that allows one to specify a single file descriptor to which a zero byte will be written by the C-level signal handler. Twisted and PyGTK I'm surprised that the byte is zero, rather than being the signal number. This would reflect the way C-level signal handlers work, where they are I believe called with the signal number. Is any discussion of this point conveniently available online? I ask for my own understanding, and definitely not out of a desire to re-open any settled issues! Yes, this was discussed previously in this thread. Go find it on mail.python.org/pipermail/python-dev/ -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Mon, 10 Dec 2007, Guido van Rossum wrote: I think we successfully resolved this. Adam Olsen will produce a patch that allows one to specify a single file descriptor to which a zero byte will be written by the C-level signal handler. Twisted and PyGTK I'm surprised that the byte is zero, rather than being the signal number. This would reflect the way C-level signal handlers work, where they are I believe called with the signal number. Is any discussion of this point conveniently available online? I ask for my own understanding, and definitely not out of a desire to re-open any settled issues! Isaac Morland CSCF Web Guru DC 2554C, x36650WWW Software Specialist ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Sat, Dec 08, 2007, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. Ayup. I am extremely far from an expert here, but anyone wanting to have an informed opinion should really re-read the threads after these posts: http://mail.python.org/pipermail/python-dev/2006-September/068569.html http://mail.python.org/pipermail/python-dev/2007-January/070772.html I would recommend requesting review from either Nick Maclaren or Tim Peters as well. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ Typing is cheap. Thinking is expensive. --Roy Smith ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 14, 2007 2:12 PM, Aahz [EMAIL PROTECTED] wrote: On Sat, Dec 08, 2007, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. Ayup. I am extremely far from an expert here, but anyone wanting to have an informed opinion should really re-read the threads after these posts: http://mail.python.org/pipermail/python-dev/2006-September/068569.html http://mail.python.org/pipermail/python-dev/2007-January/070772.html I would recommend requesting review from either Nick Maclaren or Tim Peters as well. Oh no, not Nick Maclaren! Anyway, did you see that we resolved this and have a patch ready for review? http://bugs.python.org/issue1583 -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
[private] On Fri, Dec 14, 2007, Guido van Rossum wrote: On Dec 14, 2007 2:12 PM, Aahz [EMAIL PROTECTED] wrote: On Sat, Dec 08, 2007, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. Ayup. I am extremely far from an expert here, but anyone wanting to have an informed opinion should really re-read the threads after these posts: http://mail.python.org/pipermail/python-dev/2006-September/068569.html http://mail.python.org/pipermail/python-dev/2007-January/070772.html I would recommend requesting review from either Nick Maclaren or Tim Peters as well. Oh no, not Nick Maclaren! ;-) Note carefully that I did not say you should necessarily follow Nick's advice. Still, having an expert tell you what you're doing wrong usually helps even if their advice isn't relevant. Anyway, did you see that we resolved this and have a patch ready for review? http://bugs.python.org/issue1583 Yeah, I saw that after the fact, but I still think referring to the previous threads should be useful for anyone wanting to review the patch. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ Typing is cheap. Thinking is expensive. --Roy Smith ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 12/12/07, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 12, 2007 2:42 PM, Greg Ewing [EMAIL PROTECTED] wrote: But there's no excuse for using CPU when the application truly isn't doing anything other than waiting for something to happen. There are tons of situations where polling is quite reasonable as logn as it involves a sleep. Now that I have to disagree with, possibly because sleep is ambiguous as stated. Periodic time based polling means your APIs are broken (not that one often has control over what APIs are available). Blocking only to be woken up when any of the events your interested in is always best. If you have periodic tasks that must be performed then obviously an event you're interested in is time T or X time has passed but that is distinct from a process waking up regularly to check the empty work queue length only to sleep again. Regardless this thread already resolved the issue with an acceptable solution (yay!) so further discussion is merely a bike shed. -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: Hm, Apple and Microsoft don't seem to think so. They go out of their way to implement elaborate visual effects. Well, that's a different issue -- assuming the visual effect is something wanted, then the CPU required to produce it isn't unnecessary. But there's no excuse for using CPU when the application truly isn't doing anything other than waiting for something to happen. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 12, 2007 2:42 PM, Greg Ewing [EMAIL PROTECTED] wrote: But there's no excuse for using CPU when the application truly isn't doing anything other than waiting for something to happen. There are tons of situations where polling is quite reasonable as logn as it involves a sleep. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Op vrijdag 07-12-2007 om 07:26 uur [tijdzone -0700], schreef Sean Reifschneider: I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... -- Jan Claeys ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 11, 2007 4:54 PM, Jan Claeys [EMAIL PROTECTED] wrote: Op vrijdag 07-12-2007 om 07:26 uur [tijdzone -0700], schreef Sean Reifschneider: I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... Do you have data to support this claim? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: On Dec 11, 2007 4:54 PM, Jan Claeys [EMAIL PROTECTED] wrote: Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... Do you have data to support this claim? Even if it doesn't save any power, using CPU unnecessarily is a bad thing for any application to do on a multitasking system. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 11, 2007 6:01 PM, Greg Ewing [EMAIL PROTECTED] wrote: Guido van Rossum wrote: On Dec 11, 2007 4:54 PM, Jan Claeys [EMAIL PROTECTED] wrote: Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... Do you have data to support this claim? Even if it doesn't save any power, using CPU unnecessarily is a bad thing for any application to do on a multitasking system. Hm, Apple and Microsoft don't seem to think so. They go out of their way to implement elaborate visual effects. Again -- is there any data about the cost of PyGTK's waking up 10x/sec on a typical laptop or server? (The XO is a special case because it has very different power management abilities.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: On Dec 11, 2007 4:54 PM, Jan Claeys [EMAIL PROTECTED] wrote: Op vrijdag 07-12-2007 om 07:26 uur [tijdzone -0700], schreef Sean Reifschneider: I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... Do you have data to support this claim? http://www.lesswatts.org/projects/powertop/powertop.php Some quotes plucked from that page: “In the screenshot, the laptop isn't doing very well. Most of the time the processor is in C2, and then only for an average of 4.4 milliseconds at a time. If the laptop spent most of its time in C4 for at least 20 milliseconds, the battery life would have been approximately one hour longer.” “When running a full GNOME desktop, 3 wakeups per second is achievable.” There's considerable effort being invested in the GNOME and Linux software stack at the moment to get rid of unnecessary CPU wakeups, and people are reporting significant improvements in laptop power consumption as a result of that work. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 11, 2007 11:00 PM, Andrew Bennetts [EMAIL PROTECTED] wrote: Guido van Rossum wrote: On Dec 11, 2007 4:54 PM, Jan Claeys [EMAIL PROTECTED] wrote: Op vrijdag 07-12-2007 om 07:26 uur [tijdzone -0700], schreef Sean Reifschneider: I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... Do you have data to support this claim? http://www.lesswatts.org/projects/powertop/powertop.php Some quotes plucked from that page: In the screenshot, the laptop isn't doing very well. Most of the time the processor is in C2, and then only for an average of 4.4 milliseconds at a time. If the laptop spent most of its time in C4 for at least 20 milliseconds, the battery life would have been approximately one hour longer. When running a full GNOME desktop, 3 wakeups per second is achievable. There's considerable effort being invested in the GNOME and Linux software stack at the moment to get rid of unnecessary CPU wakeups, and people are reporting significant improvements in laptop power consumption as a result of that work. There's a known issues page on there, on the bottom of which is sealert, which used python, gtk, and threads. It has since been rewritten to not use threads, but it did exhibit the problem set_wakeup_fd fixes (at least provides our half of the fix.) https://bugzilla.redhat.com/show_bug.cgi?id=239893 -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 9, 2007 10:16 AM, [EMAIL PROTECTED] wrote: On 12:21 am, [EMAIL PROTECTED] wrote: Anyway, I would still like to discuss this on #python-dev Monday. Adam, in what time zone are you? (I'm PST.) Who else is interested? I'm also interested. I'm EST, but my schedule is very flexible (I'm on IRC pretty much all day for work anyway). Just let me know when it is. Adam I are now on #python-dev. Can you join? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Adam I are now on #python-dev. Can you join? I think we successfully resolved this. Adam Olsen will produce a patch that allows one to specify a single file descriptor to which a zero byte will be written by the C-level signal handler. Twisted and PyGTK will have to coordinate about this file descriptor. Glyph believes this is possible and sufficient. (A preliminary version of the patch may be found here: http://dpaste.com/27576/ ) We considered two alternatives: (a) A patch by myself where the file descriptor would instead be passed together with a signal handler. This was eventually rejected because it places an extra burden on every piece of code that registers a signal handler. (b) A more elaborate patch by Adam which would allow many file descriptors to be registered. This was rejected for being more code and solving a problem that most likely doesn't exist (multiple independent main loops running in different threads). We also located the exact source of the 100 msec timeout in PyGTK: http://svn.gnome.org/viewvc/pygtk/trunk/gtk/gtk.override?annotate=2926 line 1075: *timeout = 100; The recommendation for the OLPC XO project is to remove this line or make the timeout much larger, as the only reason why this was even added to PyGTK is wanting a fast response to ^C from the console, which doesn't represent a viable use case on the XO. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 10, 2007 4:26 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Adam I are now on #python-dev. Can you join? I think we successfully resolved this. Adam Olsen will produce a patch that allows one to specify a single file descriptor to which a zero byte will be written by the C-level signal handler. Twisted and PyGTK will have to coordinate about this file descriptor. Glyph believes this is possible and sufficient. http://bugs.python.org/issue1583 (A preliminary version of the patch may be found here: http://dpaste.com/27576/ ) We considered two alternatives: (a) A patch by myself where the file descriptor would instead be passed together with a signal handler. This was eventually rejected because it places an extra burden on every piece of code that registers a signal handler. (b) A more elaborate patch by Adam which would allow many file descriptors to be registered. This was rejected for being more code and solving a problem that most likely doesn't exist (multiple independent main loops running in different threads). We also located the exact source of the 100 msec timeout in PyGTK: http://svn.gnome.org/viewvc/pygtk/trunk/gtk/gtk.override?annotate=2926 line 1075: *timeout = 100; The recommendation for the OLPC XO project is to remove this line or make the timeout much larger, as the only reason why this was even added to PyGTK is wanting a fast response to ^C from the console, which doesn't represent a viable use case on the XO. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/rhamph%40gmail.com -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 10/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: Adam I are now on #python-dev. Can you join? I think we successfully resolved this. Adam Olsen will produce a patch that allows one to specify a single file descriptor to which a zero byte will be written by the C-level signal handler. Twisted and PyGTK will have to coordinate about this file descriptor. Glyph believes this is possible and sufficient. Great that a solution was found at last. But to be honest I don't understand why my patch (the second one) is not good. Why is it preferable for libraries to provide their own file descriptor? Are people concerned about the overhead of always creating a pipe even it may not be used? Always creating a pipe would at least avoid the need for any coordination between PyGTK and Twisted. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 10, 2007 4:07 PM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 10/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: Adam I are now on #python-dev. Can you join? I think we successfully resolved this. Adam Olsen will produce a patch that allows one to specify a single file descriptor to which a zero byte will be written by the C-level signal handler. Twisted and PyGTK will have to coordinate about this file descriptor. Glyph believes this is possible and sufficient. Great that a solution was found at last. But to be honest I don't understand why my patch (the second one) is not good. Why is it preferable for libraries to provide their own file descriptor? So the app is in charge of creating the pipe if it wants it. We expect that if the pipe is always created, apps that assume only fds 0, 1 and 2 are open may get confused or accidentally close it, etc. Are people concerned about the overhead of always creating a pipe even it may not be used? Always creating a pipe would at least avoid the need for any coordination between PyGTK and Twisted. I don't think that anyone involved though the need for coordination was an issue. Please let this rest. We have a solution. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 11/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 10, 2007 4:07 PM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 10/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: Adam I are now on #python-dev. Can you join? I think we successfully resolved this. Adam Olsen will produce a patch that allows one to specify a single file descriptor to which a zero byte will be written by the C-level signal handler. Twisted and PyGTK will have to coordinate about this file descriptor. Glyph believes this is possible and sufficient. Great that a solution was found at last. But to be honest I don't understand why my patch (the second one) is not good. Why is it preferable for libraries to provide their own file descriptor? So the app is in charge of creating the pipe if it wants it. We expect that if the pipe is always created, apps that assume only fds 0, 1 and 2 are open may get confused or accidentally close it, etc. It's the first time I hear about this problem, but sounds plausible. Are people concerned about the overhead of always creating a pipe even it may not be used? Always creating a pipe would at least avoid the need for any coordination between PyGTK and Twisted. I don't think that anyone involved though the need for coordination was an issue. Yeah. Thinking again on this, I don't think this is a problem, not because coordination between PyGTK and Twisted can be done, but because no such coordination is needed. PyGTK can have one FD, Twisted another FD. Since only one of the main loops can be running, there's no conflict. Please let this rest. We have a solution. Sure. :-) -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 12:21 am, [EMAIL PROTECTED] wrote: Anyway, I would still like to discuss this on #python-dev Monday. Adam, in what time zone are you? (I'm PST.) Who else is interested? I'm also interested. I'm EST, but my schedule is very flexible (I'm on IRC pretty much all day for work anyway). Just let me know when it is. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Johan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. --Guido On Dec 7, 2007 3:27 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 7, 2007 2:35 PM, [EMAIL PROTECTED] wrote: On 02:48 pm, [EMAIL PROTECTED] wrote: Not only that, but current python signal handling is not theorethically async safe; there are race conditions in the Py_AddPendingCalls API, and it just happens to work most of the time. [This refers to the internal datastructures used by Py_AddPendingCalls, which aren't updated in a safe way. Hard/impossible to fix in C, but fairly easy with embedded assembly.] Twisted has encountered one such issue, described here: http://twistedmatrix.com/trac/ticket/1997#comment:12 [This refers to the overall design, which is inherently racey.] Unfortunately, I don't know enough about signals to suggest or comment on the solution. Any Python/C wrapper around a syscall which can be interrupted needs to somehow atomically check for the presence of pending python signal handlers; I don't know of any POSIX API to do that. Overall, what you'd need to do is register a wakeup function (to be called by a signal handler or another thread), and have that wakeup function cancel whatever you're doing. The hard part is it needs to work at *ANY* time while it's registered, before you've even called the library function or syscall you intend to cancel! I currently know of two methods of achieving this: 1) If reading a file or socket, first poll the fd, then do a non-blocking read. The wakeup function writes to a wakeup pipe you also poll, which then wakes you up. A wakeup after poll completes is ignored, but the non-blocking read will finish promptly enough anyway. 2) Use sigsetjmp before a syscall (I wouldn't trust a library call), then have the signal handler jump completely out of the operation. This is evil and unportable, but probably works. Additionally, this only gets SIGINT with the default behaviour to work right, as it can be entirely implemented in C. If you want to handle arbitrary signals running arbitrary python code you really need a second thread to run them in. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Not only that, but pygtk is a generic module; who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 9:20 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Sorry, I misattributed this; Gustavo Carneiro pointed this out earlier in this thread. BTW, in the referenced post (also by Gustavo), I found this: According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 Unfortunately I can't read [1] without having registered, so I can only guess what it says. But I know that Python already ensures that signals are only delivered to the main thread. What am I missing? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
At 11:17 AM +0100 12/8/07, Johan Dahlin wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Is that always required (with threads), or are things better now that Ctrl-C handling is improved (at least in the Socket module, which doesn't lose signals anymore)? -- TonyN.:' mailto:[EMAIL PROTECTED] ' http://www.georgeanelson.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? Well, actually this is also a PyGObject issue, not only PyGtk. PyGObject wraps GLib. GLib was created to serve the needs of Gtk+, but is useful by itself for writing portable programs. Among other things, GLib offers a generic main loop, which programs can use to do generic event-driven programming, such as timeouts, polling file descriptors, and doing other work when the main loop becomes idle. You could argue that non-gtk programs are rare and we shouldn't worry too much about them. Maybe it's true, I don't know. who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Funny that everyone mentions this solution, as it is the solution implemented by my patch :-) Well, to be fair, it might not be _exactly_ what is implemented by the patch. Reading between the lines, I think what you mean is to have python's C signal handler mostly untouched; it would only write a byte to a pipe _in addition to_ the normal setting the flag and Py_AddPendingCall. The patch I submitted, on the other hand, completely removes the vector of flags and Py_AddPendingCall, and instead writes the number of the signal that was raised into the pipe, and reads it back from the pipe in the Python main loop. Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). My solution is being reject because people are afraid to touch the signal handling code, which has its faults, but well know faults. But if I refactor the patch to keep the crappy-but-sort-of-working signal code, and only enhance it with the pipe, maybe it will more likely get accepted. Do I understand correctly? :-) -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 9:20 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Sorry, I misattributed this; Gustavo Carneiro pointed this out earlier in this thread. BTW, in the referenced post (also by Gustavo), I found this: According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 Unfortunately I can't read [1] without having registered, so I can only guess what it says. I can't read [1] either because the URL is not correct. Sorry :P But I know that Python already ensures that signals are only delivered to the main thread. Are you sure? In python code I see pthread_sigmask being checked for in configure.in, but I can't find it being used anywhere (unless I'm grepping for the wrong thing). What you probably meant to say was python ensures that signals are only processed from the main thread. Except when python uses GNU PTH threads; there I do see signals being delivered to the main thread. But GNU pth are not real kernel threads, anyway. What am I missing? I think Python could make sure (via pthread_sigmask) that signals are blocked for all non-main threads created by Python. Unfortunately it cannot do the same for threads not created by Python. I know at least one case where threads are created behing Python's back: the GnomeVFS library. Best regards, -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 Unfortunately I can't read [1] without having registered, so I can only guess what it says. But I know that Python already ensures that signals are only delivered to the main thread. What am I missing? Your notion of delivers differs. Python does receive signals in threads other than the main thread. However, it will only invoke the *Python* signal handler in the main thread. signalmodule.c:signal_handler calls Py_AddPendingCall, which is only ever considered in the main thread (although perhaps not in a timely manner - this is precisely where the busy-waiting in gtk comes from). Python does not attempt to block any signals, let alone using pthread_sigmask. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 9:55 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: On Dec 8, 2007 2:17 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Hm. How about making that an option? I don't think on the OLPC XO this is a valid use case (end users never have a console where they might enter ^C). It could easily be made into a compilation option which would solve the problem specifically for OLPC, but it would still be problematic for other platforms important to PyGTK (linux/gnome) where console based development is more common. But do those other platforms care about the extra CPU cycles and power used? I suspect not, at least not to the extent that OLPC cares. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 9:57 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? Well, actually this is also a PyGObject issue, not only PyGtk. PyGObject wraps GLib. GLib was created to serve the needs of Gtk+, but is useful by itself for writing portable programs. Among other things, GLib offers a generic main loop, which programs can use to do generic event-driven programming, such as timeouts, polling file descriptors, and doing other work when the main loop becomes idle. You could argue that non-gtk programs are rare and we shouldn't worry too much about them. Maybe it's true, I don't know. who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Funny that everyone mentions this solution, as it is the solution implemented by my patch :-) Mind pointing me to it again? I can't find it in the Python bug tracker. Well, to be fair, it might not be _exactly_ what is implemented by the patch. Reading between the lines, I think what you mean is to have python's C signal handler mostly untouched; it would only write a byte to a pipe _in addition to_ the normal setting the flag and Py_AddPendingCall. Well, in many cases I see no problems with the current signal handler, and people are used to it, so I'm not sure what is gained by getting rid of it. The patch I submitted, on the other hand, completely removes the vector of flags and Py_AddPendingCall, and instead writes the number of the signal that was raised into the pipe, and reads it back from the pipe in the Python main loop. I believe Py_AddPendingCall was meant to be used by other places besides the signal handling. Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). I think it's okay to drop signals if too many come. The FD should be put in non-blocking mode so the signal handler won't block forever. Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? My solution is being reject because people are afraid to touch the signal handling code, which has its faults, but well know faults. But if I refactor the patch to keep the crappy-but-sort-of-working signal code, and only enhance it with the pipe, maybe it will more likely get accepted. Do I understand correctly? :-) Not really; I don't recall seeing your patch. And I object to your subjective description of the existing signal handling; it has served us well enough. I
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 9:57 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? Well, actually this is also a PyGObject issue, not only PyGtk. PyGObject wraps GLib. GLib was created to serve the needs of Gtk+, but is useful by itself for writing portable programs. Among other things, GLib offers a generic main loop, which programs can use to do generic event-driven programming, such as timeouts, polling file descriptors, and doing other work when the main loop becomes idle. You could argue that non-gtk programs are rare and we shouldn't worry too much about them. Maybe it's true, I don't know. who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Funny that everyone mentions this solution, as it is the solution implemented by my patch :-) Mind pointing me to it again? I can't find it in the Python bug tracker. Well, to be fair, it might not be _exactly_ what is implemented by the patch. Reading between the lines, I think what you mean is to have python's C signal handler mostly untouched; it would only write a byte to a pipe _in addition to_ the normal setting the flag and Py_AddPendingCall. Well, in many cases I see no problems with the current signal handler, and people are used to it, so I'm not sure what is gained by getting rid of it. The patch I submitted, on the other hand, completely removes the vector of flags and Py_AddPendingCall, and instead writes the number of the signal that was raised into the pipe, and reads it back from the pipe in the Python main loop. I believe Py_AddPendingCall was meant to be used by other places besides the signal handling. True. The patch does not remove Py_AddPendingCall, only stops using it for signals. Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). I think it's okay to drop signals if too many come. The FD should be put in non-blocking mode so the signal handler won't block forever. Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? Good point. My solution is being reject because people are afraid to touch the signal handling code, which has its faults, but well know faults. But if I refactor the patch to keep the crappy-but-sort-of-working signal code, and only
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 1:38 PM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 9:57 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). I think it's okay to drop signals if too many come. The FD should be put in non-blocking mode so the signal handler won't block forever. Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? Good point. Note that we may drop a new signal, not the same one we got several times. I don't know if Unix will do that. Then again, I've been unable to find documentation promising it'd deliver any signal at all. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: On Dec 8, 2007 2:17 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Hm. How about making that an option? I don't think on the OLPC XO this is a valid use case (end users never have a console where they might enter ^C). It could easily be made into a compilation option which would solve the problem specifically for OLPC, but it would still be problematic for other platforms important to PyGTK (linux/gnome) where console based development is more common. Johan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: On Dec 8, 2007 9:55 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: On Dec 8, 2007 2:17 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Hm. How about making that an option? I don't think on the OLPC XO this is a valid use case (end users never have a console where they might enter ^C). It could easily be made into a compilation option which would solve the problem specifically for OLPC, but it would still be problematic for other platforms important to PyGTK (linux/gnome) where console based development is more common. But do those other platforms care about the extra CPU cycles and power used? I suspect not, at least not to the extent that OLPC cares. They most certainly do, pygtk applications are used in desktop software used on laptops where battery time is increasingly important. Johan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? No, it doesn't. However, this doesn't necessarily mean that you can just drop them when the pipe becomes full. If more than one type of signal is sharing a pipe, you don't want it getting filled up with signals of one type and causing signals of a different type to get lost. To guarantee that won't happen, you would have to either use a separate pipe for each signal type, or have some other scheme such as a bitmask keeping track of which signal types are in the pipe. I have a worry though -- if signal handlers *always* and *only* communicate by writing a byte to a FD, does that mean that the VM main loop will have to attempt to read a byte from a pipe every time it checks for signals? That sounds like a good reason for having the fd be an additional mechanism that you can turn on when you want it, but not the core mechanism for dealing with signals. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 3:57 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) Hm... Does this mean you're *always* creating an extra thread to handle signals? I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html Since OLPC has to use 2.5 they don't really have another choice besides this or making the timeout (perhaps much) larger -- I'm not going to risk a change as big as anything proposed here for 2.5.2, so nothing will change before 2.6. I've got to say that all the cross-referencing and asynchronous discussion here makes it *very* difficult to wrap my head around the various proposals. It also doesn't help that different participants appear to have different use cases in mind. E.g. do we care about threads started directly from C++ code? (These happen all the time at Google, but we don't care much about signals.) And what about restarting system calls (like Glyph brought up)? I've seen references to bug #1643738 which got a thumbs up from Tim Peters -- Adam, what do you think of that? I know it doesn't address Gustavo's issue but it seems useful in its own right. Gustavo, at some point you suggested making changes to Python so that all signals are blocked in all threads except for the main thread. I think I'd be more inclined to give
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 5:21 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:57 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) Hm... Does this mean you're *always* creating an extra thread to handle signals? Yup, Py_Initialize will do it. I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html Since OLPC has to use 2.5 they don't really have another choice besides this or making the timeout (perhaps much) larger -- I'm not going to risk a change as big as anything proposed here for 2.5.2, so nothing will change before 2.6. I've got to say that all the cross-referencing and asynchronous discussion here makes it *very* difficult to wrap my head around the various proposals. It also doesn't help that different participants appear to have different use cases in mind. E.g. do we care about threads started directly from C++ code? (These happen all the time at Google, but we don't care much about signals.) And what about restarting system calls (like Glyph brought up)? I've seen references to bug #1643738 which got a thumbs up from Tim Peters -- Adam, what do you think of that? I know it doesn't address Gustavo's issue
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 09/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:57 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) Hm... Does this mean you're *always* creating an extra thread to handle signals? I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html Since OLPC has to use 2.5 they don't really have another choice besides this or making the timeout (perhaps much) larger -- I'm not going to risk a change as big as anything proposed here for 2.5.2, so nothing will change before 2.6. I've got to say that all the cross-referencing and asynchronous discussion here makes it *very* difficult to wrap my head around the various proposals. It also doesn't help that different participants appear to have different use cases in mind. E.g. do we care about threads started directly from C++ code? (These happen all the time at Google, but we don't care much about signals.) And what about restarting system calls (like Glyph brought up)? I've seen references to bug #1643738 which got a thumbs up from Tim Peters -- Adam, what do you think of that? I know it doesn't address Gustavo's issue but it seems useful in its own right. That issue
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 6:30 PM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 09/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: Gustavo, at some point you suggested making changes to Python so that all signals are blocked in all threads except for the main thread. I think I'd be more inclined to give that the green light than the patch using pipes for all signal handling, as long as we can make sure that this blocking of all signals isn't inherited by fork()'ed children -- we had serious problems with that in 2.4 where child processes were unkillable (except for SIGKILL). I don't think that solution works after all. We can only block signals for certain threads inside the threads themselves. But we do not control all threads. Some are created by C libraries, and these threads will not have signals blocked by default, and also there is no 'thread creation hook' that we can use. Note that new threads inherit signal masks from their creator. It's only threads created before loading python that are a problem. For my threading patch I plan to document that as simply something embedders have to do. I'd also be OK with a patch that leaves the existing signal handling code intact but *adds* a way to have a signal handler written in C that writes one byte to one end of a pipe -- where the pipe is provided by Python code. I think this is most balanced approach of all. Yeah, use the existing Handlers array to record which signals have come in, rather using the byte passed through the pipe. And I missed a problem in bug #1643738: Handlers[...].tripped should be a sig_atomic_t, not int. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 5:30 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 5:21 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Hm... Does this mean you're *always* creating an extra thread to handle signals? Yup, Py_Initialize will do it. That's unacceptable. It must be possible to build Python without threads (and still support signals -- in fact one could argue that signals make *more* sense when there are no threads :-). [...] To summarize, there's two problems to be solved: 1) low-level corruption in the signal handlers as they record a new signal, such as in Py_AddPendingCalls This is purely theoretical, right? Has anyone ever observed this? 2) high-level wakeup race: check for pending signals, have a signal come in, then call a blocking syscall/library (oblivious to the new signal). Right. That's the race which really does happen, and for which the current lame-y work-around is to use a short timeout. [...] Anyway, I would still like to discuss this on #python-dev Monday. Adam, in what time zone are you? (I'm PST.) Who else is interested? MST. Unfortunately I can't stay at work later than 5:30 or so which would be too early for you I believe. I could try again after 8pm, your 9pm. Would that work at all? Otherwise I'd rather try earlier in the day if that works at all for you. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 6:54 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 5:30 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 5:21 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Hm... Does this mean you're *always* creating an extra thread to handle signals? Yup, Py_Initialize will do it. That's unacceptable. It must be possible to build Python without threads (and still support signals -- in fact one could argue that signals make *more* sense when there are no threads :-). For my patch it won't make much sense to disable threads, so I don't mind taking liberties there. [...] To summarize, there's two problems to be solved: 1) low-level corruption in the signal handlers as they record a new signal, such as in Py_AddPendingCalls This is purely theoretical, right? Has anyone ever observed this? I've never heard of it happening. If the compiler doesn't do much reordering (the CPU isn't an issue as this is only called in the main thread) then the most you might get is dropped calls. It's fairly safe the way signal handlers use it, but they'd work just as well (and easier to understand/verify) without the whole queue aspect; just setting some flags and resetting _Py_Ticker. 2) high-level wakeup race: check for pending signals, have a signal come in, then call a blocking syscall/library (oblivious to the new signal). Right. That's the race which really does happen, and for which the current lame-y work-around is to use a short timeout. [...] Anyway, I would still like to discuss this on #python-dev Monday. Adam, in what time zone are you? (I'm PST.) Who else is interested? MST. Unfortunately I can't stay at work later than 5:30 or so which would be too early for you I believe. I could try again after 8pm, your 9pm. Would that work at all? Otherwise I'd rather try earlier in the day if that works at all for you. 5:30 am or 5:30 pm? Any time after 11 am MST (10 am PST) should be fine for me. (My previous email was a little naive about how late I get up.) I shouldn't be gone until around midnight MST. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
2007/12/7, Sean Reifschneider [EMAIL PROTECTED]: FYI: I have no real interest in this, a friend of mine is interested in this, just from a why is powertop saying pygtk is waking up 10 times a second on my laptop? standpoint. So I'm just trying to shepherd this. As a Gnome user, I'm personally +1 for these improvements. But, in a general point of view, I'd hate to see a article somewhere in the future saying something like using GTK is ok, but if you want to be power-friendly, avoid using it from Python... Regards, -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Thu, Dec 06, 2007 at 10:55:12PM -0700, Adam Olsen wrote: That's pretty much what issue1564547 does. I think there's two marks Good point, I hadn't seen that before. * Using poll and fd's is pretty platform specific for what should be a general-purpose API I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. It doesn't cause breakage of other platforms, it just may not help them. Thanks, Sean -- Linux, because eventually you grow up enough to be trusted with a fork(). Sean Reifschneider, Member of Technical Staff [EMAIL PROTECTED] tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability Back off man. I'm a scientist. http://HackingSociety.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 07/12/2007, Sean Reifschneider [EMAIL PROTECTED] wrote: On Thu, Dec 06, 2007 at 10:55:12PM -0700, Adam Olsen wrote: That's pretty much what issue1564547 does. I think there's two marks Good point, I hadn't seen that before. * Using poll and fd's is pretty platform specific for what should be a general-purpose API I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. It doesn't cause breakage of other platforms, it just may not help them. Not only that, but current python signal handling is not theorethically async safe; there are race conditions in the Py_AddPendingCalls API, and it just happens to work most of the time. BTW, the problem is described here: http://mail.python.org/pipermail/python-dev/2006-September/068569.html Think of even python select.poll and multiple threads. If you have dozens of threads, each blocked in some system call, when a signal arrives it will interrupt one of the system calls, causing it to return EINTR, and then python checks for signals. Now imagine all the non-main threads are not created by python; then one of the threads that wakes up could very well be non-python one, and so python will never realize a signal was delivered. The solution of blocking signals in all but the python thread(s) is not feasible as we cannot control all threads that are created, some are created by C libraries... -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 7, 2007 2:35 PM, [EMAIL PROTECTED] wrote: On 02:48 pm, [EMAIL PROTECTED] wrote: Not only that, but current python signal handling is not theorethically async safe; there are race conditions in the Py_AddPendingCalls API, and it just happens to work most of the time. [This refers to the internal datastructures used by Py_AddPendingCalls, which aren't updated in a safe way. Hard/impossible to fix in C, but fairly easy with embedded assembly.] Twisted has encountered one such issue, described here: http://twistedmatrix.com/trac/ticket/1997#comment:12 [This refers to the overall design, which is inherently racey.] Unfortunately, I don't know enough about signals to suggest or comment on the solution. Any Python/C wrapper around a syscall which can be interrupted needs to somehow atomically check for the presence of pending python signal handlers; I don't know of any POSIX API to do that. Overall, what you'd need to do is register a wakeup function (to be called by a signal handler or another thread), and have that wakeup function cancel whatever you're doing. The hard part is it needs to work at *ANY* time while it's registered, before you've even called the library function or syscall you intend to cancel! I currently know of two methods of achieving this: 1) If reading a file or socket, first poll the fd, then do a non-blocking read. The wakeup function writes to a wakeup pipe you also poll, which then wakes you up. A wakeup after poll completes is ignored, but the non-blocking read will finish promptly enough anyway. 2) Use sigsetjmp before a syscall (I wouldn't trust a library call), then have the signal handler jump completely out of the operation. This is evil and unportable, but probably works. Additionally, this only gets SIGINT with the default behaviour to work right, as it can be entirely implemented in C. If you want to handle arbitrary signals running arbitrary python code you really need a second thread to run them in. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Overview (my reading of it): PyGTK wakes up 10x a second in it's main loop because signals may be delivered to another thread and will only get picked up if the mainloop wakes up. In the following thread: http://mail.python.org/pipermail/python-dev/2006-September/068569.html it sounds like the patch at: http://bugs.python.org/issue1564547 doesn't solve the problem. A recent gnome bug brings this issue back up: http://bugzilla.gnome.org/show_bug.cgi?id=481569 I went ahead and closed the python issue as rejected to hopefully get some more activity on it. I thought about this some, and I wondered if there was some way we could signal the sleeping thread when a signal came in on another thread. Like perhaps we could make some code to create a pipe, and put it someplace that all threads could get access to. Then, if a thread gets a signal, write on this pipe. The mainloop could include this file descriptor in the set it's watching, so it would wake up when the signal came in. Is this something Python should provide, or something PyGTK should do? If an approach like the above would work, we could make it so that select() always created this file descriptor and added it to one of the FD sets, so that it would do the right thing behind the scenes. I have no idea if this is a reasonable approach, but it's something that came to mind when I thought about the problem and was an approach I didn't see mentioned before in the discussion. Thanks, Sean -- I live my life like I type; Fast with lots of mistakes. Sean Reifschneider, Member of Technical Staff [EMAIL PROTECTED] tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability Back off man. I'm a scientist. http://HackingSociety.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
The OLPC project is also interested in getting this solved. Can you explain how Gtk uses signals and threads together? The combination strikes me as a recipe for disaster, but I'm probably missing something. --Guido On Dec 6, 2007 8:56 PM, Sean Reifschneider [EMAIL PROTECTED] wrote: Overview (my reading of it): PyGTK wakes up 10x a second in it's main loop because signals may be delivered to another thread and will only get picked up if the mainloop wakes up. In the following thread: http://mail.python.org/pipermail/python-dev/2006-September/068569.html it sounds like the patch at: http://bugs.python.org/issue1564547 doesn't solve the problem. A recent gnome bug brings this issue back up: http://bugzilla.gnome.org/show_bug.cgi?id=481569 I went ahead and closed the python issue as rejected to hopefully get some more activity on it. I thought about this some, and I wondered if there was some way we could signal the sleeping thread when a signal came in on another thread. Like perhaps we could make some code to create a pipe, and put it someplace that all threads could get access to. Then, if a thread gets a signal, write on this pipe. The mainloop could include this file descriptor in the set it's watching, so it would wake up when the signal came in. Is this something Python should provide, or something PyGTK should do? If an approach like the above would work, we could make it so that select() always created this file descriptor and added it to one of the FD sets, so that it would do the right thing behind the scenes. I have no idea if this is a reasonable approach, but it's something that came to mind when I thought about the problem and was an approach I didn't see mentioned before in the discussion. Thanks, Sean -- I live my life like I type; Fast with lots of mistakes. Sean Reifschneider, Member of Technical Staff [EMAIL PROTECTED] tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability Back off man. I'm a scientist. http://HackingSociety.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 6, 2007 9:56 PM, Sean Reifschneider [EMAIL PROTECTED] wrote: Overview (my reading of it): PyGTK wakes up 10x a second in it's main loop because signals may be delivered to another thread and will only get picked up if the mainloop wakes up. In the following thread: http://mail.python.org/pipermail/python-dev/2006-September/068569.html it sounds like the patch at: http://bugs.python.org/issue1564547 doesn't solve the problem. A recent gnome bug brings this issue back up: http://bugzilla.gnome.org/show_bug.cgi?id=481569 I went ahead and closed the python issue as rejected to hopefully get some more activity on it. I thought about this some, and I wondered if there was some way we could signal the sleeping thread when a signal came in on another thread. Like perhaps we could make some code to create a pipe, and put it someplace that all threads could get access to. Then, if a thread gets a signal, write on this pipe. The mainloop could include this file descriptor in the set it's watching, so it would wake up when the signal came in. Is this something Python should provide, or something PyGTK should do? If an approach like the above would work, we could make it so that select() always created this file descriptor and added it to one of the FD sets, so that it would do the right thing behind the scenes. I have no idea if this is a reasonable approach, but it's something that came to mind when I thought about the problem and was an approach I didn't see mentioned before in the discussion. That's pretty much what issue1564547 does. I think there's two marks against it: * Using poll and fd's is pretty platform specific for what should be a general-purpose API * Handling signals is icky, hard to get right, and nobody trusts it Since I don't think there's any more immediate solutions I'll provide a plan B: my threading patch[1] will have a dedicated signal handler thread, allowing them to be processed regardless of one blocked thread. I'm also providing an interrupt API the gtk bindings could use to support wakeups, while keeping the poll+fd details private. [1] http://code.google.com/p/python-safethread/ The patch is, of course, out of date. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
