Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Johan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. --Guido On Dec 7, 2007 3:27 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 7, 2007 2:35 PM, [EMAIL PROTECTED] wrote: On 02:48 pm, [EMAIL PROTECTED] wrote: Not only that, but current python signal handling is not theorethically async safe; there are race conditions in the Py_AddPendingCalls API, and it just happens to work most of the time. [This refers to the internal datastructures used by Py_AddPendingCalls, which aren't updated in a safe way. Hard/impossible to fix in C, but fairly easy with embedded assembly.] Twisted has encountered one such issue, described here: http://twistedmatrix.com/trac/ticket/1997#comment:12 [This refers to the overall design, which is inherently racey.] Unfortunately, I don't know enough about signals to suggest or comment on the solution. Any Python/C wrapper around a syscall which can be interrupted needs to somehow atomically check for the presence of pending python signal handlers; I don't know of any POSIX API to do that. Overall, what you'd need to do is register a wakeup function (to be called by a signal handler or another thread), and have that wakeup function cancel whatever you're doing. The hard part is it needs to work at *ANY* time while it's registered, before you've even called the library function or syscall you intend to cancel! I currently know of two methods of achieving this: 1) If reading a file or socket, first poll the fd, then do a non-blocking read. The wakeup function writes to a wakeup pipe you also poll, which then wakes you up. A wakeup after poll completes is ignored, but the non-blocking read will finish promptly enough anyway. 2) Use sigsetjmp before a syscall (I wouldn't trust a library call), then have the signal handler jump completely out of the operation. This is evil and unportable, but probably works. Additionally, this only gets SIGINT with the default behaviour to work right, as it can be entirely implemented in C. If you want to handle arbitrary signals running arbitrary python code you really need a second thread to run them in. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python tickets summary
Facundo Batista wrote: 2007/11/1, Facundo Batista [EMAIL PROTECTED]: I think the keyword and keywords interface can be improved. Do you have any plans in that direction? Surely! But, no, I have no plans to do it, as I can not make cgi scripts in my hosting, so these pages are statics, generated every night, and that's all... Well, after my hosting allowing CGI, I now improved *a lot* the interface of this page. Looks much improved! :-) Now you have more columns: - Id - Summary - Priority - Severity - Components - Versions - Keywords - Opened by (when) - Temporal location - Last update by (when) And, the biggest enhancement, you can filter by any combination of: - Priority - Severity - Component - Version - Keyword Maybe components and keywords could be combined together and use check boxes so more than one item at a time can be selected? The following suggestions will probably need changes to the data base. Maybe this be done at some point in the future. Ideally the temporal bar could be more like a mini gant/status chart which indicates the status as well as th activity. Maybe the background color can change when the status changes. The possible status sequences might be something like the following. Bugs... 1. bug bug_confirmed fix_provided fix_accepted fix_applied closed 2. bug invalid closed Features... 3. feature patch_provided patch_accepted patch_applied closed 4. feature patch_provided rejected closed 5. feature rejected closed Filtering on the above status keywords would give good results I think. The fix-provided and patch_provided status items might be split into tests, docs, and patch/fix provided. But that may not be needed. Cheers, Ron As before, you have everything paged, and with a graph of activity per day at the bottom. Enjoy it!: http://www.taniquetil.com.ar/facundo/py_tickets.html Regards, ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug Day in January?
Giampaolo Rodola' schrieb: O.T. - I noticed that the PEP-3 still refers to the old sourceforge bug tracker. Shouldn't it be rewritten? Yes, thanks! I did a minimal change, but still someone could elaborate it, describing current practices. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Not only that, but pygtk is a generic module; who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Slow tests involving bsddb - timeout
Guido I think I've seen this too when running the bsddb3 unittest. I Guido think it's caused by a previous test ending badly and leaving Guido junk behind that the test suite doesn't properly remove before Guido starting. But I don't recall the details. skip Thanks, that at least gives me some hope that I can try to narrow skip down the problem with a little binary searching. The binary search wasn't very difficult. Ran regrtest with the -v and -f flags. test_anydbm was the only test. Here's the output: % ./python.exe -E -tt ../Lib/test/regrtest.py -f tests -v -uall test_anydbm test_anydbm_creation (test.test_anydbm.AnyDBMTestCase) ... ERROR test_anydbm_keys (test.test_anydbm.AnyDBMTestCase) ... ERROR test_anydbm_modification (test.test_anydbm.AnyDBMTestCase) ... ERROR test_anydbm_read (test.test_anydbm.AnyDBMTestCase) ... ERROR == ERROR: test_anydbm_creation (test.test_anydbm.AnyDBMTestCase) -- Traceback (most recent call last): File /Users/skip/src/python/trunk/Lib/test/test_anydbm.py, line 37, in test_anydbm_creation f = anydbm.open(_fname, 'c') File /Users/skip/src/python/trunk/Lib/anydbm.py, line 83, in open return mod.open(file, flag, mode) File /Users/skip/src/python/trunk/Lib/dbhash.py, line 16, in open return bsddb.hashopen(file, flag, mode) File /Users/skip/src/python/trunk/Lib/bsddb/__init__.py, line 310, in hashopen d.open(file, db.DB_HASH, flags, mode) DBFileExistsError: (17, 'File exists -- __fop_file_setup: Retry limit (100) exceeded') == ERROR: test_anydbm_keys (test.test_anydbm.AnyDBMTestCase) -- Traceback (most recent call last): File /Users/skip/src/python/trunk/Lib/test/test_anydbm.py, line 58, in test_anydbm_keys self.init_db() File /Users/skip/src/python/trunk/Lib/test/test_anydbm.py, line 69, in init_db f = anydbm.open(_fname, 'n') File /Users/skip/src/python/trunk/Lib/anydbm.py, line 83, in open return mod.open(file, flag, mode) File /Users/skip/src/python/trunk/Lib/dbhash.py, line 16, in open return bsddb.hashopen(file, flag, mode) File /Users/skip/src/python/trunk/Lib/bsddb/__init__.py, line 310, in hashopen d.open(file, db.DB_HASH, flags, mode) DBFileExistsError: (17, 'File exists -- __fop_file_setup: Retry limit (100) exceeded') == ERROR: test_anydbm_modification (test.test_anydbm.AnyDBMTestCase) -- Traceback (most recent call last): File /Users/skip/src/python/trunk/Lib/test/test_anydbm.py, line 45, in test_anydbm_modification self.init_db() File /Users/skip/src/python/trunk/Lib/test/test_anydbm.py, line 69, in init_db f = anydbm.open(_fname, 'n') File /Users/skip/src/python/trunk/Lib/anydbm.py, line 83, in open return mod.open(file, flag, mode) File /Users/skip/src/python/trunk/Lib/dbhash.py, line 16, in open return bsddb.hashopen(file, flag, mode) File /Users/skip/src/python/trunk/Lib/bsddb/__init__.py, line 310, in hashopen d.open(file, db.DB_HASH, flags, mode) DBFileExistsError: (17, 'File exists -- __fop_file_setup: Retry limit (100) exceeded') == ERROR: test_anydbm_read (test.test_anydbm.AnyDBMTestCase) -- Traceback (most recent call last): File /Users/skip/src/python/trunk/Lib/test/test_anydbm.py, line 52, in test_anydbm_read self.init_db() File /Users/skip/src/python/trunk/Lib/test/test_anydbm.py, line 69, in init_db f = anydbm.open(_fname, 'n') File /Users/skip/src/python/trunk/Lib/anydbm.py, line 83, in open return mod.open(file, flag, mode) File /Users/skip/src/python/trunk/Lib/dbhash.py, line 16, in open return bsddb.hashopen(file, flag, mode) File /Users/skip/src/python/trunk/Lib/bsddb/__init__.py, line 310, in hashopen d.open(file, db.DB_HASH, flags, mode) DBFileExistsError: (17, 'File exists -- __fop_file_setup: Retry limit (100) exceeded') -- Ran 4 tests in 404.353s FAILED (errors=4) test test_anydbm failed -- errors occurred; run in verbose mode for details 1 test failed: test_anydbm Skip ___ Python-Dev mailing list Python-Dev@python.org
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 9:20 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Sorry, I misattributed this; Gustavo Carneiro pointed this out earlier in this thread. BTW, in the referenced post (also by Gustavo), I found this: According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 Unfortunately I can't read [1] without having registered, so I can only guess what it says. But I know that Python already ensures that signals are only delivered to the main thread. What am I missing? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
At 11:17 AM +0100 12/8/07, Johan Dahlin wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Is that always required (with threads), or are things better now that Ctrl-C handling is improved (at least in the Socket module, which doesn't lose signals anymore)? -- TonyN.:' mailto:[EMAIL PROTECTED] ' http://www.georgeanelson.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? Well, actually this is also a PyGObject issue, not only PyGtk. PyGObject wraps GLib. GLib was created to serve the needs of Gtk+, but is useful by itself for writing portable programs. Among other things, GLib offers a generic main loop, which programs can use to do generic event-driven programming, such as timeouts, polling file descriptors, and doing other work when the main loop becomes idle. You could argue that non-gtk programs are rare and we shouldn't worry too much about them. Maybe it's true, I don't know. who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Funny that everyone mentions this solution, as it is the solution implemented by my patch :-) Well, to be fair, it might not be _exactly_ what is implemented by the patch. Reading between the lines, I think what you mean is to have python's C signal handler mostly untouched; it would only write a byte to a pipe _in addition to_ the normal setting the flag and Py_AddPendingCall. The patch I submitted, on the other hand, completely removes the vector of flags and Py_AddPendingCall, and instead writes the number of the signal that was raised into the pipe, and reads it back from the pipe in the Python main loop. Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). My solution is being reject because people are afraid to touch the signal handling code, which has its faults, but well know faults. But if I refactor the patch to keep the crappy-but-sort-of-working signal code, and only enhance it with the pipe, maybe it will more likely get accepted. Do I understand correctly? :-) -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 9:20 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Sorry, I misattributed this; Gustavo Carneiro pointed this out earlier in this thread. BTW, in the referenced post (also by Gustavo), I found this: According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 Unfortunately I can't read [1] without having registered, so I can only guess what it says. I can't read [1] either because the URL is not correct. Sorry :P But I know that Python already ensures that signals are only delivered to the main thread. Are you sure? In python code I see pthread_sigmask being checked for in configure.in, but I can't find it being used anywhere (unless I'm grepping for the wrong thing). What you probably meant to say was python ensures that signals are only processed from the main thread. Except when python uses GNU PTH threads; there I do see signals being delivered to the main thread. But GNU pth are not real kernel threads, anyway. What am I missing? I think Python could make sure (via pthread_sigmask) that signals are blocked for all non-main threads created by Python. Unfortunately it cannot do the same for threads not created by Python. I know at least one case where threads are created behing Python's back: the GnomeVFS library. Best regards, -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 Unfortunately I can't read [1] without having registered, so I can only guess what it says. But I know that Python already ensures that signals are only delivered to the main thread. What am I missing? Your notion of delivers differs. Python does receive signals in threads other than the main thread. However, it will only invoke the *Python* signal handler in the main thread. signalmodule.c:signal_handler calls Py_AddPendingCall, which is only ever considered in the main thread (although perhaps not in a timely manner - this is precisely where the busy-waiting in gtk comes from). Python does not attempt to block any signals, let alone using pthread_sigmask. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
Hello Python developers. This is my first post on the list. I have been using Python for a while and I have been thinking about one feature I would like to see integrated into the language. I thought it could be a good topic for a PEP, so I decided to join the list and write about it. First problem: empty generators. Suppose you have to write an empty generator. You could do something like: def foo(): return yield 'never' Or something like: def foo(): yield iter([]).next() Or: def foo(): raise StopIteration() yield never There is an old thread discussing the diferent alternatives: http://mail.python.org/pipermail/python-list/2002-November/171588.html Of curse this is unpythonic. It violates the principle of: There should be one-- and preferably only one --obvious way to do it. Instead, we have a bunch of inelegant solutions, and no one is the obvious one. Second problem: Explicit raise without explicit try-except. Take a look at this example: def lines(): for line in my_file: if some_error(): raise StopIteration() yield line yield 'end' for line in lines(): do_something() Even when the lines function contains an explicit raise statement, there is no explicit try-except block. Of curse, the StopIteration exception is implicitly caught when the generator is called, but this looks a bit confusing. In my opinion, every explicitly raised exception should be explicitly caught by a try-except block. The solution: yield break. The solution used in C# for these problems is the 'yield break' statement. In this way, the empty generator would look like: def foo(): yield break This would be the pythonic way of writing an empty generator. In the same way, every time you want to stop the generation you should call 'yield break', for example: def lines(): for line in my_file: if some_error(): yield break yield line yield 'end' Note that 'yield break' resembles the 'break' statement used in loops, while 'StopIteration' doesn't. 'yield break' is more orthogonal to the rest of the language. I am looking forward to seeing your opinions. Manuel Cerón. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
Note that 'yield break' resembles the 'break' statement used in loops, while 'StopIteration' doesn't. 'yield break' is more orthogonal to the rest of the language. I am looking forward to seeing your opinions. -1 I do not find the meaning to be transparent and the proposal adds new syntax without adding functionality. Also, I do not find the emtpy generator use case to be even slightly motivating. If an empty iterator were needed for some reason, I would just write: iter([]) and be done with it. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 9:55 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: On Dec 8, 2007 2:17 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Hm. How about making that an option? I don't think on the OLPC XO this is a valid use case (end users never have a console where they might enter ^C). It could easily be made into a compilation option which would solve the problem specifically for OLPC, but it would still be problematic for other platforms important to PyGTK (linux/gnome) where console based development is more common. But do those other platforms care about the extra CPU cycles and power used? I suspect not, at least not to the extent that OLPC cares. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 9:57 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? Well, actually this is also a PyGObject issue, not only PyGtk. PyGObject wraps GLib. GLib was created to serve the needs of Gtk+, but is useful by itself for writing portable programs. Among other things, GLib offers a generic main loop, which programs can use to do generic event-driven programming, such as timeouts, polling file descriptors, and doing other work when the main loop becomes idle. You could argue that non-gtk programs are rare and we shouldn't worry too much about them. Maybe it's true, I don't know. who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Funny that everyone mentions this solution, as it is the solution implemented by my patch :-) Mind pointing me to it again? I can't find it in the Python bug tracker. Well, to be fair, it might not be _exactly_ what is implemented by the patch. Reading between the lines, I think what you mean is to have python's C signal handler mostly untouched; it would only write a byte to a pipe _in addition to_ the normal setting the flag and Py_AddPendingCall. Well, in many cases I see no problems with the current signal handler, and people are used to it, so I'm not sure what is gained by getting rid of it. The patch I submitted, on the other hand, completely removes the vector of flags and Py_AddPendingCall, and instead writes the number of the signal that was raised into the pipe, and reads it back from the pipe in the Python main loop. I believe Py_AddPendingCall was meant to be used by other places besides the signal handling. Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). I think it's okay to drop signals if too many come. The FD should be put in non-blocking mode so the signal handler won't block forever. Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? My solution is being reject because people are afraid to touch the signal handling code, which has its faults, but well know faults. But if I refactor the patch to keep the crappy-but-sort-of-working signal code, and only enhance it with the pipe, maybe it will more likely get accepted. Do I understand correctly? :-) Not really; I don't recall seeing your patch. And I object to your subjective description of the existing signal handling; it has served us well enough. I
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 9:57 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:58 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Not only that, but pygtk is a generic module; What does generic mean in this context? Surely not that it applies to other libraries than GTK? Well, actually this is also a PyGObject issue, not only PyGtk. PyGObject wraps GLib. GLib was created to serve the needs of Gtk+, but is useful by itself for writing portable programs. Among other things, GLib offers a generic main loop, which programs can use to do generic event-driven programming, such as timeouts, polling file descriptors, and doing other work when the main loop becomes idle. You could argue that non-gtk programs are rare and we shouldn't worry too much about them. Maybe it's true, I don't know. who are we to forbid the usage of signals if python itself allows it? If we were to ignore signals then sooner or later someone would come along and shout hey, signals work just fine in pure python, so why did pygtk have to break my signals?. Um, signals don't work just fine in pure Python. And I would argue they don't either in C. They are extremely subtle, and most code using signals is broken in some way. Just try to read the sigaction() man page and claim you understand it. Unfortunately, in Unix there are some conditions that can only be delivered using signals (e.g. SIGINTR, SIGTERM) and others for which your choices are either polling or signals (SIGCHILD, SIGWINCH). Traditionally, solutions based on select() or poll() with a short timeout (e.g. 20 or 100 ms) have worked well, as the number of instructions executed each time is really negligeable, and the response time is still reasonable on a human time scale. Unfortunately it seems recent developments in power management for ultra-low power devices have made this an issue again. Windows solves this more elegantly by having a unified wait for multiple conditions system call which can wait on I/O, semaphores, and other events (within the same process or coming from other processes). Unfortunately, in Unix, some events don't have a file descriptor associated with them, and for those select()/poll() cannot be used. The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). Funny that everyone mentions this solution, as it is the solution implemented by my patch :-) Mind pointing me to it again? I can't find it in the Python bug tracker. Well, to be fair, it might not be _exactly_ what is implemented by the patch. Reading between the lines, I think what you mean is to have python's C signal handler mostly untouched; it would only write a byte to a pipe _in addition to_ the normal setting the flag and Py_AddPendingCall. Well, in many cases I see no problems with the current signal handler, and people are used to it, so I'm not sure what is gained by getting rid of it. The patch I submitted, on the other hand, completely removes the vector of flags and Py_AddPendingCall, and instead writes the number of the signal that was raised into the pipe, and reads it back from the pipe in the Python main loop. I believe Py_AddPendingCall was meant to be used by other places besides the signal handling. True. The patch does not remove Py_AddPendingCall, only stops using it for signals. Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). I think it's okay to drop signals if too many come. The FD should be put in non-blocking mode so the signal handler won't block forever. Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? Good point. My solution is being reject because people are afraid to touch the signal handling code, which has its faults, but well know faults. But if I refactor the patch to keep the crappy-but-sort-of-working signal code, and only
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
I would prefer plain 'yield' to 'yield break' as a synonym for 'raise StopIteration'. But the raise stands out better visually as an exit point. The point about the raise not being explicitly caught is not valid since any generator could be used in the following code g = some_gen try: while True: process(g.next()) except StopIteration: whatever() where it is caught. Then under your proposal, there would be a catch without a throw ;-(. If there were a use case for empty generators, then iter() could be made to produce one, in analogy with int() returning 0, etc. But without a use case, 'iter()' is much more likely to be a bug and should be flagged as such. In the meanwhile, 'iter(())' is trivial to type. Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
2007/12/8, Raymond Hettinger [EMAIL PROTECTED]: ...the proposal adds new syntax without adding functionality. That is indeed the definition of syntactic sugar [1]. Python is full of that, for example the import statement. 2007/12/8, Paul Svensson [EMAIL PROTECTED]: What is the problem that is not solved by iter(()) ? 'yield iter(()).next()' it's obscure, you can't know its purpose at first sight. There are other alternatives but none of them seems to be the obvious way to do it. But that is the less important problem. The real problem is that raising StopIteration is not orthogonal. It is confusing for the reasons I exposed. Generators are something in the middle between a language feature and a framework/library. With 'yield break' they will be completely a language feature. [1] http://en.wikipedia.org/wiki/Syntactic_sugar Manuel. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
In your example, why do you raise StopIteration instead just writing return? - Original Message - From: Manuel Alejandro Cerón Estrada [EMAIL PROTECTED] Take a look at this example: def lines(): for line in my_file: if some_error(): raise StopIteration() yield line yield 'end' for line in lines(): do_something() ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 1:38 PM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 08/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 9:57 AM, Gustavo Carneiro [EMAIL PROTECTED] wrote: Which is the best solution? I think my patch fixes two problems: 1. the need to have a FD to wake up poll() (t o fix the problem with what we are discussing in this thread), and 2. make Python's signal handling more reliable (not 100% reliable because it doesn't handle longer bursts of signals than the pipe buffer can take, but at least is race free). I think it's okay to drop signals if too many come. The FD should be put in non-blocking mode so the signal handler won't block forever. Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? Good point. Note that we may drop a new signal, not the same one we got several times. I don't know if Unix will do that. Then again, I've been unable to find documentation promising it'd deliver any signal at all. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
Manuel Alejandro Cerón Estrada wrote: 2007/12/8, Raymond Hettinger [EMAIL PROTECTED]: ...the proposal adds new syntax without adding functionality. That is indeed the definition of syntactic sugar [1]. Python is full of that, for example the import statement. 2007/12/8, Paul Svensson [EMAIL PROTECTED]: What is the problem that is not solved by iter(()) ? 'yield iter(()).next()' it's obscure, you can't know its purpose at first sight. There are other alternatives but none of them seems to be the obvious way to do it. But surely iter(()) itself (and indeed iter(any empty sequence) is precisely the empty generator you seek, without having to use a def statement to create your own in Python. for i in iter(()): ... print Result ... So I don't see what's so magical about creating your own empty iterator, nor why you consider it necessary. The 2002 discussion to which you refer is clearly outdated. But that is the less important problem. The real problem is that raising StopIteration is not orthogonal. It is confusing for the reasons I exposed. Generators are something in the middle between a language feature and a framework/library. With 'yield break' they will be completely a language feature. [1] http://en.wikipedia.org/wiki/Syntactic_sugar It's necessary, in discussions of syntactic sugar, to admit from the start that a certain amount of sweetness is desirable. If you would really like to remove the import statement you might consider programming in brainfuck, one of the least sugary languages around. http://en.wikipedia.org/wiki/Brainfuck Of course, Python is popular because it hits many people's sweet spot for the desirable balance between syntactic sugar and minimalism, making it possible to write compact solutions to advanced problems. There are definitely limits, though. yield break seems ugly and unnecessary to me. I'd rather consider Icon's break break to exit two levels of looping, but I don't see that flying either. You still haven't made a convincing use case for yield break, IMHO. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
2007/12/8, Raymond Hettinger [EMAIL PROTECTED]: In your example, why do you raise StopIteration instead just writing return? In the example would be better to use 'return' rather than 'raise StopIteration'. In most cases 'return' will have the same behavior than 'raise StopIteration', but not always. Acording to PEP 255: Note that return isn't always equivalent to raising StopIteration: the difference lies in how enclosing try/except constructs are treated. The trivial example is the empty generator where return is ambiguous with the same statement for normal functions. Maybe my proposal can change to Replace 'return' with 'yield break' in generators Again, PEP 255 talks about the possibility of removing 'return' from generators Q. Why allow return at all? Why not force termination to be spelled raise StopIteration? A. The mechanics of StopIteration are low-level details, much like the mechanics of IndexError in Python 2.1: the implementation needs to do *something* well-defined under the covers, and Python exposes these mechanisms for advanced users. That's not an argument for forcing everyone to work at that level, though. return means I'm done in any kind of function, and that's easy to explain and to use. Note that return isn't always equivalent to raise StopIteration in try/except construct, either (see the Specification: Return section). Of curse, the problem of low level details it's solved by 'yield break' Manuel. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Security Advisory for unicode repr() bug?
___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] blocking a non-blocking socket
An interesting question has come up in the development of the SSL module. The function ssl.wrap_socket() takes a flag, do_handshake_on_connect, which tells it whether to do the SSL handshake before returning an SSLSocket object to the caller. If the socket being wrapped is non-blocking, the code in wrap_socket() must invoke do_handshake() multiple times, and effectively block until the handshake is done. Right now, I'm doing it with this loop: if do_handshake_on_connect: # have to loop to support non-blocking sockets while True: try: self.do_handshake() break except SSLError, err: if err.args[0] == SSL_ERROR_WANT_READ: select.select([self], [], []) elif err.args[0] == SSL_ERROR_WANT_WRITE: select.select([], [self], []) else: raise Hello Bill, Another way of doing it could be to expose a connect() method on the ssl objects. It changes the socket.ssl api, but I'd say it is in the same spirit as the do_handshake_on_connect parameter since no existing code will break. The caller then calls connect() until it does not return SSL_ERROR_WANT_[WRITE|READ]. This only applies when the underlying socket is non-blocking and the do_handshake_on_connect parameter is false. Arguably, this is similar to how normal sockets are treated in asyncore, only that he caller must be prepared to respond to (multiple?) readable and writable events for the connection to be established. Cheers Audun ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] python25 on Windows 2003 Enterprise Server OS
Hello, Does python25 supports on the Windows 2003 Enterprise Server OS ? Thanks Meerovich Dima, Software Intel - Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] python25 on Windows 2003 Enterprise Server OS
Hello, Does python25 supports on the Windows 2003 Enterprise Server OS ? Thanks Meerovich Dima, Software Intel - Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Does anyone care enough about asyncore and asynchat to help adapt their APIs for Py3k?
On Dec 5, 2007 9:19 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The asyncore and asynchat modules are in a difficult position when it comes to Python 3000. None of the core developers use it or particularly care about it (AFAIK), and the API has problems because it wasn't written to deal with bytes vs. unicode. E.g. in http://bugs.python.org/issue1067, Thomas suggests that these modules need to be rewritten to use bytes internally and have separate APIs to handle (unicode) text as desired, similar to the way file I/O was redesigned. Another alternative would be to make these modules deal strictly in bytes, but that would probably vastly reduce their usefulness (though I don't know -- as I said, I don't use them). I can look into it later this month, but for the short term, I'm a little squeezed for time (work, finishing school, etc.). I am a bit curious, however, I could have sworn that bytes were to become, essentially, the 2.x str type (without some methods). If that is the case, no changes should really be necessary, except for a few small changes to asynchat with regards to line terminators. Then again, I haven't really been keeping up in the p3k/pydev lists for the last 6-8 months, and only the subject line of this posting caught my eye (because I use asyncore/asynchat, and support users of asyncore/asynchat that contact me directly). - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Does anyone care enough about asyncore and asynchat to help adapt their APIs for Py3k?
Guido, These modules provide the server component of many existing versions of the Zope system where they have served well (pun intended).I would be concerned were they disappear in Python 3000. On Dec 5, 2007 10:19 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The asyncore and asynchat modules are in a difficult position when it comes to Python 3000. None of the core developers use it or particularly care about it (AFAIK), and the API has problems because it wasn't written to deal with bytes vs. unicode. E.g. in http://bugs.python.org/issue1067, Thomas suggests that these modules need to be rewritten to use bytes internally and have separate APIs to handle (unicode) text as desired, similar to the way file I/O was redesigned. Another alternative would be to make these modules deal strictly in bytes, but that would probably vastly reduce their usefulness (though I don't know -- as I said, I don't use them). -- --Guido van Rossum (home page: http://www.python.org/~guido/http://www.python.org/%7Eguido/ ) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/allison%40shasta.stanford.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: On Dec 8, 2007 2:17 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Hm. How about making that an option? I don't think on the OLPC XO this is a valid use case (end users never have a console where they might enter ^C). It could easily be made into a compilation option which would solve the problem specifically for OLPC, but it would still be problematic for other platforms important to PyGTK (linux/gnome) where console based development is more common. Johan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: On Dec 8, 2007 9:55 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: On Dec 8, 2007 2:17 AM, Johan Dahlin [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Adam, perhaps at some point (Monday?) we could get together on #python-dev and interact in real time on this issue. Probably even better on the phone. This offer is open to anyone who is serious about getting this resolved. Someone please take it -- I'm offering free consulting here! I'm curious -- is there anyone here who understands why [Py]GTK is using signals anyway? It's not like writing robust signal handling code in C is at all easy or obvious. If instead of a signal a file descriptor could be used, all problems would likely be gone. The timeout handler was added for KeyboardInterrupt to be able to work when you want to Ctrl-C yourself out of the gtk.main() loop. Hm. How about making that an option? I don't think on the OLPC XO this is a valid use case (end users never have a console where they might enter ^C). It could easily be made into a compilation option which would solve the problem specifically for OLPC, but it would still be problematic for other platforms important to PyGTK (linux/gnome) where console based development is more common. But do those other platforms care about the extra CPU cycles and power used? I suspect not, at least not to the extent that OLPC cares. They most certainly do, pygtk applications are used in desktop software used on laptops where battery time is increasingly important. Johan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
Terry Reedy wrote: I would prefer plain 'yield' to 'yield break' as a synonym for 'raise StopIteration'. Don't we already have a plain yield these days meaning something else? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
2007/12/8, Raymond Hettinger [EMAIL PROTECTED]: ...the proposal adds new syntax without adding functionality. That is indeed the definition of syntactic sugar [1]. Python is full of that, for example the import statement. . . . The real problem is that raising StopIteration is not orthogonal. It is confusing for the reasons I exposed. Generators are something in the middle between a language feature and a framework/library. With 'yield break' they will be completely a language feature. This personal notion of purity carries almost zero weight and is more a matter of personal taste/style than anything else. It certainly does not warrant a change to the grammar. I've been a big fan of generators and tend to use them everywhere. I've *never* had a use case for yield break and find it uncommon to even raise StopIteration. The tone of your messages suggests that you're going to stick to this idea like glue, so to avoid further time wasting, this will likely be my last post on the subject. If for some reason, you persist through to writing a PEP, it will be almost mandatory to scan existing, published code to find examples of code fragments that would be improved by 'yield break' . My bet is that the examples will be rare and any improvement will be marginal at best and simply a matter of taste at worst. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
Manuel Alejandro Cerón Estrada wrote: The real problem is that raising StopIteration is not orthogonal. This is a non-problem as far as I can see. In a generator, the way to stop the iteration is simply to return. In a non-generator iterator, you're still going to have to raise StopIteration. So I see no advantage at all to this proposal. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: Does Unix even promise that a signal gets delivered twice if it gets sent quickly twice in a row? No, it doesn't. However, this doesn't necessarily mean that you can just drop them when the pipe becomes full. If more than one type of signal is sharing a pipe, you don't want it getting filled up with signals of one type and causing signals of a different type to get lost. To guarantee that won't happen, you would have to either use a separate pipe for each signal type, or have some other scheme such as a bitmask keeping track of which signal types are in the pipe. I have a worry though -- if signal handlers *always* and *only* communicate by writing a byte to a FD, does that mean that the VM main loop will have to attempt to read a byte from a pipe every time it checks for signals? That sounds like a good reason for having the fd be an additional mechanism that you can turn on when you want it, but not the core mechanism for dealing with signals. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
At 12:30 PM 12/9/2007 +1300, Greg Ewing wrote: Terry Reedy wrote: I would prefer plain 'yield' to 'yield break' as a synonym for 'raise StopIteration'. Don't we already have a plain yield these days meaning something else? Yes. It's equivalent to 'yield None'. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
Manuel Alejandro Cerón Estrada wrote: Acording to PEP 255: Note that return isn't always equivalent to raising StopIteration: the difference lies in how enclosing try/except constructs are treated. All that means is that def g(): try: if 0: yield return except StopIteration: print Spam won't print Spam. But since this involves catching the exception with an explicit try-except, it doesn't fall under the scope of your complaint. Of curse, the problem of low level details it's solved by 'yield break' I would put it the other way around -- the problem that 'yield break' is meant to solve is already solved by 'return'. So there's no need for change. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 3:57 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) Hm... Does this mean you're *always* creating an extra thread to handle signals? I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html Since OLPC has to use 2.5 they don't really have another choice besides this or making the timeout (perhaps much) larger -- I'm not going to risk a change as big as anything proposed here for 2.5.2, so nothing will change before 2.6. I've got to say that all the cross-referencing and asynchronous discussion here makes it *very* difficult to wrap my head around the various proposals. It also doesn't help that different participants appear to have different use cases in mind. E.g. do we care about threads started directly from C++ code? (These happen all the time at Google, but we don't care much about signals.) And what about restarting system calls (like Glyph brought up)? I've seen references to bug #1643738 which got a thumbs up from Tim Peters -- Adam, what do you think of that? I know it doesn't address Gustavo's issue but it seems useful in its own right. Gustavo, at some point you suggested making changes to Python so that all signals are blocked in all threads except for the main thread. I think I'd be more inclined to give
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
2007/12/8, Greg Ewing [EMAIL PROTECTED]: I would put it the other way around -- the problem that 'yield break' is meant to solve is already solved by 'return'. So there's no need for change. I have been re-thinking the problem and this is true. The only exception would be empty generators, but they are rare. 'return' works well for 99.99% of the cases. And the rest 0.01% does not worth a syntax change. Thank you very much for all your comments. Manuel. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] bsddb185 v1.0 for Python 2.6 and 3.0
Python 3.0 will dispense with the rarely used, but occasionally indispensible, bsddb185 module. I extracted the source code and unit tests from the current Python trunk, wrote a setup.py, made a couple slight mods so it would build and pass tests under both Python 2.6 and 3.0. It's available from the Python Package Index: http://pypi.python.org/pypi/bsddb185 Cheers, -- Skip Montanaro - [EMAIL PROTECTED] - http://www.webfast.com/~skip/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
At 07:55 PM 12/8/2007 -0500, Manuel Alejandro Cerón Estrada wrote: 2007/12/8, Greg Ewing [EMAIL PROTECTED]: I would put it the other way around -- the problem that 'yield break' is meant to solve is already solved by 'return'. So there's no need for change. I have been re-thinking the problem and this is true. The only exception would be empty generators, but they are rare. Note that: def g(): return yield Is a valid empty generator. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 5:21 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:57 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) Hm... Does this mean you're *always* creating an extra thread to handle signals? Yup, Py_Initialize will do it. I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html Since OLPC has to use 2.5 they don't really have another choice besides this or making the timeout (perhaps much) larger -- I'm not going to risk a change as big as anything proposed here for 2.5.2, so nothing will change before 2.6. I've got to say that all the cross-referencing and asynchronous discussion here makes it *very* difficult to wrap my head around the various proposals. It also doesn't help that different participants appear to have different use cases in mind. E.g. do we care about threads started directly from C++ code? (These happen all the time at Google, but we don't care much about signals.) And what about restarting system calls (like Glyph brought up)? I've seen references to bug #1643738 which got a thumbs up from Tim Peters -- Adam, what do you think of that? I know it doesn't address Gustavo's issue
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On 09/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 3:57 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 4:28 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:36 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 2:56 PM, [EMAIL PROTECTED] wrote: On 05:20 pm, [EMAIL PROTECTED] wrote: The best solution I can think of is to add a new API that takes a signal and a file descriptor and registers a C-level handler for that signal which writes a byte to the file descriptor. You can then create a pipe, connect the signal handler to the write end, and add the read end to your list of file descriptors passed to select() or poll(). The handler must be written in C in order to avoid the race condition referred to by Glyph (signals arriving after the signal check in the VM main loop but before the select()/poll() system call is entered will not be noticed until the select()/poll() call completes). This paragraph jogged my memory. I remember this exact solution being discussed now, a year ago when I was last talking about these issues. There's another benefit to implementing a write-a-byte C signal handler. Without this feature, it wouldn't make sense to have passed the SA_RESTART flag to sigaction, because and GUIs written in Python could have spent an indefinite amount of time waiting to deliver their signal to Python code. So, if you had to handle SIGCHLD in Python, for example, calls like file().write() would suddenly start raising a new exception (EINTR). With it, you could avoid a whole class of subtle error-handling code in Twisted programs. SA_RESTART still isn't useful. The low-level poll call (not write!) must stop and call back into python. If that doesn't indicate an error you can safely restart your poll call though, and follow it with a (probably non-blocking) write. Can't say I understand all of this, but it does reiterate that there are more problems with signals than just the issue that Gustavo is trying to squash. The possibility of having *any* I/O interrupted is indeed a big worry. Though perhaps this could be alleviated by rigging things so that signals get delivered (at the C level) to the main thread and the rest of the code runs in a non-main thread? That's the approach my threading patch will take, although reversed (signals are handled by a background thread, leaving the main thread as the *main* thread.) Hm... Does this mean you're *always* creating an extra thread to handle signals? I share your concern about interrupting whatever random syscalls (not even limited to I/O!) that a library happens to use. Note that the only reason to use C for a low-level handler here is give access to sigatomic_t and avoid needing locks. If you ran the signal handler in a background thread (using sigwait to trigger them) you could use a python handler. I haven't seen Gustavo's patch yet, but *my* reason for using a C handler was different -- it was because writing a byte to a pipe in Python would do nothing to fix Gustavo's issue. Looking at the man page for sigwait() it could be an alternative solution, but I'm not sure how it would actually allow PyGTK to catch KeyboardInterrupt. My mail at [1] was referring to this. Option 1 involved writing to a pipe that gets polled while option 2 requires we generate a new signal targeting the specific thread we want to interrupt. I'd like to propose an interim solution though: pygtk could install their own SIGINT handler during the gtk mainloop (or all gtk code?), have it write to a pipe monitored by gtk, and have gtk raise KeyboardInterrupt if it gets used. This won't allow custom SIGINT handlers or any other signal handlers to run promptly, but it should be good enough for OLPC's use case. [1] http://mail.python.org/pipermail/python-dev/2007-December/075607.html Since OLPC has to use 2.5 they don't really have another choice besides this or making the timeout (perhaps much) larger -- I'm not going to risk a change as big as anything proposed here for 2.5.2, so nothing will change before 2.6. I've got to say that all the cross-referencing and asynchronous discussion here makes it *very* difficult to wrap my head around the various proposals. It also doesn't help that different participants appear to have different use cases in mind. E.g. do we care about threads started directly from C++ code? (These happen all the time at Google, but we don't care much about signals.) And what about restarting system calls (like Glyph brought up)? I've seen references to bug #1643738 which got a thumbs up from Tim Peters -- Adam, what do you think of that? I know it doesn't address Gustavo's issue but it seems useful in its own right. That issue
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 6:30 PM, Gustavo Carneiro [EMAIL PROTECTED] wrote: On 09/12/2007, Guido van Rossum [EMAIL PROTECTED] wrote: Gustavo, at some point you suggested making changes to Python so that all signals are blocked in all threads except for the main thread. I think I'd be more inclined to give that the green light than the patch using pipes for all signal handling, as long as we can make sure that this blocking of all signals isn't inherited by fork()'ed children -- we had serious problems with that in 2.4 where child processes were unkillable (except for SIGKILL). I don't think that solution works after all. We can only block signals for certain threads inside the threads themselves. But we do not control all threads. Some are created by C libraries, and these threads will not have signals blocked by default, and also there is no 'thread creation hook' that we can use. Note that new threads inherit signal masks from their creator. It's only threads created before loading python that are a problem. For my threading patch I plan to document that as simply something embedders have to do. I'd also be OK with a patch that leaves the existing signal handling code intact but *adds* a way to have a signal handler written in C that writes one byte to one end of a pipe -- where the pipe is provided by Python code. I think this is most balanced approach of all. Yeah, use the existing Handlers array to record which signals have come in, rather using the byte passed through the pipe. And I missed a problem in bug #1643738: Handlers[...].tripped should be a sig_atomic_t, not int. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python25 on Windows 2003 Enterprise Server OS
Probably. Have you tried it? On Dec 5, 2007 1:41 AM, Meerovich, Demitry [EMAIL PROTECTED] wrote: Hello, Does python25 supports on the Windows 2003 Enterprise Server OS ? Thanks Meerovich Dima, Software Intel - Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 5:30 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 5:21 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Hm... Does this mean you're *always* creating an extra thread to handle signals? Yup, Py_Initialize will do it. That's unacceptable. It must be possible to build Python without threads (and still support signals -- in fact one could argue that signals make *more* sense when there are no threads :-). [...] To summarize, there's two problems to be solved: 1) low-level corruption in the signal handlers as they record a new signal, such as in Py_AddPendingCalls This is purely theoretical, right? Has anyone ever observed this? 2) high-level wakeup race: check for pending signals, have a signal come in, then call a blocking syscall/library (oblivious to the new signal). Right. That's the race which really does happen, and for which the current lame-y work-around is to use a short timeout. [...] Anyway, I would still like to discuss this on #python-dev Monday. Adam, in what time zone are you? (I'm PST.) Who else is interested? MST. Unfortunately I can't stay at work later than 5:30 or so which would be too early for you I believe. I could try again after 8pm, your 9pm. Would that work at all? Otherwise I'd rather try earlier in the day if that works at all for you. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
On Dec 8, 2007 6:54 PM, Guido van Rossum [EMAIL PROTECTED] wrote: On Dec 8, 2007 5:30 PM, Adam Olsen [EMAIL PROTECTED] wrote: On Dec 8, 2007 5:21 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Hm... Does this mean you're *always* creating an extra thread to handle signals? Yup, Py_Initialize will do it. That's unacceptable. It must be possible to build Python without threads (and still support signals -- in fact one could argue that signals make *more* sense when there are no threads :-). For my patch it won't make much sense to disable threads, so I don't mind taking liberties there. [...] To summarize, there's two problems to be solved: 1) low-level corruption in the signal handlers as they record a new signal, such as in Py_AddPendingCalls This is purely theoretical, right? Has anyone ever observed this? I've never heard of it happening. If the compiler doesn't do much reordering (the CPU isn't an issue as this is only called in the main thread) then the most you might get is dropped calls. It's fairly safe the way signal handlers use it, but they'd work just as well (and easier to understand/verify) without the whole queue aspect; just setting some flags and resetting _Py_Ticker. 2) high-level wakeup race: check for pending signals, have a signal come in, then call a blocking syscall/library (oblivious to the new signal). Right. That's the race which really does happen, and for which the current lame-y work-around is to use a short timeout. [...] Anyway, I would still like to discuss this on #python-dev Monday. Adam, in what time zone are you? (I'm PST.) Who else is interested? MST. Unfortunately I can't stay at work later than 5:30 or so which would be too early for you I believe. I could try again after 8pm, your 9pm. Would that work at all? Otherwise I'd rather try earlier in the day if that works at all for you. 5:30 am or 5:30 pm? Any time after 11 am MST (10 am PST) should be fine for me. (My previous email was a little naive about how late I get up.) I shouldn't be gone until around midnight MST. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python-Dev Digest, Vol 53, Issue 23
Josiah Carlson [EMAIL PROTECTED] wrote: On Dec 5, 2007 9:19 AM, Guido van Rossum [EMAIL PROTECTED] wrote: The asyncore and asynchat modules are in a difficult position when it comes to Python 3000. None of the core developers use it or particularly care about it (AFAIK), and the API has problems because it wasn't written to deal with bytes vs. unicode. E.g. in http://bugs.python.org/issue1067, Thomas suggests that these modules need to be rewritten to use bytes internally and have separate APIs to handle (unicode) text as desired, similar to the way file I/O was redesigned. Another alternative would be to make these modules deal strictly in bytes, but that would probably vastly reduce their usefulness (though I don't know -- as I said, I don't use them). I can look into it later this month, but for the short term, I'm a little squeezed for time (work, finishing school, etc.). I am a bit curious, however, I could have sworn that bytes were to become, essentially, the 2.x str type (without some methods). If that is the case, no changes should really be necessary, except for a few small changes to asynchat with regards to line terminators. Then again, I haven't really been keeping up in the p3k/pydev lists for the last 6-8 months, and only the subject line of this posting caught my eye (because I use asyncore/asynchat, and support users of asyncore/asynchat that contact me directly). You're exactly right about the (lack of) problems and the correct way to fix them. I've placed a patch in the bug tracker that takes that very approach. http://bugs.python.org/issue1563 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP Idea: Syntactic sugar for StopIteration.
* Phillip J. Eby [EMAIL PROTECTED] [2007-12-08 20:19:29 -0500]: At 07:55 PM 12/8/2007 -0500, Manuel Alejandro Cerón Estrada wrote: 2007/12/8, Greg Ewing [EMAIL PROTECTED]: I would put it the other way around -- the problem that 'yield break' is meant to solve is already solved by 'return'. So there's no need for change. I have been re-thinking the problem and this is true. The only exception would be empty generators, but they are rare. Note that: def g(): return yield And this isn't a generator at all, but is almost the same: def h(): return iter([]) -- mithrandi, i Ainil en-Balandor, a faer Ambar signature.asc Description: Digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
