Re: [Python-Dev] Removing the GIL (Me, not you!)
What do you think? I'm going to have to agree with Martin here, although I'm not sure I understand what you're saying entirely. Perhaps if you explained where the benefits of this approach come from, it would clear up what you're thinking. After a few days of thought, I'm starting to realize the difficulty of maintaining compatibility with existing C extensions after removing the GIL. The possible C-level side effects are very difficult to work around without kernel or hardware level transaction support. I see a couple of approaches that might work (though I probably haven't thought of everything). 1. Use message passing and transactions. Put every module into its own tasklet that ends up getting owned by one thread or another. Every call to an object that is owned by that module is put into a module wide message queue and delivered sequentially to its objects. All this does is serialize requests to objects implemented in C to slightly mitigate the need to lock. Then use transactions to protect any python object. You still have the problem of C side effects going unnoticed (IE Thread A executes function, object sets c-state in a certain way, Thread B calls the same function, changes all the C-state, A reacts to return value that no longer reflects on the actual state). So, this doesn't actually work, but its close since python objects will remain consistent w/transactions and conflicting C-code won't execute simultaneously. 2. Do it perl style. Perl just spawns off multiple interpreters and doesn't share state between them. This would require cleaning up what state belongs where, and probably providing some global state lock free. For instance, all the numbers, letters, and None are read only, so we could probably work out a way to share them between threads. In fact, any python global could be read only until it is written to. Then it belongs to the thread that wrote to it and is updated in the other threads via some sort of cache-coherency protocol. I haven't really wrapped my head around how C extensions would play with this yet, but essentially code operating in different threads would be operating on different copies of the modules. That seems fair to me. 3. Come up with an elegant way of handling multiple python processes. Of course, this has some downsides. I don't really want to pickle python objects around if I decide they need to be in another address space, which I would probably occasionally need to do if I abstracted away the fact that a bunch of interpreters had been spawned off. 4. Remove the GIL, use transactions for python objects, and adapt all C-extensions to be thread safe. Woo. I'll keep kicking around ideas for a while; hopefully they'll become more refined as I explore the code more. Justin PS. A good paper on how hardware transactional memory could help us out: http://www-faculty.cs.uiuc.edu/~zilles/papers/python_htm.dls2006.pdf A few of you have probably read this already. Martin is even acknowledged, but it was news to me! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL certs
My understanding is that the client tells the server which hostname it wants to use; the server should then pass down that information. That's how virtual hosting works in the first place. The only difference with SSL is that the hostname must have a unique IP address, so that when the client does a reverse DNS to validate the IP address presented by the SSL certificate, it all comes together correctly. Unfortunately, it does not quite work that way. The client tells the server what hostname to use only *after* the SSL connection has been established, and certificates being exchanged (in the Host: header). So the Host: header cannot be used for selecting what certificate to present to the client. *That* is the reason why people typically assume they have to have different IP addresses for different SSL hosts: certificate selection must be done based on IP address (which is already known before the SSL handshaking starts). There is no need for the client to do a reverse name lookup, and indeed, the client should *not* do a reverse DNS lookup to check the server's identity. Instead, it should check the host name it wants to talk to against the certificate. However, there is an alternative to using multiple IP addresses: one could also use multiple subject alternative names, and create a certificate that lists them all. There are, of course, wildcard certs; I don't understand how those work. The same way: the client does *not* perform a reverse name lookup. Instead, it just matches the hostname against the name in the certificate; if the certificate is for *.python.org (say) and the client wants to talk to pypi.python.org, it matches, and hostname verification passes. It would also pass if the client wanted to talk to cheeseshop.python.org, or wiki.python.org (which all have the same IP address). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Phillip J. Eby wrote: It's not just caches and counters. It's also every built-in type structure, builtin module, builtin function... any Python object that's a built-in, period. That includes things like None, True, and False. Caches would include such things as the pre-created integers -100 through 255, the 1-byte character strings for chr(0)-chr(255), and the interned strings cache, to name a few. Most of these things I've mentioned are truly global, and not specific to an individual interpreter. Pardon my ignorance but why does Python do reference counting for truly global and static objects like None, True, False, small and cached integers, sys and other builtins? If I understand it correctly these objects are never garbaged collected (at least they shouldn't) until the interpreter exits. Wouldn't it decrease the overhead and increase speed when Py_INCREF and Py_DECREF are NOOPs for static and immutable objects? Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
* Christian Heimes wrote: Pardon my ignorance but why does Python do reference counting for truly global and static objects like None, True, False, small and cached integers, sys and other builtins? If I understand it correctly these objects are never garbaged collected (at least they shouldn't) until the interpreter exits. Wouldn't it decrease the overhead and increase speed when Py_INCREF and Py_DECREF are NOOPs for static and immutable objects? The check what kind of object you have takes time, too. Right now, just counting up or down is most likely faster than that check on every refcount operation. nd ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
On 13/09/2007, David Bolen [EMAIL PROTECTED] wrote: Mark Hammond [EMAIL PROTECTED] writes: It might be possible to try and use build_ssl.py to locate the openssl directory, but this will still require that someone building it has Python built from source - I'm fairly sure that someone installing a Python binary will not have build_ssl.py, nor are they likely to have a suitable openssl directory or installation just hanging around either. Yep - even if a Windows user has an appropriate development environment in general (and can build most standalone extensions with just a binary Python install), as you say the odds are pretty small they'd have an OpenSSL source tree around, with libraries built. It is possible to build extensions on Windows using the mingw gcc toolchain. Users doing this may well have some or all of the gnuwin32 (http://gnuwin32.sf.net) utilities installed. Gnuwin32 includes openssl (both headers, link libraries, and DLLs). It seems to me a perfectly reasonable option for someone wanting to build the SSL extension to grab mingw and gnuwin32 openssl. I tried building with this config last night, but didn't have the time to deal with hacking the setup.py - I see someone else has covered this. I'll have another go with the new version when I get a chance. At the same time, I suspect that only a small percentage of Windows users will want to rebuild the extension - rather they'll just want a binary installer, something not uncommon to be published for Windows users of many extension modules. So that pushes the problem upstream a bit where having a Python development tree might be more common or familiar. Agreed. I assume Windows binary builds will be published, so it's only early adopters, or people who want to work with their own builds for some other reason, who might care. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
On 13/09/2007, David Bolen [EMAIL PROTECTED] wrote: Bill Janssen [EMAIL PROTECTED] writes: In that case, I think your idea of just hard-coding a path is probably the right thing to do. I'll add a note that this is how you need to do it if you are going to try python setup.py build. Presumably the binary then built with python setup.py bdist will install on a Windows machine regardless of where OpenSSL is installed? Yes (though typically bdist_wininst for the Windows installer), but perhaps not for the reason you think. I think where there's probably a small disconnect here is that, there really isn't an OpenSSL installed on the end user's machine. Well, there could be, but Python isn't using it. The OpenSSL library is statically linked as part of the _ssl.pyd module, as it will be with your _ssl2.pyd module. (That's also why there is no OpenSSL to find in your setup even with Python installed - at least not any libraries you can use). That's not 100% true, is it? If I use mingw and Gnuwin32 openssl, I believe the default is a dynamic link of openssl (it depends on the import library used, and gnuwin32 supplies dynamic libs by default). So the openssl DLLs need to be on the user's PATH for the extension module to work. In other words, both the standard and your extension module on Windows bring along their own OpenSSL. For the extension, you may need to (1) document that the user needs to have the openssl DLLs on their PATH and possibly (1a) supply a zipfile with the necessary DLLs as a supplemental download, or (2) arrange for the openssl DLLs to be included in the extension installer, and installed alongside the .pyd file. Alternatively, it *may* be possible with setup.py magic to force a static openssl link (but would that need hard coding for the gnuwin32 naming conventions?) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Thu, Sep 13, 2007 at 12:19:21PM +0200, André Malo wrote: Pardon my ignorance but why does Python do reference counting for truly global and static objects like None, True, False, small and cached integers, sys and other builtins? If I understand it correctly these objects are never garbaged collected (at least they shouldn't) until the interpreter exits. Wouldn't it decrease the overhead and increase speed when Py_INCREF and Py_DECREF are NOOPs for static and immutable objects? The check what kind of object you have takes time, too. Right now, just counting up or down is most likely faster than that check on every refcount operation. To put it another way, would it actually matter if the reference counts for such objects became hopelessly wrong due to non-atomic adjustments? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
On 9/13/07, Paul Moore [EMAIL PROTECTED] wrote: On 13/09/2007, David Bolen [EMAIL PROTECTED] wrote: I think where there's probably a small disconnect here is that, there really isn't an OpenSSL installed on the end user's machine. Well, there could be, but Python isn't using it. The OpenSSL library is statically linked as part of the _ssl.pyd module, as it will be with your _ssl2.pyd module. (That's also why there is no OpenSSL to find in your setup even with Python installed - at least not any libraries you can use). That's not 100% true, is it? If I use mingw and Gnuwin32 openssl, I believe the default is a dynamic link of openssl (it depends on the import library used, and gnuwin32 supplies dynamic libs by default). So the openssl DLLs need to be on the user's PATH for the extension module to work. That's a fair point - my comments are all related to the standard Python distribution and building extensions with the VS.NET compiler (including the binary installer I had built for Bill). For the extension, you may need to (1) document that the user needs to have the openssl DLLs on their PATH and possibly (1a) supply a zipfile with the necessary DLLs as a supplemental download, or (2) arrange for the openssl DLLs to be included in the extension installer, and installed alongside the .pyd file. If we're talking about the construction of binary Windows installers, I'd just suggest that they get built as the built-in SSL module does, including the static linking with a pure Windows OpenSSL build, which is a bit simpler for the typical end user and has no other external requirements. Of course, that certainly doesn't stop a person who has already set up their system for using mingw for extensions from doing their own compilation, although it does raise a question as to whether the setup.py would need some further adjustments to cover that case most cleanly. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
To put it another way, would it actually matter if the reference counts for such objects became hopelessly wrong due to non-atomic adjustments? If they drop to zero (which may happen due to non-atomic adjustments), Python will try to release the static memory, which will crash the malloc implementation. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Thu, Sep 13, 2007 at 01:15:39PM +0200, Martin v. Löwis wrote: To put it another way, would it actually matter if the reference counts for such objects became hopelessly wrong due to non-atomic adjustments? If they drop to zero (which may happen due to non-atomic adjustments), Python will try to release the static memory, which will crash the malloc implementation. That could be avoided by a flag on the object which is checked in free(). I'm just suggesting it as an alternative as it sounds like it might be more efficient than either locking or avoiding having reference counts on these objects (especially if the reference count is initialised to MAX_INT/2 or whatever). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
On 13/09/2007, David Bolen [EMAIL PROTECTED] wrote: That's a fair point - my comments are all related to the standard Python distribution and building extensions with the VS.NET compiler (including the binary installer I had built for Bill). [...] If we're talking about the construction of binary Windows installers, I'd just suggest that they get built as the built-in SSL module does, including the static linking with a pure Windows OpenSSL build, which is a bit simpler for the typical end user and has no other external requirements. OK. Building with mingw is a bit of a hobby horse of mine, as the requirement for the (expensive) VS.NET compiler forces many users to rely on binary builds. I know other alternatives can be made to work, but it's often too much pain to bother. I'd much rather extensions which *can* be built using free tools, support actually doing so out of the box. Of course, that certainly doesn't stop a person who has already set up their system for using mingw for extensions from doing their own compilation, although it does raise a question as to whether the setup.py would need some further adjustments to cover that case most cleanly. And that's my point. I'd rather work to ensure that mingw works out of the box, than leave things requiring VS.NET for a clean build. It's not relevant here, but I've certainly been in a situation with other extensions where I can't upgrade a Python install because the distributors of a particular extension haven't produced a build for the new version yet (*cough* mod_python *cough*) - and I live in fear of support for some extensions dying, as I'll then have to move away from them or stop upgrading Python. Anyway, philosophy aside, I'll try to make some time in the next few days to get a working setup.py for the SSL package using mingw. Hopefully, Bill will then integrate this and we'll have mingw as a supported option. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Jon To put it another way, would it actually matter if the reference Jon counts for such objects became hopelessly wrong due to non-atomic Jon adjustments? I believe this was suggested and tried by someone (within the last few years). It wasn't any benefit. The costs of special-casing outweighed the costs of uniform reference counting, not to mention the code got more complex. Or something like that. Anyway, it didn't work. Just thinking out loud here, what if ... we use atomic test-and-set to handle reference counting (with a lock for those CPU architectures where we haven't written the necessary assembler fragment), then implement a lock for each mutable type and another for global state (thread state, interpreter state, etc)? Might that be close enough to free threading to provide some benefits, but not so fine-grained that lock contention becomes a bottleneck? Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Thu, 2007-09-13 at 13:15 +0200, Martin v. Löwis wrote: To put it another way, would it actually matter if the reference counts for such objects became hopelessly wrong due to non-atomic adjustments? If they drop to zero (which may happen due to non-atomic adjustments), Python will try to release the static memory, which will crash the malloc implementation. More precisely, Python will call the deallocator appropriate for the object type. If that deallocator does nothing, the object continues to live. Such objects could also start out with a refcount of sys.maxint or so to ensure that calls to the no-op deallocator are unlikely. The part I don't understand is how Python would know which objects are global/static. Testing for such a thing sounds like something that would be slower than atomic incref/decref. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Sep 13, 2007, at 10:12 AM, Martin v. Löwis wrote: What do you think? I think what you are describing is the situation of today, except in a less-performant way. The kernel *already* implements such a synchronization server, except that all CPUs can act as such. You write Since we are guaranteeing that synchronized code is running on a single core, it is the equivalent of a lock at the cost of a context switch. This is precisely what a lock costs today: a context switch. Really? Wouldn't we save some memory allocation overhead (since in my design, the lock is a really just simple kernel instruction as opposed to a full blown object) thereby lowering lock overhead (and allowing us to go with finer-grained locks? Since we're using an asynch message queue for the synch-server, it sounds like a standard lock-free algorithm. Since the Python interpreter is synchronized all of the time, it would completely run on the synchronization server all of the time. As you identify, that single CPU might get overloaded, so your scheme would give no benefits (since Python code could never run in parallel), I think I neglected to mention that the locking would still need to be more fine grained - perhaps only do the context switch around refcounts (and the other places where the GIL is critical). If we can do this in a way that allows simple list comprehensions to run in parallel, that would be really helpful (like a truly parallel map function). and only disadvantages (since multiple Python interpreters today can run on multiple CPUs, but could not anymore under your scheme). Well, you could still run python code in parallel if you used multiple processes (each process having its own 'synchronization server'). Is that what you meant? On Sep 13, 2007, at 12:38 PM, Justin Tulloss wrote: What do you think? I'm going to have to agree with Martin here, although I'm not sure I understand what you're saying entirely. Perhaps if you explained where the benefits of this approach come from, it would clear up what you're thinking. Well, my interpretation of the current problem is that removing the GIL has not been productive because of problems with lock contention on multi-core machines. Naturally, we need to make the locking more fine-grained to resolve this. Hopefully we can do so in a way that does not increase the lock overhead (hence my suggestion for a lock free approach using an asynch queue and a core as dedicated server). If we can somehow guarantee all GC operations (which is why the GIL is needed in the first place) run on a single core, we get locking for free without actually having to have threads spinning. regards, Prateek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Since we are guaranteeing that synchronized code is running on a single core, it is the equivalent of a lock at the cost of a context switch. This is precisely what a lock costs today: a context switch. Really? Wouldn't we save some memory allocation overhead (since in my design, the lock is a really just simple kernel instruction as opposed to a full blown object) The GIL is a single variable, not larger than 50 Bytes or so. Locking it requires no memory at all in user-space, and might require 8 bytes or so per waiting thread in kernel-space. thereby lowering lock overhead Why do you think lock overhead is related to memory consumption? Since we're using an asynch message queue for the synch-server, it sounds like a standard lock-free algorithm. You lost me here. What are you trying to achieve? It's not the lock that people complain about, but that Python runs serially most of the time. I think I neglected to mention that the locking would still need to be more fine grained - perhaps only do the context switch around refcounts (and the other places where the GIL is critical). I think this is the point where I need to say good luck implementing it. Well, my interpretation of the current problem is that removing the GIL has not been productive because of problems with lock contention on multi-core machines. My guess is that this interpretation is wrong. It was reported that there was a slowdown by a factor of 2 in a single-threaded application. That can't be due to lock contention. If we can somehow guarantee all GC operations (which is why the GIL is needed in the first place) No, unless we disagree on what a GC operation is. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
In other words, both the standard and your extension module on Windows bring along their own OpenSSL. I see -- thanks. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
Anyway, philosophy aside, I'll try to make some time in the next few days to get a working setup.py for the SSL package using mingw. Hopefully, Bill will then integrate this and we'll have mingw as a supported option. I'll be happy to do that! Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Sep 13, 2007, at 9:25 PM, Martin v. Löwis wrote: Since we are guaranteeing that synchronized code is running on a single core, it is the equivalent of a lock at the cost of a context switch. This is precisely what a lock costs today: a context switch. Really? Wouldn't we save some memory allocation overhead (since in my design, the lock is a really just simple kernel instruction as opposed to a full blown object) The GIL is a single variable, not larger than 50 Bytes or so. Locking it requires no memory at all in user-space, and might require 8 bytes or so per waiting thread in kernel-space. thereby lowering lock overhead Why do you think lock overhead is related to memory consumption? Well, it can be one (or both) of two things - 1) memory consumption, 2) cost of acquiring and releasing the locks (which you said is the same as a context switch). Since we've also identified (according to GvR's post: http:// www.artima.com/weblogs/viewpost.jsp?thread=214235) that the slowdown was 2x in a single threaded application (which couldn't be due to lock contention), it must be due to lock overhead (unless the programming was otherwise faulty or there is something else about locks that I don't know about - Martin?). Hence I'm assuming that we need to reduce lock overhead. If acquiring and releasing locks (part of lock overhead) is a simple context switch (and I don't doubt you here), then the only remaining thing to optimize is memory operations related to lock objects. Since we're using an asynch message queue for the synch-server, it sounds like a standard lock-free algorithm. You lost me here. What are you trying to achieve? It's not the lock that people complain about, but that Python runs serially most of the time. http://en.wikipedia.org/wiki/Lock-free_and_wait- free_algorithms#The_lock-free_approach Specifically, i'm trying to achieve the approach using a deposit request. I think I neglected to mention that the locking would still need to be more fine grained - perhaps only do the context switch around refcounts (and the other places where the GIL is critical). I think this is the point where I need to say good luck implementing it. I don't mean to be unhelpful. Its just that this discussion started because people (not me - although I would definitely benefit) showed interest in removing the GIL. Well, my interpretation of the current problem is that removing the GIL has not been productive because of problems with lock contention on multi-core machines. My guess is that this interpretation is wrong. It was reported that there was a slowdown by a factor of 2 in a single-threaded application. That can't be due to lock contention. I agree with your point Martin (see my analysis above). Regarding lock contention: I'm guessing that if single threaded applications are so badly affected, then the cumulative overhead on multithreaded applications will be even worse. So we need to reduce the overhead. But then since all Python code runs under the GIL - which is a pretty coarse lock, we have to make the new locking more fine-grained (which is what I think the original patch by Greg Stein did). I'm also guessing that if you do that then for each refcount you're going to have to acquire a lock... which happens *very* frequently (and I think by your earlier responses you concur). So that means anytime multiple threads try to access the same object, they will need to do an incref/decref. e.g. If you access a global variable inside a for- loop from multiple threads. If we can somehow guarantee all GC operations (which is why the GIL is needed in the first place) No, unless we disagree on what a GC operation is. Ok. Other people know more about the specifics of the GIL than I do. However, the main issue with removing the GIL seems to be the reference counting algorithm. That is what I was alluding to. In any case, it is not relevant for the rest of the discussion. regards, Prateek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
http://www.artima.com/weblogs/viewpost.jsp?thread=214235) that the slowdown was 2x in a single threaded application (which couldn't be due to lock contention), it must be due to lock overhead (unless the programming was otherwise faulty or there is something else about locks that I don't know about - Martin?). Hence I'm assuming that we need to reduce lock overhead. If acquiring and releasing locks (part of lock overhead) is a simple context switch (and I don't doubt you here), then the only remaining thing to optimize is memory operations related to lock objects. I think you are putting too many assumptions on top of each other. It might also have been that the locks in the slow implementation were too fine-grained, and that some performance could have been regained by making them coarser again. You lost me here. What are you trying to achieve? It's not the lock that people complain about, but that Python runs serially most of the time. http://en.wikipedia.org/wiki/Lock-free_and_wait-free_algorithms#The_lock-free_approach The asynchronous model assumes that the sender can continue to process data without needing a reply. This is not true for the Python threading model: if the thread needs access to some data structure, it really needs to wait for the result of that access, because that's the semantics of the operations. Specifically, i'm trying to achieve the approach using a deposit request. For that to work, you need to produce a list of requests that can be processed asynchronously. I can't see any in the Python interpreter. I'm also guessing that if you do that then for each refcount you're going to have to acquire a lock... which happens *very* frequently (and I think by your earlier responses you concur). In that it occurs frequently - not in that you have to acquire a lock to modify the refcount. You don't. Ok. Other people know more about the specifics of the GIL than I do. However, the main issue with removing the GIL seems to be the reference counting algorithm. It isn't. Reference counting could be done easily without the GIL. It's rather the container objects, and the global variables, that need protection. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/13/07, Hrvoje Nikšić [EMAIL PROTECTED] wrote: On Thu, 2007-09-13 at 13:15 +0200, Martin v. Löwis wrote: To put it another way, would it actually matter if the reference counts for such objects became hopelessly wrong due to non-atomic adjustments? If they drop to zero (which may happen due to non-atomic adjustments), Python will try to release the static memory, which will crash the malloc implementation. More precisely, Python will call the deallocator appropriate for the object type. If that deallocator does nothing, the object continues to live. Such objects could also start out with a refcount of sys.maxint or so to ensure that calls to the no-op deallocator are unlikely. The part I don't understand is how Python would know which objects are global/static. Testing for such a thing sounds like something that would be slower than atomic incref/decref. I've explained my experiments here: http://www.artima.com/forums/flat.jsp?forum=106thread=214235start=30msRange=15#279978 Basically though, atomic incref/decref won't work. Once you've got two threads modifying the same location the costs skyrocket. Even without being properly atomic you'll get the same slowdown on x86 (who's cache coherency is fairly strict.) The only two options are: A) Don't modify an object on every incref/decref. Deletion must be delayed. This lets you share (thread-safe) objects. B) Don't share *any* objects. This is a process model (even if they're lightweight like erlang). For the near future, it's much easier to do this using real processes though. Threading is much more powerful, but it remains to be proven that it can be done efficiently. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL certs
However, there is an alternative to using multiple IP addresses: one could also use multiple subject alternative names, and create a certificate that lists them all. Unfortunately, much of the client code that does the hostname verification is wrapped up in gullible Web browsers or Java HTTPS libraries that swallowed RFC 2818 whole, and not easily accessible by applications. Does any of it recognize and accept subject alternative name? It's possible to at least override the default Java client-side hostname verification handling in a new application. And Python is lucky; because there was no client-side hostname verification possible, RFC 2818 hasn't been plastered into the Python standard library :-). Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL certs
However, there is an alternative to using multiple IP addresses: one could also use multiple subject alternative names, and create a certificate that lists them all. Unfortunately, much of the client code that does the hostname verification is wrapped up in gullible Web browsers or Java HTTPS libraries that swallowed RFC 2818 whole, and not easily accessible by applications. Does any of it recognize and accept subject alternative name? Works fine with Firefox and MSIE. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/13/07, Justin Tulloss [EMAIL PROTECTED] wrote: 1. Use message passing and transactions. [...] 2. Do it perl style. [...] 3. Come up with an elegant way of handling multiple python processes. [...] 4. Remove the GIL, use transactions for python objects, [...] The SpiderMonkey JavaScript engine takes a very different approach, described here: http://developer.mozilla.org/en/docs/SpiderMonkey_Internals:_Thread_Safety The SpiderMonkey C API threading model should sound familiar: C code can assume that simple operations, like dictionary lookups, are atomic and thread-safe. C code must explicitly JS_SuspendRequest() before doing blocking I/O or number-crunching (just like Py_BEGIN_ALLOW_THREADS). The main difference is that SpiderMonkey's requests are not mutually exclusive, the way the GIL is. SpiderMonkey does fine-grained locking for mutable objects to avoid race conditions. The clever bit is that SpiderMonkey's per-object locking does *not* require a context switch or even an atomic instruction, in the usual case where an object is *not* shared among threads. (Programs that embed SpiderMonkey therefore run faster if they manage to ensure that threads share relatively few mutable objects. JavaScript doesn't have modules.) Suppose Python went this route. There would still have to be a stop-the-world global lock, because the cycle collector won't work if other threads are going about changing pointers. (SpiderMonkey's GC does the same thing.) Retaining such a lock has another advantage: this change could be completely backward-compatible to extensions. Just use this global lock as the GIL when entering a non-thread-safe extension (all existing extensions would be considered non-thread-safe). This means non-thread-safe extensions would be hoggish (but not much worse than they are already!). Making an existing extension thread-safe would require some thought, but it wouldn't be terribly hard. In the simplest cases, the extension writer could just add a flag to the type saying ok, I'm thread-safe. Refcounting is another major issue. SpiderMonkey uses GC instead. CPython would need to do atomic increfs/decrefs. (Deferred refcounting could mitigate the cost.) The main drawback (aside from the amount of work) is the patent. SpiderMonkey's license grants a worldwide, royalty-free license, but not under the Python license. I think this could be wrangled, if the technical approach looks worthwhile. -j ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] base64 -- should b64encode introduce line breaks?
I see that base64.b64encode and base64.standard_b64encode no longer introduce line breaks into the output strings, as base64.encodestring does. Shouldn't there be an option on one of them to do this? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] base64 -- should b64encode introduce line breaks?
I see that base64.b64encode and base64.standard_b64encode no longer introduce line breaks into the output strings, as base64.encodestring does. Shouldn't there be an option on one of them to do this? See: http://mail.python.org/pipermail/python-bugs-list/2001-October/007856.html section 2.1 of http://www.faqs.org/rfcs/rfc3548.html Perhaps adding MIME_b64encode() and PEM_b64encode() routines? Or just an optional parameter to standard_b64encode, called max_line_length, defaulting to 0, meaning no max? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python tickets summary
2007/9/10, Facundo Batista [EMAIL PROTECTED]: I modified my tool, whichs makes a summary of all the Python tickets (I moved the source where the info is taken from SF to our Roundup). In result, the summary is now, again, updated daily: Taking an idea from Jeff Rush, now there're separate listings in function of the keyword of the ticket. This way, you can see only the Py3k tickets, or the patchs, etc. All the listings are accesible from the same pages, start here: http://www.taniquetil.com.ar/facundo/py_tickets.html (remember to refresh) Any idea to improve these pages is welcomed. Regards, -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Tracker-discuss] Python tickets summary
On 9/13/07, Facundo Batista [EMAIL PROTECTED] wrote: http://www.taniquetil.com.ar/facundo/py_tickets.html It looks like the column Opened by contains information for Last update by and vice versa. At least, that is the case with issue 1159. Thanks, Raghu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/13/07, Justin Tulloss [EMAIL PROTECTED] wrote: On 9/13/07, Adam Olsen [EMAIL PROTECTED] wrote: Basically though, atomic incref/decref won't work. Once you've got two threads modifying the same location the costs skyrocket. Even without being properly atomic you'll get the same slowdown on x86 (who's cache coherency is fairly strict.) I'm a bit skeptical of the actual costs of atomic incref. For there to be contention, you would need to have to be modifying the same memory location at the exact same time. That seems unlikely to ever happen. We can't bank on it never happening, but an occasionally expensive operation is ok. After all, it's occasional. That was my initial expectation too. However, the incref *is* a modification. It's not simply an issue of the exact same time, but anything that causes the cache entries to bounce back and forth and delay the rest of the pipeline. If you have a simple loop like for i in range(count): 1.0+n, then the 1.0 literal will get shared between threads, and the refcount will get hammered. Is it reasonable to expect that much sharing? I think it is. Literals are an obvious example, but there's also configuration data passed between threads. Pystone seems to have enough sharing to kill performance. And after all, isn't sharing the whole point (even in the definition) of threads? -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
On 13/09/2007, Bill Janssen [EMAIL PROTECTED] wrote: Anyway, philosophy aside, I'll try to make some time in the next few days to get a working setup.py for the SSL package using mingw. Hopefully, Bill will then integrate this and we'll have mingw as a supported option. I'll be happy to do that! OK, the following patch to setup.py works for mingw32. You need to set 2 variables - 1. The location where you installed gnuwin32 2. Whether you want a static or dynamic build I've checked both versions on Python 2.5.1 and they pass all tests. Static build is 670k, dynamic is 26k (but depends on the openssl DLLs libssl32.dll and libeay32.dll). Ideally, these should be settable via command line options or something. Also, it would be nice to detect the use of MSVC and do something equivalent (but presumably somewhat different), but I don't know how to detect the type of compiler the user has selected :-( Anyway, I hope it's useful. If nothing else, it offers a way for people to build the module with free software on Windows. I could build some Windows installers if you want, but I'd need to download and install some extra versions of Python, so you'd have to tell me which you want doing (and I can't offer to commit to doing this on a regular basis...) Paul. mingw.patch Description: Binary data ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/13/07, Jason Orendorff [EMAIL PROTECTED] wrote: On 9/13/07, Justin Tulloss [EMAIL PROTECTED] wrote: 1. Use message passing and transactions. [...] 2. Do it perl style. [...] 3. Come up with an elegant way of handling multiple python processes. [...] 4. Remove the GIL, use transactions for python objects, [...] The SpiderMonkey JavaScript engine takes a very different approach, described here: http://developer.mozilla.org/en/docs/SpiderMonkey_Internals:_Thread_Safety This is basically the same as what perl does, as far as I understand it. There are differences, but they're not that substantial. It's basically the idea of keeping all state separate and treating global access as a special case. I think this is a pretty solid approach, since globals shouldn't be accessed that often. What we would want to do differently is make sure that read-only globals can be cheaply accessed from any thread. Otherwise we lose the performance benefit of having them in the first place. Refcounting is another major issue. SpiderMonkey uses GC instead. CPython would need to do atomic increfs/decrefs. (Deferred refcounting could mitigate the cost.) This is definitely something to think about. I don't really have an answer straight off, but there are several things we could try. The main drawback (aside from the amount of work) is the patent. SpiderMonkey's license grants a worldwide, royalty-free license, but not under the Python license. I think this could be wrangled, if the technical approach looks worthwhile. I'm not sure this is an issue. It's not like we would be using the code, just the patented algorithm. Any code we wrote to implement the algorithm would of course be covered under the python license. I'm not a legal guy though. Justin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
I could build some Windows installers if you want, but I'd need to download and install some extra versions of Python, so you'd have to tell me which you want doing (and I can't offer to commit to doing this on a regular basis...) Thanks, but let's wait till this settles down a bit (say, a week passes without me saying anything about it :-). Then I'll definitely want both VS and mingw versions to upload to the Cheeseshop. But it's not quite ready yet. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
On 13/09/2007, Bill Janssen [EMAIL PROTECTED] wrote: I could build some Windows installers if you want, but I'd need to download and install some extra versions of Python, so you'd have to tell me which you want doing (and I can't offer to commit to doing this on a regular basis...) Thanks, but let's wait till this settles down a bit (say, a week passes without me saying anything about it :-). Then I'll definitely want both VS and mingw versions to upload to the Cheeseshop. But it's not quite ready yet. OK, ignore my other message then (except as an indication that I can build them when you're ready :-)). You don't need VS and mingw binary installers, though - the mingw ones will work for any Python (ignoring specialised custom builds, and anyone doing one of them is probably capable of building the ssl module!). Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/13/07, Adam Olsen [EMAIL PROTECTED] wrote: Basically though, atomic incref/decref won't work. Once you've got two threads modifying the same location the costs skyrocket. Even without being properly atomic you'll get the same slowdown on x86 (who's cache coherency is fairly strict.) I'm a bit skeptical of the actual costs of atomic incref. For there to be contention, you would need to have to be modifying the same memory location at the exact same time. That seems unlikely to ever happen. We can't bank on it never happening, but an occasionally expensive operation is ok. After all, it's occasional. Justin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Martin v. Löwis wrote: Now we are getting into details: you do NOT have to lock an object to modify its reference count. An atomic increment/decrement operation is enough. I stand corrected. But if it were as simple as that, I think it would have been done by now. I got the impression that this had already been tried, and it was still too slow. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
You don't need VS and mingw binary installers, though - the mingw ones will work for any Python (ignoring specialised custom builds, and anyone doing one of them is probably capable of building the ssl module!). Why I appreciate your points about building the extension with free tools, wouldn't it be prudent to release binaries using the same compiler as Python itself, assuming that option is available? If I read this thread correctly, a mingw build will rely on an openssl DLL being available or installed, which would seem to be less desirable than the way it builds with the openssl Python itself builds with. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Hrvoje More precisely, Python will call the deallocator appropriate for Hrvoje the object type. If that deallocator does nothing, the object Hrvoje continues to live. Such objects could also start out with a Hrvoje refcount of sys.maxint or so to ensure that calls to the no-op Hrvoje deallocator are unlikely. Maybe sys.maxint/2? You'd hate for the first incref to invoke the deallocator even if it was a no-op. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Thu, Sep 13, 2007 at 06:38:05PM -0500, [EMAIL PROTECTED] wrote: Hrvoje More precisely, Python will call the deallocator appropriate for Hrvoje the object type. If that deallocator does nothing, the object Hrvoje continues to live. Such objects could also start out with a Hrvoje refcount of sys.maxint or so to ensure that calls to the no-op Hrvoje deallocator are unlikely. Maybe sys.maxint/2? You'd hate for the first incref to invoke the deallocator even if it was a no-op. I do believe I already suggested that ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Jon Ribbens wrote: To put it another way, would it actually matter if the reference counts for such objects became hopelessly wrong due to non-atomic adjustments? Again, it would cost time to check whether you could get away with doing non-atomic refcounting. If you're thinking that no check would be needed because only things like True, False and None would be shared between threads, that's quite wrong. If the threads are to communicate at all, they need to share some kind of data somewhere. Also keep in mind that there is one case of wrong refcounting that would be distastrous, which is the case where the refcount becomes zero prematurely. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
[EMAIL PROTECTED] wrote: what if ... we use atomic test-and-set to handle reference counting (with a lock for those CPU architectures where we haven't written the necessary assembler fragment), then implement a lock for each mutable type and another for global state (thread state, interpreter state, etc)? Could be worth a try. A first step might be to just implement the atomic refcounting, and run that single-threaded to see if it has terribly bad effects on performance. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Prateek Sureka wrote: Naturally, we need to make the locking more fine-grained to resolve this. Hopefully we can do so in a way that does not increase the lock overhead (hence my suggestion for a lock free approach using an asynch queue and a core as dedicated server). What you don't seem to see is that this would have no less overhead, and probably a lot *more*, than a mutex or other standard synchronisation mechanism. Certainly a lot more than an atomic instruction for the incref/decref. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Jason Orendorff wrote: The clever bit is that SpiderMonkey's per-object locking does *not* require a context switch or even an atomic instruction, in the usual case where an object is *not* shared among threads. How does it tell whether an object is shared between threads? That sounds like the really clever bit to me. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Pardon me for talking with no experience in such matters, but... Okay, incrementing a reference counter is atomic, therefore the cheapest possible operation. Is it possible to keep reference counting atomic in a multi-thread model? Could you do the following... let's consider two threads, A and B. Each time an object is created, a reference count is created in both A and B. Let's suppose A has a real reference and B has no reference really. Couldn't the GC check two reference registers for a reference count? The object would then be cleaned up only if both registers were 0. To exploit multiple CPUs, you could have two persistent Python processes on each CPU with its own mini-GIL. Object creation would then involve a call to each process to create the reference and GC would involve checking each process to see what their count is. However, it would mean that within each process, threads could create additional references or remove references in an atomic way. In a single-CPU system, this would be the same cost as currently, since I think that situation would devolve to having just one place to check for references. This seems to mean that it is the case that it would be no more expensive for a single-CPU system. In a two-CPU system, I'm no expertise on the actual call overheads of object creation and garbage collection, but logically it would double the effort of object creation and destruction (all such operations now need to occur on both processes) but would keep reference increments and decrements atomic. Once again, I'm really sorry if I'm completely off-base since I have never done any actual coding in this area, but I thought I'd make the suggestion just in case it happened to have relevance. Thanks, -Tennessee ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/13/07, Greg Ewing [EMAIL PROTECTED] wrote: Jason Orendorff wrote: The clever bit is that SpiderMonkey's per-object locking does *not* require a context switch or even an atomic instruction, in the usual case where an object is *not* shared among threads. How does it tell whether an object is shared between threads? That sounds like the really clever bit to me. If you look at the article, they have a code sample. Basically a global is owned by the first thread that touches it. That thread can do whatever it wants with that global. If another thread wants to touch the global, it locks everything to do so. This is a pretty good idea except that in Python there are so many globals that all threads benefit from having access to. Luckily, except for their reference counts, they're mostly read-only. Therefore, if we can work out this reference count, we can probably use a similar concept. Justin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
