Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Gregory P. Smith [EMAIL PROTECTED] writes: On Mon, Sep 19, 2005 at 09:12:05PM +0100, Michael Hudson wrote: Martin Blais [EMAIL PROTECTED] writes: http://www.gotw.ca/publications/concurrency-ddj.htm The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software Herb Sutter March 2005 I was disappointed that that article (hey, it was the only issue of ddj I've ever actually bought! :) didn't consider any concurrency models other than shared memory threading. Beware. Multi-core and/or multi-threaded cpus are the only thing the high end CPU manufacturers are able to produce today that they can still claim to be faster. There is a HUGE incentive for them to create demand for their design lest it become irrelevant and they be forced to sell only low-margin commodity single core hardware. This means we'll see a ton of papers and people paid or coerced into suggesting that this is the best thing since time sliced bread. I'd imagine that the more interesting/easy to deal with concurrency models can and will be implemented atop shared memory threads. At least, for the reasons you give, I hope so :) Cheers, mwh -- ZAPHOD: OK, so ten out of ten for style, but minus several million for good thinking, eh? -- The Hitch-Hikers Guide to the Galaxy, Episode 2 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On 9/21/05, Josiah Carlson [EMAIL PROTECTED] wrote: Antoine Pitrou [EMAIL PROTECTED] wrote: The best way to make people stop complaining about the GIL and start using process-based multiprogramming is to provide solid, standardized support for process-based multiprogramming. 100% agreed. I needed a portable way to know if a program was already running (and then to send it a simple command), the only simple solution I found was to listen on a local TCP socket and store the port number in a file in a known location. Not very elegant IMO. On *nix, use a unix domain socket (location in the filesystem which acts as a listening socket). On Windows, you can use cTypes, pywin32, etc., to create a global mutex and/or COM interface. Yes, but that's the point - having a cross-platform solution available out of the box (where the box may be the stdlib, or it may be third party). The problems I see are: 1. Encapsulating a suitable core feature set in a nice API 2. Making it cross-platform With something covering both of these, a case could be made for including it in the stdlib, to support/encourage multiprocess style coding in place of multithreading. I use threading quite a lot, often to spawn multiple workers all doing the same thing on different targets. A simple thread pool plus threading.Queue is perfect for this. A package which provided a similar abstraction using worker processes would be fine for me. Of course, *I* want it for Windows, and most multiprocess stuff tends to be Unix specific (where multiprocessing is natural and fast). And I'm happy enough with threads (I'm scaling up the number of blocking calls I'm waiting on - which already release the GIL - and so I don't care too much about SMP per se). This is getting offtopic, though, and probably more suitable for c.l.p, where design issues of a multiprocessing module should be thrashed out before coming back to python-dev with a proposal for the stdlib. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Hi, On *nix, use a unix domain socket (location in the filesystem which acts as a listening socket). On Windows, you can use cTypes, pywin32, etc., to create a global mutex and/or COM interface. Ok, but for a very simple and crude need like mine (the application code using this IPC means is not more than 20 lines long, and it has no performance requirement), it would be useful to have an abstraction in the standard library. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] GIL, Python 3, and MP vs. UP
Ok. While windows already prefers threads, while linux-2.6 improves thread support and speed, while process startup expensive on time and memory, while we ought to dublicate our data and/or use obscure shared memory, while it is much simpler to make threaded program with care just about locks than multiprocessing with inventing interchange protocol (which abuses both CPU - sender and reciever), MULTIPROCESSING RULES!!! And on Opterons: you will win with multiprocessing ONLY if OS will bind process and processor/memory place pair. That would be difficult. Guido van Rossum wrote: That's an understatement. I expect that *most* problems (even most problems that we will be programming 10-20 years from now) get little benefit out of MP. They are allready here. Servers they are. It is so simple to write application server in Python. It is so difficult to make it scallable in CPython. Hardware is cheap, development time is expensive. To make scallable CPython-backed server one needs too much time. And he goes to Java, .NET and others (and rare to Jython, IronPython (in the future)). CPython will not be wide popular without real multithreading. --I want my Apache, written in Python ! :-) --I want my MySQL, written in Python ! ;-) It is a joke. But every joke has a peace of truth (as russian sayes). (Excuse my English) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On Thu, Sep 22, 2005 at 10:27:10AM +0400, Sokolov Yura wrote: MULTIPROCESSING RULES!!! Everything in programming is about divide and conquer. Separate and control. Modules. Classes. Namespaces. And now that multithreading with shared memory. That's an evil idea, it causes a lot of troubles, it requires locks and semaphores, and still leads to problems. Shared memory is evil. Multithreading is harmful. Hardware is cheap, development time is expensive. Yes. Do not spend developer's time finding bugs in locks and semaphores, fixing race conditions, etc. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
At 10:27 AM 9/22/2005 +0400, Sokolov Yura wrote: It is so simple to write application server in Python. It is so difficult to make it scallable in CPython. It seems you've never heard of fork(), which works just fine to scale Python processes on multiprocessor boxes. I've actually done this, and it didn't take a lot of time to implement, so I don't know what you're talking about here. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Phillip J. Eby wrote: It is so simple to write application server in Python. It is so difficult to make it scallable in CPython. It seems you've never heard of fork(), which works just fine to scale Python processes on multiprocessor boxes. there's a version of fork hidden somewhere in CPython that solves all interprocess communication and synchronization issues? cool. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
At 08:42 PM 9/22/2005 +0200, Fredrik Lundh wrote: Phillip J. Eby wrote: At 10:27 AM 9/22/2005 +0400, Sokolov Yura wrote: It is so simple to write application server in Python. It is so difficult to make it scallable in CPython. It seems you've never heard of fork(), which works just fine to scale Python processes on multiprocessor boxes. there's a version of fork hidden somewhere in CPython that solves all interprocess communication and synchronization issues? cool. Yep, it's called fork stateless application servers that keep their state in a relational database. 0.1 wink It works for an astonishingly wide assortment of business problems, anyway. :) Don't get me wrong, I'm not opposed to adding better IPC capabilities; a Linda-like IPC facility would be nice to have, too. I was just pointing out the fallacy in the original post (whose attribution you deleted for some reason), that *scalability* is the issue. The original poster was claiming (in effect) that IPC and synchronization are trivial and that the GIL is responsible for some mythical lack of scalability. Threads can't scale past one machine, no matter how many processors you put in it, so it's a red herring IMO. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Sokolov Jura writes: It is so simple to write application server in Python. It is so difficult to make it scallable in CPython. CPython will not be wide popular without real multithreading. He's right. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Bill Janssen [EMAIL PROTECTED] wrote in message news:05Sep22.141518pdt.58617@synergy1.parc.xerox.com... Sokolov Jura writes: It is so simple to write application server in Python. It is so difficult to make it scallable in CPython. CPython will not be wide popular without real multithreading. He's right. I think the post is at least somewhat wrong on both technical and topicalness levels. On a technical level, the second statement seems wrong as an absolute statement. Phillip Eby claims that *some* application servers are easily scaled even if others are not. I suspect this is more likely to be the truth. In any case, the statement conveys no new technical information while to me it has the form and emotional load of a flame. The third statement is an opinion about the indefinite future which is neither right nor wrong as it stands. To ever judge it so, we need an operational definition of 'wide popular' to guide measurement. By some standards, Python is already widely popular. By others, it never will be. Guido already said that he will consider a high-quality maintainable 'real threading' patch that proves his scepticism wrong and that is neutral to beneficial to most all users, so I see little point in further flogging the issue, at least on this forum. Terry J. Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On 9/21/05, Donovan Baarda [EMAIL PROTECTED] wrote: The reality is threads were invented as a low overhead way of easily implementing concurrent applications... ON A SINGLE PROCESSOR. Taking into account threading's limitations and objectives, Python's GIL is the best way to support threads. QOTF candidate! (I wonder if this thread could be summarized into a PEP we can use instead of future discussions rehashing the same issues?) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On 21 sep 2005, at 12.33, Donovan Baarda wrote: In the short term there will be various hacks to try and make the existing plethora of threading applications run better on multiple processors, but ultimately the overheads of shared memory will force serious multi-processing to use IPC channels. If you want serious MP, use processes, not threads. The problem here is that while Python offers some support for thread-based multiprogramming in its standard library: theaad, threading, Queue, etc., there doesn't seem to exist much support for process-based multiprogramming beyond the basics. How is the developer helped? with data sharing, communication, control over running processes, dealing out tasks to be handled, etc. The best way to make people stop complaining about the GIL and start using process-based multiprogramming is to provide solid, standardized support for process-based multiprogramming. //Simon ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
The best way to make people stop complaining about the GIL and start using process-based multiprogramming is to provide solid, standardized support for process-based multiprogramming. 100% agreed. I needed a portable way to know if a program was already running (and then to send it a simple command), the only simple solution I found was to listen on a local TCP socket and store the port number in a file in a known location. Not very elegant IMO. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
The best way to make people stop complaining about the GIL and start using process-based multiprogramming is to provide solid, standardized support for process-based multiprogramming. +100 Huge amounts of effort would have to be invested to remove the GIL for the benefit of threads. Not only would the effort be huge, the difficulty and complexity of writing extension modules would be increased. Regardless of the arguments about SMP systems and the GIL, Python should provide as much support for process-based multi-programming as it does for threading. How about sinking that same effort into better Python support for process-based multi-programming? All the categories that Simon suggested are great candidates for the targets of this effort. Are there any existing efforts that I don't know about? Jonathan LaCour http://cleverdevil.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Antoine Pitrou [EMAIL PROTECTED] wrote: The best way to make people stop complaining about the GIL and start using process-based multiprogramming is to provide solid, standardized support for process-based multiprogramming. 100% agreed. I needed a portable way to know if a program was already running (and then to send it a simple command), the only simple solution I found was to listen on a local TCP socket and store the port number in a file in a known location. Not very elegant IMO. On *nix, use a unix domain socket (location in the filesystem which acts as a listening socket). On Windows, you can use cTypes, pywin32, etc., to create a global mutex and/or COM interface. Alternatively, you can write your own cross-platform registry for services, have it running on your machines all the time, and never worry again. Or one can pick a port to use on all systems and not bother with the the file listing the port, ignore domain sockets, COM interfaces, and work-alikes. In terms of IPC, the application may determine which is most usable. If one had a service distributed across multiple machines, sockets are necessary. If one only needed local access, shared memory (perhaps via a memory mapped file) would be significantly faster. In the case of multiple machines with multiple processors, one could opt for shared memory locally but sockets globally, which could complicate the interface, result in non-uniform performance for transfers, which may then necessate complicating the application in order to gain the highest throughput (I've done this previously using MPI, it can be a PITA). - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
(Interestign to see the PyObjC folks disagree. :-) On 9/21/05, Ronald Oussoren [EMAIL PROTECTED] wrote: On 21-sep-2005, at 0:10, Bob Ippolito wrote: My use case for this isn't so much about threads, but plug-ins. Writing multiple Python-based plug-ins for an application is always a mess, because they share too much (sys.modules, sys.path, etc.). PyObjC would benefit greatly from this feature, because you can write Python-based plug-ins for any Cocoa app that supports plug-ins, even if they're otherwise unaware of Python's existence. There are workarounds, of course, with import hooks and similar hacks. I think that mod_python would also benefit from this, and probably other such systems. For PyObjC having multiple interpreters in a process would cause me severe headaches. Due to the nature of PyObjC it would be way too easy to accidently access objects from one interpreter in another interpreter (in a hypothetical universe where it would be easy to use multiple interpreters). And lets not get started about the GIL, I don't think its accidental that PyGILState_Ensure only works with one interpreter. Actually Python itself has a hard time keeping multiple interpreters truly separate. Also of course there are some shared resources maintained by the operating system: current directory, open file descriptors, signal settings, child processes, that sort of thing. If we were to completely drop this feature, we could make built-in classes modifyable. A system like Java's classloader would be helpfull, where the classloader of a class is used to load the classes used by that class. I have no idea if this can be adapted to python at all. A strict coding style seems to work for now. You can do something like this using the restricted execution support, which works by setting the __builtins__ name in a dict where you exec code, and overriding __import__ in that __builtins__ dict. (I can't explain it too well in one paragraph, just go look up the rexec.py source code.) It's not great for guaranteeing there's absolutely no escape possible from the sandbox, but it works well enough to make accidental resource sharing a non-issue (apart from the OS shared resources and the built-in types). A misfeature (for this purpose) is that certain kinds of introspection are disabled (this was of course to enable restricted execution). I'd be willing to entertain improvements that improve the insulation this provides. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Bob Ippolito [EMAIL PROTECTED] writes: Personally my biggest issue with the way the CPython VM works is that you can't reliably do multiple interpreters in a single process. If that worked well, you could start an independent interpreter per thread and with a per-interpreter GIL you'd have pretty much everything you needed... but this would horribly break the C API and may be a performance hit. My use case for this isn't so much about threads, but plug-ins. Writing multiple Python-based plug-ins for an application is always a mess, because they share too much (sys.modules, sys.path, etc.). PyObjC would benefit greatly from this feature, because you can write Python-based plug-ins for any Cocoa app that supports plug-ins, even if they're otherwise unaware of Python's existence. There are workarounds, of course, with import hooks and similar hacks. I think that mod_python would also benefit from this, and probably other such systems. Just a note in case you didn't know this already, probably off-topic for python-dev, but not meant as advertisement for py2exe): the same (plug-in) problem occurs with an inprocess COM server and a Python client program, or more than one inproc COM server in any client program. The 0.6 py2exe release with it's bundle-file option allows to build COM dlls (and client EXE programs, if needed) that APPEAR to have a statically linked copy of Python##.dll, and so have several CPython VMs running in the same process. In case anybody want's to take a look or experiment with it. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
At 12:04 PM 9/21/2005 -0700, Guido van Rossum wrote: Actually Python itself has a hard time keeping multiple interpreters truly separate. Also of course there are some shared resources maintained by the operating system: current directory, open file descriptors, signal settings, child processes, that sort of thing. If we were to completely drop this feature, we could make built-in classes modifyable. I'd personally much rather we got back the ability to change the type of an instance of a builtin to that of a Python subclass of that builtin type, or to change it back. I have more use cases for that than for actually modifying builtins. (E.g. observable lists/dicts, hooking module __getattr__, etc.) A system like Java's classloader would be helpfull, where the classloader of a class is used to load the classes used by that class. I have no idea if this can be adapted to python at all. A strict coding style seems to work for now. You can do something like this using the restricted execution support, which works by setting the __builtins__ name in a dict where you exec code, and overriding __import__ in that __builtins__ dict. (I can't explain it too well in one paragraph, just go look up the rexec.py source code.) It's not great for guaranteeing there's absolutely no escape possible from the sandbox, but it works well enough to make accidental resource sharing a non-issue (apart from the OS shared resources and the built-in types). A misfeature (for this purpose) is that certain kinds of introspection are disabled (this was of course to enable restricted execution). Another misfeature is that some C-level Python code expects to obtain sys.modules, builtins, etc. via the interpreter struct. Thus, you tend to have to reimplement those things in Python to get them to respect a virtualization of sys.modules. I have to admit I've only dabbled in attempting this, just long enough to hit a stumbling block or two and then discover that they were because sys.modules is in the interpreter struct. Of course, my next thought then was to just expose the multi-interpreter API as an extension module, so that you could create interpreters from Python code. The project I'd originally planned to do this for never materialized though, so I never actually attempted it. My thought, though, was that by swapping the current interpreter of the thread state when crossing code boundaries, you could keep both the Python-level and C-level code happy. However, it might also suffice to have a way to switch in and out the interpreter configuration (sys.modules, sys.__dict__, and __builtins__ at minimum; I don't have any clear use case for changing the three codec_* vars at the moment). I'd be willing to entertain improvements that improve the insulation this provides. Since there's already a way to change __builtins__ in the threadstate, maybe the C API could be changed to obtain the six interpreter variables via builtins rather than the other way around. This would allow us to drop the multi-interpreter API from C (along with support for restricted mode) but still allow complete virtualization from inside Python code. The steps would be: 1. Remove restricted mode support 2. Change the tstate structure to have a 'builtins' 3. Change code that does tstate-interp lookups to instead lookup special names in the tstate's builtins At that point, you can exec code with new builtins to bootstrap a virtual Python, subject to limitations like being able to load a given extension module only once. Systems like mod_python that use the multi-interpreter API now would just need to bootstrap a new __builtins__. Sadly, this doesn't *really* cure the GIL-ensure problem, in that you still don't have a specially-distinguished __builtins__ to use when you call into Python from a C-started thread. On the other hand, I suspect that the use cases for that, and the use cases for virtualization don't overlap much, so having a distinguished place to hold the default (i.e. initial) builtins probably doesn't hurt virtualization much, since you can always *modify* that set of builtins if you need to. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
The best way to make people stop complaining about the GIL and start using process-based multiprogramming is to provide solid, standardized support for process-based multiprogramming. And the model provided by the thread abstraction is a good API for that support... Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Bill Janssen [EMAIL PROTECTED] wrote: The best way to make people stop complaining about the GIL and start using process-based multiprogramming is to provide solid, standardized support for process-based multiprogramming. And the model provided by the thread abstraction is a good API for that support... While creating a thread is generally quite easy, the threading abstraction assumes globally shared memory. Certainly there are multiprocessing systems which handle transferring of data between disparate memories automatically (Linda/PyLinda, POSH, etc.), but no doubt some users will want a more fine-grained control of data transfer (and this is saying nothing of the shared-memory model's currently horrible performance in Python). As such, there is always the option of using the tried and true MPI and PyMPI, which looks to have been recently updated. Or even XMLRPC and PickleRPC over sockets and/or mmap'd files. Then again, with how easy it is to distribute workloads using (Py)Linda, I'd be hard pressed to suggest any other module for multiprocessing into the standard library (unless it was a work-alike)...though perhaps we should wait until it has been sped up a bit, supports more data types, etc. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)
On Sun, 18 Sep 2005, Guido van Rossum wrote: On 9/17/05, John J Lee [EMAIL PROTECTED] wrote: [...snip...] [guido] If my hunch is right, I expect that instead of writing massively parallel applications, we will continue to write single-threaded applications that are tied together at the process level rather than at the thread level. I tend to agree. [...] I realize that not all algorithms (nor all computational problems) scale well to MP hardware. Is it feasible to usefully compile both MP and a UP binaries from one Python source code base? That's an understatement. I expect that *most* problems (even most problems that we will be programming 10-20 years from now) get little benefit out of MP. Perhaps, but I suspect we'll also get better at thinking up multiprocessor algorithms when better motivated by lack of exponential uniprocessor speed increases. ducks, fearing barrage of theorems... [...] Of course, it still takes a (anti-)hero to step forward and do the work... Be my guest. Prove me wrong. Talk is cheap; instead of arguing my points (all of which can be argued ad infinitum), come back when you've got a working GIL-free Python. Doesn't have to be CPython-based -- C# would be fine too. I don't actively want a GIL-free Python. I was just making some arguments in favour of GIL-removal that I hadn't seen made on a public list before. (In particular, removal now, since now is a special time.) John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On Mon, 19 Sep 2005, Florian Weimer wrote: The real problem is that you can ditch most extension modules. 8-( [...] *Is* that a showstopper for Python 3.0, though? John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)
On Tue, 20 Sep 2005, John J Lee wrote: [...] I don't actively want a GIL-free Python. I was just making some arguments [...] Actually, FWIW, I don't know if I even *passively* want a GIL-free Python, if it encourages threaded code (though I'd like to have that option for my occasional personal use, it seems to be an attractive nuisance for many other programmers). John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On 9/20/05, John J Lee [EMAIL PROTECTED] wrote: On Mon, 19 Sep 2005, Florian Weimer wrote: The real problem is that you can ditch most extension modules. 8-( [...] *Is* that a showstopper for Python 3.0, though? Who knows. I bet Guido doesn't even know how much breakage he is going to want to push. Some people have rather strongly pointed out (usually after I have proposed something), breaking stuff without a good reason is not worth the added level of breakage for when people try to update code to Python 3.0. Completely changing how garbage collection works is not exactly a minor thing and there is the possibility it won't pan out. It would really suck for everyone to have to learn an entirely new way of handling garbage collection and have there not be a perk for doing so, especially since this kind of change will not be directly visible to the language itself. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On Mon, 19 Sep 2005, Tim Lesher wrote: On 9/19/05, Michael Hudson [EMAIL PROTECTED] wrote: I was disappointed that that article (hey, it was the only issue of ddj I've ever actually bought! :) didn't consider any concurrency models other than shared memory threading. The problem is that, for all its limitations, shared-memory threading is the most popular concurrency model on the most popular operating system, so future hardware platforms targeting that system will be optimizing for that case. We can either rail against the sea, or accept it. Hmm, that's an interesting point. Aside from that point I tend to agree with Guido: threading is not the only, nor the best, concurrency model. But maybe these chips designed with threading in mind blow that argument out of the water. I don't know enough to know whether that's true or not... John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On Tue, 20 Sep 2005, Brett Cannon wrote: On 9/20/05, John J Lee [EMAIL PROTECTED] wrote: On Mon, 19 Sep 2005, Florian Weimer wrote: The real problem is that you can ditch most extension modules. 8-( [...] *Is* that a showstopper for Python 3.0, though? Who knows. I bet Guido doesn't even know how much breakage he is going to want to push. Some people have rather strongly pointed out (usually after I have proposed something), breaking stuff without a good reason is not worth the added level of breakage for when people try to update code to Python 3.0. Oh, absolutely. Completely changing how garbage collection works is not exactly a minor thing and there is the possibility it won't pan out. It would really suck for everyone to have to learn an entirely new way of handling garbage collection and have there not be a perk for doing so, especially since this kind of change will not be directly visible to the language itself. I didn't intend to refer to garbage collection in particular, but to removing the GIL, thus breaking extension modules (perhaps in a less-drastic way than implied by the copying garbage-collection proposal). John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On 9/20/05, John J Lee [EMAIL PROTECTED] wrote: threading is not the only, nor the best, concurrency model. But maybe these chips designed with threading in mind blow that argument out of the water. I don't know enough to know whether that's true or not... I don't know that any chips are designed with threading in mind. Fast threading benefits from fast context switches which benefits from small register sets. I believe the trend is towards ever large register sets. Also, multiple processors with shared memory don't scall all that well; multiple processors with explicit IPC channels scale much better. All arguments for multi-processing and against multi-threading. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On Sep 20, 2005, at 5:43 PM, Guido van Rossum wrote: On 9/20/05, John J Lee [EMAIL PROTECTED] wrote: threading is not the only, nor the best, concurrency model. But maybe these chips designed with threading in mind blow that argument out of the water. I don't know enough to know whether that's true or not... I don't know that any chips are designed with threading in mind. Fast threading benefits from fast context switches which benefits from small register sets. I believe the trend is towards ever large register sets. Also, multiple processors with shared memory don't scall all that well; multiple processors with explicit IPC channels scale much better. All arguments for multi-processing and against multi-threading. Well, in many operating systems, there isn't a whole lot of difference between threads and processes (both are kernel threads). When using threads, you can typically still use IPC, so you could scale similarly. I think the biggest argument for threading is simply that lots of existing C/C++ code wants to use threads. What we have now works OK, but you can't reliably use Python in a real-time thread (e.g. a CoreAudio callback) or (reliably) in a signal handler because it might block too long because of something else going on. Of course, a lot of other design decisions in Python would prevent you from using it in that context too, so GIL-free threading wouldn't be a panacea. Personally my biggest issue with the way the CPython VM works is that you can't reliably do multiple interpreters in a single process. If that worked well, you could start an independent interpreter per thread and with a per-interpreter GIL you'd have pretty much everything you needed... but this would horribly break the C API and may be a performance hit. My use case for this isn't so much about threads, but plug-ins. Writing multiple Python-based plug-ins for an application is always a mess, because they share too much (sys.modules, sys.path, etc.). PyObjC would benefit greatly from this feature, because you can write Python-based plug-ins for any Cocoa app that supports plug-ins, even if they're otherwise unaware of Python's existence. There are workarounds, of course, with import hooks and similar hacks. I think that mod_python would also benefit from this, and probably other such systems. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
At 06:10 PM 9/20/2005 -0400, Bob Ippolito wrote: My use case for this isn't so much about threads, but plug-ins. Writing multiple Python-based plug-ins for an application is always a mess, because they share too much (sys.modules, sys.path, etc.). PyObjC would benefit greatly from this feature, because you can write Python-based plug-ins for any Cocoa app that supports plug-ins, even if they're otherwise unaware of Python's existence. There are workarounds, of course, with import hooks and similar hacks. I think that mod_python would also benefit from this, and probably other such systems. To that list of shared things, one can easily add sys.argv, sys.stdin, sys.stdout, sys.stderr, os.environ, and on and on and on. But the ones you mention present special issues for testing, especially testing of import hooks, path management tools and so on. Decimal contexts are slightly better in that they're at least thread-local. But even if all of these things were thread-local, you'd still have problems with task switches between pseudothreads, or just maintaining logically separate contexts between e.g. different plugins. I've actually got an idea of how to solve these problems, at least for Python-level code, although for it to work right with Python's import internals it'd be necessary to use PyObject_* APIs against sys.modules instead of PyDict_* ones. Apart from that, it's implementable in pure Python, so non-CPython implementations should be able to use it too. I've been considering writing a PEP for it, since it also adds a generally-useful way of dealing with context variables (like the Decimal context and the sys.* variables), while being able to support switching *all* context variables simultaneously and instantaneously when changing execution contexts (like switching between coroutines). For example, it would let 1000 pseudothreads each have a unique current value for sys.stdout or the current decimal context, addressing a gap in the fit between PEP 343 and PEP 342. There's currently no reasonable way to task switch between co-routines in the body of a 'with:' statement, so things like with redirecting_stdout() to two places in different coroutines breaks horribly. But context managers implemented via context variables would handle such situations automatically, and with isolation of effects to the current thread. It would also allow you to have logically distinct plugins by swapping their context variables in when you make callbacks to them, so the notion of the task to which a set of context variables applies is not limited to coroutines. My prototype implementation of the basic idea is 200 lines of Python and very fast: ~2usec on my budget PC to switch between two arbitrarily-sized sets of context variables. The data structure is O(1) for switching sets or changing values; it's only O(N) the next time you change a value after taking a read-only snapshot of the whole context. (Taking a snapshot is O(1) due to copy-on-write; you can take as many snapshots of the same unchanged state without creating any new objects.) It also takes about 2uS to read a context variable, alas, but a C version could drop the overhead down to as little as a single dictionary lookup if the context pointer were kept in the thread state structure. (A fair bit of the Python implementation's overhead is just getting to the context mapping for the current thread.) The idea isn't fully PEP-ready at this point, though. I still need to flesh out my ideas regarding how context variables would be initialized in new threads, for example, or when you spawn new coroutines. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)
On 9/18/05, Guido van Rossum [EMAIL PROTECTED] wrote: On 9/17/05, John J Lee [EMAIL PROTECTED] wrote: c. Since time is needed to iron out bugs (and perhaps also to reimplememt some pieces of code from scratch), very early in the life of Python 3 seems like the least-worst time to begin work on such a change. I realize that not all algorithms (nor all computational problems) scale well to MP hardware. Is it feasible to usefully compile both MP and a UP binaries from one Python source code base? That's an understatement. I expect that *most* problems (even most problems that we will be programming 10-20 years from now) get little benefit out of MP. Some are saying it won't be a matter of choice if we want to get the software to run faster (you know, that MORE MORE MORE! thing we all seem to suffer from): http://www.gotw.ca/publications/concurrency-ddj.htm The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software Herb Sutter March 2005 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Martin Blais [EMAIL PROTECTED] writes: On 9/18/05, Guido van Rossum [EMAIL PROTECTED] wrote: On 9/17/05, John J Lee [EMAIL PROTECTED] wrote: c. Since time is needed to iron out bugs (and perhaps also to reimplememt some pieces of code from scratch), very early in the life of Python 3 seems like the least-worst time to begin work on such a change. I realize that not all algorithms (nor all computational problems) scale well to MP hardware. Is it feasible to usefully compile both MP and a UP binaries from one Python source code base? That's an understatement. I expect that *most* problems (even most problems that we will be programming 10-20 years from now) get little benefit out of MP. Some are saying it won't be a matter of choice if we want to get the software to run faster (you know, that MORE MORE MORE! thing we all seem to suffer from): People have been saying this for _years_, and it hasn't happened yet. This time around it's a bit more convincing, but I reserve the right to remain a touch skeptical. http://www.gotw.ca/publications/concurrency-ddj.htm The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software Herb Sutter March 2005 I was disappointed that that article (hey, it was the only issue of ddj I've ever actually bought! :) didn't consider any concurrency models other than shared memory threading. Cheers, mwh -- . - the pointyour article - . |- a long way | -- Christophe Rhodes, ucam.chat ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On 9/19/05, Michael Hudson [EMAIL PROTECTED] wrote: I was disappointed that that article (hey, it was the only issue of ddj I've ever actually bought! :) didn't consider any concurrency models other than shared memory threading. The problem is that, for all its limitations, shared-memory threading is the most popular concurrency model on the most popular operating system, so future hardware platforms targeting that system will be optimizing for that case. We can either rail against the sea, or accept it. -- Tim Lesher [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
* Martin Blais: http://www.gotw.ca/publications/concurrency-ddj.htm The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software Herb Sutter March 2005 This piece is fundamentally wrong. We all have been writing concurrent server-side software for eons. I don't know what Herb was thinking when he wrote that piece. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
* Guido van Rossum: That assumes a very specific model for how all that MP power is going to be used. Indeed. I personally don't think the threaded programming model as found in Java works all that well; without locks you end up with concurrent modification errors, with locks you get deadlocks and livelocks. Java is bascially forced into that model because VM startup costs are so high. To some extent, Python has similar problems, but you don't have to care about preserving class loader semantics, so you should be in a better position to cut down process creation time. Be my guest. Prove me wrong. Talk is cheap; instead of arguing my points (all of which can be argued ad infinitum), come back when you've got a working GIL-free Python. Doesn't have to be CPython-based -- C# would be fine too. By the way, has anybody ever tried to create a CPython variant which uses a (mostly) copying garbage collector (or something else except reference counting or Boehm GC)? Copying GC might help to get rid of the GIL *and* improve performance in the accept+fork model (because read-only object access does not trigger copy-on-write anymore). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Florian Weimer [EMAIL PROTECTED] writes: By the way, has anybody ever tried to create a CPython variant which uses a (mostly) copying garbage collector (or something else except reference counting or Boehm GC)? Not to my knowledge. I've always thought that it would be pretty hard. I'd be interested in being proved wrong. PyPy will hopefully experiment with this sort of thing. Copying GC might help to get rid of the GIL *and* improve performance in the accept+fork model (because read-only object access does not trigger copy-on-write anymore). How does a copying gc differ much from a non-copying non-refcounted gc here? Cheers, mwh -- Darn! I've only got 10 minutes left to get falling-down drunk! I suppose I'll have to smoke crack instead now. -- Tim Peters is checking things in on 2002-12-31 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
* Michael Hudson: Not to my knowledge. I've always thought that it would be pretty hard. I'd be interested in being proved wrong. The real problem is that you can ditch most extension modules. 8-( It sounds more like a fun project for the Python core, though. Copying GC might help to get rid of the GIL *and* improve performance in the accept+fork model (because read-only object access does not trigger copy-on-write anymore). How does a copying gc differ much from a non-copying non-refcounted gc here? You could copy immutable objects to a separate set of pages and never collect them (especially if recursively refer to immutable objects only). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
Michael Hudson wrote: How does a copying gc differ much from a non-copying non-refcounted gc here? One important issue for C coded modules is that addresses may change when a GC is invoked, so no remembering addresses in your module; you must recalculate before each use. -- Scott David Daniels [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL, Python 3, and MP vs. UP
On Mon, 2005-09-19 at 17:52, Scott David Daniels wrote: Michael Hudson wrote: How does a copying gc differ much from a non-copying non-refcounted gc here? One important issue for C coded modules is that addresses may change when a GC is invoked, so no remembering addresses in your module; you must recalculate before each use. And then you're left with fun like JNI's pining and such. That won't suck at all! :) -Barry signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)
On Fri, 16 Sep 2005, Martin v. Löwis wrote: Sokolov Yura wrote: I think I know how to remove GIL Obviously I am an idiot. Not an idiot, just lazy :-) Please try to implement your ideas, and I predict that you will find: 1. it is a lot of work to implement 2. it requires changes to all C files, in particular to extension modules outside the Python source tree proper. 3. performing the conversion, even in a semi-mechanical way, will introduce many new bugs, in the form of race conditions because of missing locks. Optionally, you may also find that the performance of the interpreter will decrease. [...] Given the points you make, and the facts that both Python 3 and real problems with continuing to scale down semiconductor chip feature sizes are on the horizon, it seems that now would be an excellent time to start work on this, with the goal of introducing it at the same time as Python 3. a. Python 3.0 will break lots of code anyway, so the extension module issue becomes far less significant. b. In x years time (x 10?) it seems likely multiprocessor (MP) users will be in the majority. (As a result, the uniprocessor (UP) slowdown becomes less important in practice, and also Python has the opportunity of avoiding the risk of being sidelined by a real or perceived lack of MP performance.) c. Since time is needed to iron out bugs (and perhaps also to reimplememt some pieces of code from scratch), very early in the life of Python 3 seems like the least-worst time to begin work on such a change. I realize that not all algorithms (nor all computational problems) scale well to MP hardware. Is it feasible to usefully compile both MP and a UP binaries from one Python source code base? (I'm also quite aware that the GIL does not prevent all means of achieving efficient use of multiprocessors. I'm just concious that different parellisation problems are presumably best expressed using different tools, and that Python 3 and increased prevalance of MP systems might tip the balance in favour of removing the GIL.) Of course, it still takes a (anti-)hero to step forward and do the work... John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
