Re: GIL in alternative implementations
On Tue, 07 Jun 2011 01:03:55 -0300, Gabriel Genellina wrote: En Sat, 28 May 2011 14:05:16 -0300, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info escribió: On Sat, 28 May 2011 09:39:08 -0700, John Nagle wrote: Python allows patching code while the code is executing. Can you give an example of what you mean by this? [...] I think John Nagle was thinking about rebinding names: def f(self, a, b): while b0: b = g(b) c = a + b d = self.h(c*3) return hello world*d both g and self.h may change its meaning from one iteration to the next, so a complete name lookup is required at each iteration. This is very useful sometimes, but affects performance a lot. Ah, that was what I was missing. Thanks Gabriel. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
On Jun 7, 12:03 am, Gabriel Genellina gagsl-...@yahoo.com.ar wrote: En Sat, 28 May 2011 14:05:16 -0300, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info escribi : On Sat, 28 May 2011 09:39:08 -0700, John Nagle wrote: Python allows patching code while the code is executing. Can you give an example of what you mean by this? If I have a function: def f(a, b): c = a + b d = c*3 return hello world*d how would I patch this function while it is executing? I think John Nagle was thinking about rebinding names: def f(self, a, b): while b0: b = g(b) c = a + b d = self.h(c*3) return hello world*d both g and self.h may change its meaning from one iteration to the next, so a complete name lookup is required at each iteration. This is very useful sometimes, but affects performance a lot. And even the original example, with only + and * can have side- effects. Who knows how a defines __add__? Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
On Monday, June 6, 2011 9:03:55 PM UTC-7, Gabriel Genellina wrote: En Sat, 28 May 2011 14:05:16 -0300, Steven D'Aprano steve+comp@pearwood.info escribi�: On Sat, 28 May 2011 09:39:08 -0700, John Nagle wrote: Python allows patching code while the code is executing. Can you give an example of what you mean by this? If I have a function: def f(a, b): c = a + b d = c*3 return hello world*d how would I patch this function while it is executing? I think John Nagle was thinking about rebinding names: def f(self, a, b): while b0: b = g(b) c = a + b d = self.h(c*3) return hello world*d both g and self.h may change its meaning from one iteration to the next, so a complete name lookup is required at each iteration. This is very useful sometimes, but affects performance a lot. It's main affect performance is that it prevents an optimizer from inlining a function call(which is a good chunk of the payoff you get in languages that can do that). I'm not sure where he gets the idea that this has any impact on concurrency, though. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
Carl Banks wrote: On Monday, June 6, 2011 9:03:55 PM UTC-7, Gabriel Genellina wrote: En Sat, 28 May 2011 14:05:16 -0300, Steven D'Aprano steve+comp@pearwood.info escribi�: On Sat, 28 May 2011 09:39:08 -0700, John Nagle wrote: Python allows patching code while the code is executing. Can you give an example of what you mean by this? If I have a function: def f(a, b): c = a + b d = c*3 return hello world*d how would I patch this function while it is executing? I think John Nagle was thinking about rebinding names: def f(self, a, b): while b0: b = g(b) c = a + b d = self.h(c*3) return hello world*d both g and self.h may change its meaning from one iteration to the next, so a complete name lookup is required at each iteration. This is very useful sometimes, but affects performance a lot. It's main affect performance is that it prevents an optimizer from inlining a function call(which is a good chunk of the payoff you get in languages that can do that). I'm not sure where he gets the idea that this has any impact on concurrency, though. What if f has two calls to self.h() [or some other function], and self.h changes in between? Surely that would be a major headache. ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
On Tue, Jun 7, 2011 at 11:51 AM, Ethan Furman et...@stoneleaf.us wrote: I'm not sure where he gets the idea that this has any impact on concurrency, though. What if f has two calls to self.h() [or some other function], and self.h changes in between? Surely that would be a major headache. I could imagine a problem similar to the non-atomic increment example arising from continuations. from functools import partial def g(value): print(value) return partial(g, value+1) f = partial(0) for i in range(1): f = f() With a single thread, this will print the numbers 0 to . With multiple threads executing the loop simultaneously, not so much. Note that this is not the same example as the non-atomic increment, because making value += 1 atomic would not fix this. You would have to make the entire function call (and subsequent assignment) atomic to make this concurrent. Cheers, Ian -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
On Tue, Jun 7, 2011 at 2:22 PM, Ian Kelly ian.g.ke...@gmail.com wrote: from functools import partial def g(value): print(value) return partial(g, value+1) f = partial(0) for i in range(1): f = f() The partial(0) should read partial(g, 0), of course. -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
En Sat, 28 May 2011 14:05:16 -0300, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info escribió: On Sat, 28 May 2011 09:39:08 -0700, John Nagle wrote: Python allows patching code while the code is executing. Can you give an example of what you mean by this? If I have a function: def f(a, b): c = a + b d = c*3 return hello world*d how would I patch this function while it is executing? I think John Nagle was thinking about rebinding names: def f(self, a, b): while b0: b = g(b) c = a + b d = self.h(c*3) return hello world*d both g and self.h may change its meaning from one iteration to the next, so a complete name lookup is required at each iteration. This is very useful sometimes, but affects performance a lot. -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
Thanks for the details on IronPython's implementation B-) Hopefully Pypy will eventually get rid of its own Gil, since it doesn't do refcounting either. Regards, Pascal Le 28/05/2011 00:52, Dino Viehland a écrit : In IronPython we have fine grained locking on our mutable data structures. In particular we have a custom dictionary type which is designed to allow lock-free readers on common operations while writers take a lock. Our list implementation is similar but in some ways that's trickier to pull off due to features like slicing so if I recall correctly we only have lock-free reads when accessing a single element. For .NET data structures they follow the .NET convention which is up to the data structure. So if you wanted to get every last bit of performance out of your app you could handle thread safety yourself and switch to using the .NET dictionary or list types (although they're a lot less friendly to Python developers). Because of these locks on micro-benchmarks that involve simple list/dict manipulations you do see noticeably worse performance in IronPython vs. CPython. http://ironpython.codeplex.com/wikipage?title=IP27A1VsCPy27PerfreferringTitle=IronPython%20Performance http://ironpython.codeplex.com/wikipage?title=IP27A1VsCPy27PerfreferringTitle=IronPython%20Performance - See the SimpleListManipulation and SimpleDictManipulation as the core examples here. Also CPython's dictionary is so heavily tuned it's hard to beat anyway, but this is a big factor. Finally one of the big differences with both Jython and IronPython is that we have good garbage collectors which don't rely upon reference counting. So one area where CPython gains from having a GIL is a non-issue for us as we don't need to protect ref counts or use interlocked operations for ref counting. *From:* python-list-bounces+dinov=exchange.microsoft@python.org [mailto:python-list-bounces+dinov=exchange.microsoft@python.org] *On Behalf Of *Pascal Chambon *Sent:* Friday, May 27, 2011 2:22 PM *To:* python-list@python.org Python List *Subject:* GIL in alternative implementations Hello everyone, I've already read quite a bit about the reasons for the GIL in CPython, i.e to summarize, that a more-fine graine locking, allowing real concurrency in multithreaded applications, would bring too much overhead for single-threaded python applications. However, I've also heard that other python implementations (ironpython, jython...) have no GIL, and yet nobody blames them for performance penalties that would be caused by that lack (I especially think about IronPython, whose performances compare quite well to CPython). So I'd like to know: how do these other implementations handle concurrency matters for their primitive types, and prevent them from getting corrupted in multithreaded programs (if they do) ? I'm not only thinking about python types, but also primitive containers and types used in .Net and Java VMs, which aren't atomic elements either at an assembly-level point of view. Do these VMs have some GIL-like limitations, that aren't spoken about ? Are there functionings completely different from the CPython VM, so that the question is not relevant ? Do people consider that they always concern multithreaded applications, and so accept performance penalties that they wouldn't allow in their CPython scripts ? I think you in advance for your lights on these questions. Regards, Pkl [[ Important Note: this is a serious question, trolls and emotionally disturbed persons had better go on their way. ]] -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
Daniel Kluev dan.kl...@gmail.com wrote: test.py: from threading import Thread class X(object): pass obj = X() obj.x = 0 def f(*args): for i in range(1): obj.x += 1 threads = [] for i in range(100): t = Thread(target=f) threads.append(t) t.start() for t in threads: while t.isAlive(): t.join(1) print(obj.x) python test.py 100 pypy test.py 100 jython-2.5 test.py 19217 ipy test.py 59040 Not that this thing is reasonable to do in real code, but cpython and other implementations with GIL at least give you some safety margin. Sure? ;) for p in python2.4 python2.5 python2.6 python2.7 python3.1 python3.2; do echo $p; $p /tmp/test.py; $p /tmp/test.py; done python2.4 525369 736202 python2.5 449496 551023 python2.6 903405 937335 python2.7 885834 910144 python3.1 866557 766842 python3.2 100 100 So even CPython (at least 3.2) isn't safe. And I wouldn't rely on 3.2 not to break. Marc -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
Daniel Kluev wrote: So I'd like to know: how do these other implementations handle concurrency matters for their primitive types, and prevent them from getting corrupted in multithreaded programs (if they do) ? I'm not only thinking about python types, but also primitive containers and types used in .Net and Java VMs, which aren't atomic elements either at an assembly-level point of view. Well, they definitely have some shortcomings: test.py: from threading import Thread class X(object): pass obj = X() obj.x = 0 def f(*args): for i in range(1): obj.x += 1 threads = [] for i in range(100): t = Thread(target=f) threads.append(t) t.start() for t in threads: while t.isAlive(): t.join(1) print(obj.x) python test.py 100 pypy test.py 100 jython-2.5 test.py 19217 ipy test.py 59040 Not that this thing is reasonable to do in real code, but cpython and other implementations with GIL at least give you some safety margin. The problem with your code is that it gives the wrong result when a thread reads obj.x, then suspends, then another thread reads obj.x and finally both threads store the same value+1 back. That problem has nothing to do with the GIL which just prohibits that two threads can run at the same time, on different cores. It occurs because obj.x += 1 is not an atomic operation. That lack of atomicity should be obvious if you take a look at the byte-code: def f(): ... obj.x += 1 ... dis.dis(f) 2 0 LOAD_GLOBAL 0 (obj) 3 DUP_TOP 4 LOAD_ATTR1 (x) 7 LOAD_CONST 1 (1) 10 INPLACE_ADD 11 ROT_TWO 12 STORE_ATTR 1 (x) 15 LOAD_CONST 0 (None) 18 RETURN_VALUE [Marc Christiansen] So even CPython (at least 3.2) isn't safe. And I wouldn't rely on 3.2 not to break. I don't know why it /seems/ to work in 3.2 more often than in 2.x, but in general bugs that occur only sporadically are typical for multithreaded code... -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
On Samstag 28 Mai 2011, Marc Christiansen wrote: And I wouldn't rely on 3.2 not to break. it breaks too like it should, but only rarely like one out of 10 times i5:/pub/src/gitgames/kajongg/src$ python3.2 test.py 100 i5:/pub/src/gitgames/kajongg/src$ python3.2 test.py 100 i5:/pub/src/gitgames/kajongg/src$ python3.2 test.py 98 i5:/pub/src/gitgames/kajongg/src$ python3.2 test.py 100 -- Wolfgang -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
On 5/27/2011 7:06 PM, Daniel Kluev wrote: So I'd like to know: how do these other implementations handle concurrency matters for their primitive types, and prevent them from getting corrupted in multithreaded programs (if they do) ? I'm not only thinking about python types, but also primitive containers and types used in .Net and Java VMs, which aren't atomic elements either at an assembly-level point of view. += is not guaranteed to be atomic in most languages. Some C and C++ implementations have atomic_inc, etc., which is guaranteed to execute as an atomic operation. How do most safe languages handle concurrency? For the unboxed primitive types, like numbers, the hardware handles it. For memory allocation, there's a lock. Most don't allow patching code on the fly. Concurrent garbage collection prevents deleting something if there's a pointer to it anywhere. This was all worked out for LISP and SELF decades ago. Python allows patching code while the code is executing. This implies a sizable performance penalty, and causes incredible complexity in PyPy. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
On Sat, 28 May 2011 09:39:08 -0700, John Nagle wrote: Python allows patching code while the code is executing. Can you give an example of what you mean by this? If I have a function: def f(a, b): c = a + b d = c*3 return hello world*d how would I patch this function while it is executing? -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL in alternative implementations
So I'd like to know: how do these other implementations handle concurrency matters for their primitive types, and prevent them from getting corrupted in multithreaded programs (if they do) ? I'm not only thinking about python types, but also primitive containers and types used in .Net and Java VMs, which aren't atomic elements either at an assembly-level point of view. Well, they definitely have some shortcomings: test.py: from threading import Thread class X(object): pass obj = X() obj.x = 0 def f(*args): for i in range(1): obj.x += 1 threads = [] for i in range(100): t = Thread(target=f) threads.append(t) t.start() for t in threads: while t.isAlive(): t.join(1) print(obj.x) python test.py 100 pypy test.py 100 jython-2.5 test.py 19217 ipy test.py 59040 Not that this thing is reasonable to do in real code, but cpython and other implementations with GIL at least give you some safety margin. -- With best regards, Daniel Kluev -- http://mail.python.org/mailman/listinfo/python-list
