Re: python performance on Solaris
Antoine Pitrou solip...@pitrou.net writes on Thu, 15 Oct 2009 16:25:43 + (UTC): Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écrit : Note that multithreaded compute-bound Python programs really suck on multiprocessors. Adding a second CPU makes the program go slower, due to a lame mechanism for resolving conflicts over the global interpreter lock. I'm not sure what you're talking about. Python has no mechanism for resolving conflicts over the global interpreter lock (let alone a lame one :-)), it just trusts the OS to schedule a thread only when it is not waiting on an unavailable resource (a lock). The GIL is just an OS-level synchronization primitive and its behaviour (fairness, performance) will depend on the behaviour of the underlying OS. But, independent from the OS and the fairness/performance of the GIL management itself: the GIL is there to prevent concurrent execution of Python code. Thus, at any time, at most one thread (in a process) is executing Python code -- other threads may run as well, as long as they are inside non Python code but cannot be executing Python bytecode, independent of available CPU resources. This implies that Python cannot fully exploit the power of multiprocessors. It is also true that adding CPUs may in fact reduce performance for compute bound multithreaded Python programs. While the additional computational resources cannot be use by Python, the additional overhead (switching between CPUs) may reduce overall performance. I agree with you that it is difficult to understand when this overhead were really significant. Dieter -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écrit : Note that multithreaded compute-bound Python programs really suck on multiprocessors. Adding a second CPU makes the program go slower, due to a lame mechanism for resolving conflicts over the global interpreter lock. I'm not sure what you're talking about. Python has no mechanism for resolving conflicts over the global interpreter lock (let alone a lame one :-)), it just trusts the OS to schedule a thread only when it is not waiting on an unavailable resource (a lock). The GIL is just an OS-level synchronization primitive and its behaviour (fairness, performance) will depend on the behaviour of the underlying OS. If this belief is derived from Dave Beazley's slides (*) about Python's multi-threading issues, I'd say some of his observations are rather questionable. First because the measurements were done on OS X, which has its own serious concurrency problems (**). Second because one of the benchmarks is so synthetic that it doesn't reflect real-world use of Python at all. This is not to say there aren't any issues, of course. (*) http://www.dabeaz.com/python/GIL.pdf (**) http://www.nabble.com/Mutex-performance-td24892454.html -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
In article 1a4707f5-85be-4f5f-ac3e-cf8f5bd21...@b15g2000yqd.googlegroups.com, inaf cem.ezbe...@gmail.com wrote: I have been following this group for quite some time and I figured (after searching enough on google --and on this group-- and not finding anything useful) I could pose this question here. Can anyone shed some light on python's performance on Solaris? My code seem to return lookups from a in memory data structure I build combining bunch of dictionaries and lists 6-8 times faster on a 32 bit Linux box than on a Solaris zone. Also, assuming that a zone is some kind of virtual machine, it will definitely impose its own overhead. -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it. --re...@lion.austin.ibm.com -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
On Thu, Oct 15, 2009 at 4:35 PM, Aahz a...@pythoncraft.com wrote: In article 1a4707f5-85be-4f5f-ac3e-cf8f5bd21...@b15g2000yqd.googlegroups.com, inaf cem.ezbe...@gmail.com wrote: I have been following this group for quite some time and I figured (after searching enough on google --and on this group-- and not finding anything useful) I could pose this question here. Can anyone shed some light on python's performance on Solaris? My code seem to return lookups from a in memory data structure I build combining bunch of dictionaries and lists 6-8 times faster on a 32 bit Linux box than on a Solaris zone. Also, assuming that a zone is some kind of virtual machine, it will definitely impose its own overhead. -- This list is old and not extremely accurate but a decent representation. http://unixconsult.org/jails_vs_zones_vs_xen.html opensolaris zones and freebsd jails can be thought of as an enhanced chroot enviro. Hence there is very little overhead wise %1 -- Adam Vande More -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
inaf cem.ezberci at gmail.com writes: Good point. I failed to compare the CPU power on these machines.. 32 bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I guess that explains :) Thank you for the tip.. You have to compare not only CPU frequencies but the CPU models. Recently Sun has been selling CPUs optimized for multi-threading (e.g. the UltraSPARC T2 or Niagara CPUs) which have, by design, very poor single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x difference in single-threaded performance compared to a modern Intel or AMD CPU is totally expected. Regards Antoine. -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
On Wed, 2009-10-14, Antoine Pitrou wrote: inaf cem.ezberci at gmail.com writes: Good point. I failed to compare the CPU power on these machines.. 32 bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I guess that explains :) Thank you for the tip.. You have to compare not only CPU frequencies but the CPU models. Yes, at least that. Megahertz figures have been useless for decades, except in advertising. Recently Sun has been selling CPUs optimized for multi-threading (e.g. the UltraSPARC T2 or Niagara CPUs) which have, by design, very poor single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x difference in single-threaded performance compared to a modern Intel or AMD CPU is totally expected. (Had to Google it. A Solaris Zone is apparently some kind of virtualization thing, with low CPU overhead.) s/multi-threading/multi-programming/ I suppose. I certainly hope you can still get performance while running many separate true processes in parallel. /Jorgen -- // Jorgen Grahn grahn@ Oo o. . . \X/ snipabacken.se O o . -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
I use python in almost the same environment. I use it on Joyent and on the Rackspace cloud. Joyent is faster for a few reasons (cpu bursting and faster disks) but these aren't real benchmarks until they are on the same machines. James On Wed, Oct 14, 2009 at 9:59 AM, Jorgen Grahn grahn+n...@snipabacken.segrahn%2bn...@snipabacken.se wrote: On Wed, 2009-10-14, Antoine Pitrou wrote: inaf cem.ezberci at gmail.com writes: Good point. I failed to compare the CPU power on these machines.. 32 bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I guess that explains :) Thank you for the tip.. You have to compare not only CPU frequencies but the CPU models. Yes, at least that. Megahertz figures have been useless for decades, except in advertising. Recently Sun has been selling CPUs optimized for multi-threading (e.g. the UltraSPARC T2 or Niagara CPUs) which have, by design, very poor single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x difference in single-threaded performance compared to a modern Intel or AMD CPU is totally expected. (Had to Google it. A Solaris Zone is apparently some kind of virtualization thing, with low CPU overhead.) s/multi-threading/multi-programming/ I suppose. I certainly hope you can still get performance while running many separate true processes in parallel. /Jorgen -- // Jorgen Grahn grahn@ Oo o. . . \X/ snipabacken.se O o . -- http://mail.python.org/mailman/listinfo/python-list -- http://www.goldwatches.com http://www.jewelerslounge.com -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
On Oct 14, 7:15 am, Antoine Pitrou solip...@pitrou.net wrote: inaf cem.ezberci at gmail.com writes: Good point. I failed to compare the CPU power on these machines.. 32 bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I guess that explains :) Thank you for the tip.. You have to compare not only CPU frequencies but the CPU models. Recently Sun has been selling CPUs optimized for multi-threading (e.g. the UltraSPARC T2 or Niagara CPUs) which have, by design, very poor single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x difference in single-threaded performance compared to a modern Intel or AMD CPU is totally expected. Regards Antoine. Antonie -- yes, you are right. Even the architecture of the two types make a difference. I was under the impression that RISC based CPUs did not need to have a very high clock speed and that they can perform similarly compared to an x86 processor with higher clock speed. That is why I was a bit surprised. I guess there could be other factors at play. That's why I was asking if there are specific things to be done while compiling Python on Solaris. I found some tips online which led me to compile it with a different threading lib resulting in slightly better performance after my original post. In terms of the processors I have, please see below for details: Status of virtual processor 40 as of: 10/14/2009 17:13:51 on-line since 07/23/2009 18:48:21. The sparcv9 processor operates at 1415 MHz, and has a sparcv9 floating point processor. So I guess this is not one of those you are talking about.. Thanks.. -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
inaf wrote: I have been following this group for quite some time and I figured (after searching enough on google --and on this group-- and not finding anything useful) I could pose this question here. Can anyone shed some light on python's performance on Solaris? Note that multithreaded compute-bound Python programs really suck on multiprocessors. Adding a second CPU makes the program go slower, due to a lame mechanism for resolving conflicts over the global interpreter lock. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
inaf cem.ezberci at gmail.com writes: My code seem to return lookups from a in memory data structure I build combining bunch of dictionaries and lists 6-8 times faster on a 32 bit Linux box than on a Solaris zone. Well, if your workload is CPU-bound, the issue here is not really Solaris vs. Linux but rather CPU power. You should try to run a generic (non-Python) CPU benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only Python shows such a performance difference, on the other hand, perhaps you can give us more precisions on those systems. Regards Antoine. (*) for example one of the C programs on http://shootout.alioth.debian.org/u64/c.php -- http://mail.python.org/mailman/listinfo/python-list
Re: python performance on Solaris
On Oct 11, 6:59 am, Antoine Pitrou solip...@pitrou.net wrote: inaf cem.ezberci at gmail.com writes: My code seem to return lookups from a in memory data structure I build combining bunch of dictionaries and lists 6-8 times faster on a 32 bit Linux box than on a Solaris zone. Well, if your workload is CPU-bound, the issue here is not really Solaris vs. Linux but rather CPU power. You should try to run a generic (non-Python) CPU benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only Python shows such a performance difference, on the other hand, perhaps you can give us more precisions on those systems. Regards Antoine. (*) for example one of the C programs onhttp://shootout.alioth.debian.org/u64/c.php Good point. I failed to compare the CPU power on these machines.. 32 bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I guess that explains :) Thank you for the tip.. -- http://mail.python.org/mailman/listinfo/python-list
