Re: Parallel python in the cloud

On 2014-05-24 07:46, Charles Gagnon wrote: We were happily using PiCloud for several long calculations and we very happy with with it. With their realtime cores, we could take really large calculations set and run through fairly quickly. Now that PiCloud is going away, we ran a few tests on Mu

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

Hi. On 25.11.2013. 14:42, Jurko Gospodnetić wrote: So far all tests seem to indicate that things work out fine if we install to some dummy target folder, copy the target folder to some version specific location & uninstall. That leaves us with a working Python folder sans the start menu and

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

Hi. On 25.11.2013. 17:38, Terry Reedy wrote: So far all tests seem to indicate that things work out fine if we install to some dummy target folder, copy the target folder to some version specific location & uninstall. If the dummy folder had 3.3.0, you should not need to uninstall to inst

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

On 11/25/2013 8:42 AM, Jurko Gospodnetić wrote: So far all tests seem to indicate that things work out fine if we install to some dummy target folder, copy the target folder to some version specific location & uninstall. If the dummy folder had 3.3.0, you should not need to uninstall to in

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

Hi. On 25.11.2013. 15:15, Albert-Jan Roskam wrote: > Are you sure? http://stackoverflow.com/questions/1534210/use-different-python-version-with-virtualenv Yup, I'm pretty sure by now (based on reading the docs, not trying it out though). Virtualenv allows you to set up differen

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

On Tue, Nov 26, 2013 at 1:15 AM, Albert-Jan Roskam wrote: > Below is a little terminal session. I often switch between python 3.3 and > python 2.7. My virtualenv for python 3.3 is called "python33". "workon" is a > virtualenv wrapper command. And check out the envlist in tox.ini on > http://to

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

On Mon, 11/25/13, Jurko Gospodnetić wrote: Subject: Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine To: python-list@python.org Date: Monday, November 25, 2013, 2:57 PM   Hi. On 25.11.2013. 14:20, Albert-Jan

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

Hi. On 25.11.2013. 14:20, Albert-Jan Roskam wrote: Check out the following packages: virtualenv, virtualenvwrapper, tox virtualenv + wrapper make it very easy to switch from one python version to another. Stricly speaking you don't need virtualenvwrapper, but it makes working with virtualenv a

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

Hi. On 25.11.2013. 13:46, Ned Batchelder wrote: IIRC, Python itself doesn't read those registry entries, except when installing pre-compiled .msi or .exe kits. Once you have Python installed, you can move the directory someplace else, then install another version of Python. If you need to

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

On Tue, Nov 26, 2013 at 12:42 AM, Jurko Gospodnetić wrote: > Yup, we could do that, but at first glance it really smells like an > overkill. Not to mention the potential licensing issues with Windows and an > unlimited number of Windows installations. :-) Ah, heh... didn't think of that. When I

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

Hi. On 25.11.2013. 14:04, Chris Angelico wrote: Is it possible to set up virtualization to help you out? Create a virtual machine in something like VirtualBox, then clone it for every Python patch you want to support (you could have one VM that handles all the .0 releases and another that hand

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

On Mon, 11/25/13, Jurko Gospodnetić wrote: Subject: Parallel Python x.y.A and x.y.B installations on a single Windows machine To: python-list@python.org Date: Monday, November 25, 2013, 1:32 PM   Hi all.   I was wondering what is the best way

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

On Mon, Nov 25, 2013 at 11:32 PM, Jurko Gospodnetić wrote: > Most of the advice seems to boil down to 'do not use such versions together, > use only the latest'. > > We would like to run automated tests on one of our projects (packaged as a > Python library) with different Python versions, and s

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

On Monday, November 25, 2013 7:32:30 AM UTC-5, Jurko Gospodnetić wrote: > Hi all. > >I was wondering what is the best way to install multiple Python > installations on a single Windows machine. > >Regular Windows installer works great as long as all your > installations have a separate

Re: Parallel python + ??

Gerhard Häring wrote: > This is of course OS-specific. On Linux, you can parse the proc > filesystem: > > >>> open("/proc/%i/stat" % os.getpid()).read().split()[39] > > You can use the "taskset" utility to query or set CPU affinity on Linux. > It is going to be in Linux (mainly) I was thinking

Re: Parallel python + ??

Thor wrote: Hi, I am running a program using Parallel Python and I wonder if there is a way/module to know in which CPU/core the process is running in. Is that possible? This is of course OS-specific. On Linux, you can parse the proc filesystem: >>> open("/proc/%i/stat" % os.getpid()).read().

RE: Parallel Python environments..

* bruce (Tue, 6 Nov 2007 13:43:10 -0800) > if i have python 2.4.3 installed, it gets placed in the python2.4 dir.. if i > don't do anything different, and install python 2.4.2, it too will get > placed in the python2.4 tree... which is not what i want. > > i'm running rhel4/5... So you're using r

RE: Parallel Python environments..

r 06, 2007 2:07 PM To: python-list@python.org Subject: Re: Parallel Python environments.. En Tue, 06 Nov 2007 18:43:10 -0300, bruce <[EMAIL PROTECTED]> escribió: > if i have python 2.4.3 installed, it gets placed in the python2.4 dir.. > if i > don't do anything different, an

Re: Parallel Python environments..

En Tue, 06 Nov 2007 18:43:10 -0300, bruce <[EMAIL PROTECTED]> escribió: > if i have python 2.4.3 installed, it gets placed in the python2.4 dir.. > if i > don't do anything different, and install python 2.4.2, it too will get > placed in the python2.4 tree... which is not what i want. Any rea

RE: Parallel Python environments..

e in order to be able to run multiple versions of python, and to switch back/forth between the versions. thanks -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Thorsten Kampe Sent: Tuesday, November 06, 2007 8:19 AM To: python-list@python.org Subject: Re

RE: Parallel Python environments..

i'm running rhel... so there isn't a python-config script as far as i know.. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of [EMAIL PROTECTED] Sent: Tuesday, November 06, 2007 8:26 AM To: python-list@python.org Subject: Re: Parallel Python en

Re: Parallel Python environments..

bruce wrote: > Hi.. > > If I wanted to be able to build/test/use parallel python versions, what > would I need to do/set (paths/libs/etc...) and where would I need to place > the 2nd python version, so as not to screw up my initial python dev env. > > I'd like to be able to switch back/forth bet

Re: Parallel Python environments..

In Gentoo Linux you can select between installed python version using python-config script. -- http://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python environments..

* bruce (Tue, 6 Nov 2007 07:13:43 -0800) > If I wanted to be able to build/test/use parallel python versions, what > would I need to do/set (paths/libs/etc...) nothing > and where would I need to place the 2nd python version, so as not to > screw up my initial python dev env. Anywhere you like (

Re: Parallel Python

On Jan 12, 11:52 am, Neal Becker <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > > Has anybody tried to runparallelpythonapplications? > > It appears that if your application is computation-bound using 'thread' > > or 'threading' modules will not get you any speedup. That is because > >pyth

Re: Parallel Python

> Looks interesting, but is there any way to use this for a cluster of > machines over a network (not smp)? Networking capabilities will be included in the next release of Parallel Python software (http://www.parallelpython.com), which is coming soon. > Couldn't you just provide similar convenie

Re: Parallel Python

Konrad Hinsen wrote: Perhaps we should start a > special interest group? Not so much in order to work on a single > project; I believe that at the current state of parallel computing we > still need many different approaches to be tried. But an exchange of > experience could well be useful fo

Re: Parallel Python

On Jan 12, 2007, at 15:08, Paul Boddie wrote: > It seems to me that a more useful first step would be to create an > overview of the different modules and put it on the python.org Wiki: > > http://wiki.python.org/moin/FrontPage > http://wiki.python.org/moin/UsefulModules (a reasonable entry point)

Re: Parallel Python

Konrad Hinsen wrote: > > That's one more project... It seems that there is significant > interest in parallel computing in Python. Perhaps we should start a > special interest group? Not so much in order to work on a single > project; I believe that at the current state of parallel computing we > s

Re: Parallel Python

On Jan 12, 2007, at 11:21, Paul Boddie wrote: > done. My own experience with regard to releasing software is that even > with an open source licence, most people are likely to ignore your > projects than to suddenly jump on board and take control, and even if My experience is exactly the same. An

Re: Parallel Python

robert wrote: > Paul Boddie wrote: > > > > [1] http://www.python.org/pypi/parallel > > I'd be interested in an overview. I think we've briefly discussed the above solution before, and I don't think you're too enthusiastic about anything using interprocess communication, which is what the above sol

Re: Parallel Python

[EMAIL PROTECTED] wrote: > Has anybody tried to run parallel python applications? > It appears that if your application is computation-bound using 'thread' > or 'threading' modules will not get you any speedup. That is because > python interpreter uses GIL(Global Interpreter Lock) for internal > b

Re: Parallel Python

Paul Boddie wrote: > [EMAIL PROTECTED] wrote: >> The main difference between MPI python solutions and ppsmp is that with >> MPI you have to organize both computations >> {MPI_Comm_rank(MPI_COMM_WORLD, &id); if id==1 then ... else } and >> data distribution (MPI_Send / MPI_Recv) by yourself. Whi

Re: Parallel Python

In article <[EMAIL PROTECTED]>, "Paul Boddie" <[EMAIL PROTECTED]> writes: |> [EMAIL PROTECTED] wrote: |> > |> > The main difference between MPI python solutions and ppsmp is that with |> > MPI you have to organize both computations |> > {MPI_Comm_rank(MPI_COMM_WORLD, &id); if id==1 then ... else .

Re: Parallel Python

[EMAIL PROTECTED] wrote: > > The main difference between MPI python solutions and ppsmp is that with > MPI you have to organize both computations > {MPI_Comm_rank(MPI_COMM_WORLD, &id); if id==1 then ... else } and > data distribution (MPI_Send / MPI_Recv) by yourself. While with ppsmp > you jus

Re: Parallel Python

Hi, You guys forgot pyMPI, http://pympi.sourceforge.net/ It works fine !!! A little hard installation and configuration but finally works !!! Cordially, Jairo Serrano Bucaramanga, Colombia [EMAIL PROTECTED] wrote: > > > > Thus there are different levels of parallelization: > > > > 1 file/datab

Re: Parallel Python

> > Thus there are different levels of parallelization: > > 1 file/database based; multiple batch jobs > 2 Message Passing, IPC, RPC, ... > 3 Object Sharing > 4 Sharing of global data space (Threads) > 5 Local parallelism / Vector computing, MMX, 3DNow,... > > There are good reasons for all of thes

Re: Parallel Python

sturlamolden wrote: > [EMAIL PROTECTED] wrote: > > >That's right. ppsmp starts multiple interpreters in separate > > processes and organize communication between them through IPC. > > Thus you are basically reinventing MPI. > > http://mpi4py.scipy.org/ > http://en.wikipedia.org/wiki/Message_Pas

Re: Parallel Python

On Jan 8, 2007, at 11:33, Duncan Booth wrote: > The 'parallel python' site seems very sparse on the details of how > it is > implemented but it looks like all it is doing is spawning some > subprocesses > and using some simple ipc to pass details of the calls and results. > I can't > tell fr

Re: Parallel Python

In article <[EMAIL PROTECTED]>, robert <[EMAIL PROTECTED]> writes: |> |> Thus there are different levels of parallelization: |> |> 1 file/database based; multiple batch jobs |> 2 Message Passing, IPC, RPC, ... |> 3 Object Sharing |> 4 Sharing of global data space (Threads) |> 5 Local parallelis

Re: Parallel Python

sturlamolden wrote: > robert wrote: > >> Thus communicated data is "serialized" - not directly used as with threads >> or with custom shared memory techniques like POSH object sharing. > > Correct, and that is precisely why MPI code is a lot easier to write > and debug than thread code. The OP u

Re: Parallel Python

[EMAIL PROTECTED] (Nick Maclaren) writes: [...] > I mean precisely the first. > > The C99 standard uses a bizarre consistency model, which requires serial > execution, and its consistency is defined in terms of only volatile > objects and external I/O. Any form of memory access, signalling or > wh

Re: Parallel Python

robert wrote: > Thus communicated data is "serialized" - not directly used as with threads or > with custom shared memory techniques like POSH object sharing. Correct, and that is precisely why MPI code is a lot easier to write and debug than thread code. The OP used a similar technique in his

Re: Parallel Python

In article <[EMAIL PROTECTED]>, robert <[EMAIL PROTECTED]> writes: |> |> Most threads on this planet are not used for number crunching jobs, |> but for "organization of execution". That is true, and it is effectively what POSIX and Microsoft threads are suitable for. With reservations, even the

Re: Parallel Python

In article <[EMAIL PROTECTED]>, Sergei Organov <[EMAIL PROTECTED]> writes: |> |> OK, then I don't think the POSIX threads were "perpetrated" to be idle |> most of time. Perhaps I was being unclear. I should have added "In the case where there are more threads per system than CPUs per system".

Re: Parallel Python

[EMAIL PROTECTED] (Nick Maclaren) writes: > In article <[EMAIL PROTECTED]>, > Sergei Organov <[EMAIL PROTECTED]> writes: > |> > |> Do you mean that POSIX threads are inherently designed and implemented > |> to stay idle most of the time?! If so, I'm afraid those guys that > |> designed POSIX thre

Re: Parallel Python

Nick Maclaren wrote: > In article <[EMAIL PROTECTED]>, > Paul Rubin writes: > |> > |> > Yes, I know that it is a bit Irish for the best way to use a shared > |> > memory system to be to not share memory, but that's how it is. > |> > |> But I thought serious MPI implement

Re: Parallel Python

sturlamolden wrote: > Nick Maclaren wrote: > > I wonder if too much emphasis is put on thread programming these days. > Threads may be nice for programming web servers and the like, but not > for numerical computing. Reading books about thread programming, one > can easily get the impression that

Re: Parallel Python

In article <[EMAIL PROTECTED]>, "Carl J. Van Arsdall" <[EMAIL PROTECTED]> writes: |> |> Just as something to note, but many HPC applications will use a |> combination of both MPI and threading (OpenMP usually, as for the |> underlying thread implementation i don't have much to say). Its |> in

Re: Parallel Python

Just as something to note, but many HPC applications will use a combination of both MPI and threading (OpenMP usually, as for the underlying thread implementation i don't have much to say). Its interesting to see on this message board this huge "anti-threading" mindset, but the HPC community s

Re: Parallel Python

In article <[EMAIL PROTECTED]>, Sergei Organov <[EMAIL PROTECTED]> writes: |> |> Do you mean that POSIX threads are inherently designed and implemented |> to stay idle most of the time?! If so, I'm afraid those guys that |> designed POSIX threads won't agree with you. In particular, as far as I |

Re: Parallel Python

[EMAIL PROTECTED] (Nick Maclaren) writes: > In article <[EMAIL PROTECTED]>, > "sturlamolden" <[EMAIL PROTECTED]> writes: [...] > |> I wonder if too much emphasis is put on thread programming these days. > |> Threads may be nice for programming web servers and the like, but not > |> for numerical co

Re: Parallel Python

In article <[EMAIL PROTECTED]>, Paul Rubin writes: |> |> > Yes, I know that it is a bit Irish for the best way to use a shared |> > memory system to be to not share memory, but that's how it is. |> |> But I thought serious MPI implementations use shared memory if they |

Re: Parallel Python

In article <[EMAIL PROTECTED]>, "sturlamolden" <[EMAIL PROTECTED]> writes: |> |> In any case, this means that Python can happily keep its GIL, as the |> CPU bound 'HPC' tasks for which the GIL does matter should be done |> using multiple processes (not threads) anyway. That leaves threads as a |>

Re: Parallel Python

[EMAIL PROTECTED] (Nick Maclaren) writes: > Yes, I know that it is a bit Irish for the best way to use a shared > memory system to be to not share memory, but that's how it is. But I thought serious MPI implementations use shared memory if they can. That's the beauty of it, you can run your appli

Re: Parallel Python

Nick Maclaren wrote: > as the ones that you have to play for threaded programs. Yes, I know > that it is a bit Irish for the best way to use a shared memory system > to be to not share memory, but that's how it is. Thank you for clearing that up. In any case, this means that Python can happily

Re: Parallel Python

In article <[EMAIL PROTECTED]>, "sturlamolden" <[EMAIL PROTECTED]> writes: |> |> MPI is becoming the de facto standard for high-performance parallel |> computing, both on shared memory systems (SMPs) and clusters. It has been for some time, and is still gaining ground. |> Spawning |> threads or

Re: Parallel Python

[EMAIL PROTECTED] wrote: >That's right. ppsmp starts multiple interpreters in separate > processes and organize communication between them through IPC. Thus you are basically reinventing MPI. http://mpi4py.scipy.org/ http://en.wikipedia.org/wiki/Message_Passing_Interface -- http://mail.p

Re: Parallel Python

robert wrote: > Thats true. IPC through sockets or (somewhat faster) shared memory - cPickle > at least - is usually the maximum of such approaches. > See > http://groups.google.de/group/comp.lang.python/browse_frm/thread/f822ec289f30b26a > > For tasks really requiring threading one can consid

Re: Parallel Python

> I always thought that if you use multiple processes (e.g. os.fork) then > Python can take advantage of multiple processors. I think the GIL locks > one processor only. The problem is that one interpreted can be run on > one processor only. Am I not right? Is your ppm module runs the same > interp

Re: Parallel Python

On 1/8/07, Laszlo Nagy <[EMAIL PROTECTED]> wrote: I always thought that if you use multiple processes (e.g. os.fork) then Python can take advantage of multiple processors. I think the GIL locks one processor only. The problem is that one interpreted can be run on one processor only. Am I not rig

Re: Parallel Python

Duncan Booth wrote: > Laszlo Nagy <[EMAIL PROTECTED]> wrote: > > > The 'parallel python' site seems very sparse on the details of how it is > implemented but it looks like all it is doing is spawning some subprocesses > and using some simple ipc to pass details of the calls and results. I can't

Re: Parallel Python

Laszlo Nagy <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: >> Has anybody tried to run parallel python applications? >> It appears that if your application is computation-bound using 'thread' >> or 'threading' modules will not get you any speedup. That is because >> python interpreter uses

Re: Parallel Python

[EMAIL PROTECTED] wrote: > Has anybody tried to run parallel python applications? > It appears that if your application is computation-bound using 'thread' > or 'threading' modules will not get you any speedup. That is because > python interpreter uses GIL(Global Interpreter Lock) for internal > bo

Re: Parallel Python on PowerMac?

Alan Kennedy wrote: Although, iff your prospective machine supports System V IPC, you might want to check out PoSH. http://poshmodule.sourceforge.net It uses inline assembly, so that's a no-go on the PPC unless someone ports the assembly code. -- Robert Kern [EMAIL PROTECTED] "In the fields of

Re: Parallel Python on PowerMac?

Hello, and thanks for your reply. > But I would venture to say that, *in the general case*, the "most > efficient" way to benefit from a second cpu, both in terms of coding > time and execution efficiency, is to use either jython *cough* *choke* Err, no, sorry, not for me. > Although, iff yo

Re: Parallel Python on PowerMac?

[Wolfgang Keller] > as I might get a dual-G5 PowerMac someday in the not to distant > future, I was wondering what options are available for making Python > benefit from the second CPU? Running two interpreters and using Pyro > would not be the most efficient (and easiest) way, I guess? Qualifier: