Nathaniel Smith wrote: > On Thu, Mar 12, 2009 at 9:51 AM, Mark Larsen <larsen...@gmail.com> wrote: >> I took your advice and implemented using "Parallel Python". Can you comment >> on your choice of pyprocessing? It seems like a simple re-write to switch >> over. > > I use pyprocessing partly because it gave me the impression that it > was more tuned in to the general python community... but mostly > because it's just what I happened to bump into first. I'm sure they > both work fine; whatever works for you... (I've also used IPython's > distributed computing stuff, and it was nice too.) > > The fact that pyprocessing just forks has sometimes come in handy -- > in one case I had a giant multi-gigabyte structure that I needed to > pull data out of, so I loaded it once to a global variable in the > parent process before spawning the workers, and they all shared the > data structure via COW memory without any extra work on my part. (This > is an ugly Unix-specific hack, of course, but an effective one!) And > I've sometimes had trouble when calling into rpy2 with memory usage > growing unboundedly (still need to track this down),
I have experienced a growing process when running rpy2 for a long time (although this was not with multiprocessing). An explicit request for garbage collection on the Python side (import gc; gc.collect()) seem to have solved the issue so far (I have not found the time to look closely at it). > and in that case > it's really handy that it's cheap and easy to just restart the slaves. > > -- Nathaniel > > ------------------------------------------------------------------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > rpy-list mailing list > rpy-list@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rpy-list ------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ rpy-list mailing list rpy-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/rpy-list
