On Tue, Sep 29, 2009 at 12:21 PM, Gary Strangman wrote: > > Hi Laurent, > > The only way to reduce the number of transformations is to add an > equivalent number of columns to the dataframe (so that instead of several > hundred thousand conversions, I need several hundred thousand columns), > and then passing this beast back-and-forth between python and R for > regression fits. I figured that was completely untenable.
Indeed - passing loads of data back and forth between Python and R is a bad idea (for performance and likely memory usage). In this case you want to use R for regressions, so you probably only need the data loaded in R. If you already have the raw data in a format R can read, I would suggest you apply the KISS principle. By that I mean (from Python via rpy or pry2), tell R to load this data using its native file parsing to give a dataframe (which should be kept as an R object). I did something similar a few years back (with rpy 1), and while I tried various ways of passing the data to R (lists, dicts, numpy, etc) for me the simplest/most robust solution was very low tech: I used Python to write a plain text tab separated (or comma - whatever you like best) file of the data, and then (from Python via rpy) got R to load this. [In this case, having the data dumped to a simple file was actually a useful bonus - very handy for debugging, for example via an R shell.] However, if you can get this working nicely with rpy2, I would be interested to see the benchmarks for the different approaches. Peter ------------------------------------------------------------------------------ Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf _______________________________________________ rpy-list mailing list rpy-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/rpy-list
