Hi Laurent,
The only way to reduce the number of transformations is to add an
equivalent number of columns to the dataframe (so that instead of several
hundred thousand conversions, I need several hundred thousand columns),
and then passing this beast back-and-forth between python and R for
regression fits. I figured that was completely untenable.
Could you give an example of how to convert a variable such as d (below)
to a dataframe, with column names in colnames (below), using the
rinterface? I'm capable with R and python, but I'm thoroughly confused on
what the best order is for conversion. For example, I recall that (at one
point anyway) converting a list of lists like mine required something
like:
1) d_transpose = zip(*d), to change the list-of-lists from row-wise to
column-wise
2) apply array.array() to each tuple in d_transpose (for which where you
need to know the data types in advance)
3) loop to make a dictionary out of the colnames plus the list of
array.array()s
4) pass the dictionary to RDataFrame()
Without benchmarking, that seems mighty inefficient. Nathaniel Smith's
rnumpy mostly allows the following:
df = rnumpy.r.data_frame(numpy.array(d,np.object))
... which is 2 conversions (rather than 4), but I haven't been able to get
the column names attached in this case. (My inexperience, I'm sure.)
I'm hoping to make this general-purpose, so I may not know in advance
whether column 1 will be a StrSexpVector or an IntSexpVector or whatever.
Any code-samples (I'm happy to run benchmarks on a set of options and
provide them for your site) using the trivial datastructure immediately
below would /really/ help.
d = [['S80', 'C', 137.5, 0, 1],
['S82', 'C', 155.1, 1, 3],
['S83', 'T', 11.96, 0, 5],
['S84', 'T', 47, 1, 6]]
['S85', 'T', numpy.nan, 1, 31]]
colnames = ['code','pop','score','boolflag','counts']
-best
Gary
On Tue, 29 Sep 2009, Laurent Gautier wrote:
> Gary,
>
> Two things come to my mind:
>
> - Try having an initial Python data structure that requires less
> transformations than your current one in order to become a DataFrame.
>
> - Use rpy2.rinterface when speed matters. This can already get you faster
> than R.
> http://rpy.sourceforge.net/rpy2/doc-dev/html/performances.html
>
>
> L.
>
>
>
>
> Gary Strangman wrote:
>> Hi all,
>>
>> I have a python list of lists (each sublist is a row of data), plus a list
>> of column names. Something like this ...
>>
>>>>> d = [['S80', 'C', 137.5, 0],
>> ['S82', 'C', 155.1, 1],
>> ['S83', 'T', 11.96, 0],
>> ['S84', 'T', 47, 1]]
>> ['S85', 'T', numpy.nan, 1]]
>>>>> colnames = ['code','pop','score','flag']
>>
>> I'm looking for the /fastest/ way to create an R dataframe (via rpy2) using
>> these two variables. It could be via dictionaries, numpy object arrays,
>> whatever, it just needs to be fast. Note that the data has mixed types
>> (some columns are strings, some are floats, some are ints), and there are
>> missing values which I'd like R to interpret as NA. I can pre-transform the
>> elements of the d variable as required to facilitate this.
>>
>> I need to do this step several hundred thousand times (yes, different data
>> each time) on up to ~10,000 element datasets, so any speedup suggestions
>> are welcome.
>>
>> -best
>> Gary
>>
>>
>> ------------------------------------------------------------------------------
>> Come build with us! The BlackBerry® Developer Conference in SF, CA
>> is the only developer event you need to attend this year. Jumpstart your
>> developing skills, take BlackBerry mobile applications to market and stay
>> ahead of the curve. Join us from November 9-12, 2009. Register now!
>> http://p.sf.net/sfu/devconf
>> _______________________________________________
>> rpy-list mailing list
>> rpy-list@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/rpy-list
>
>
>
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
rpy-list mailing list
rpy-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rpy-list
