Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-15 Thread Gabor Grothendieck
On Fri, Oct 15, 2010 at 6:14 AM, Chris Howden ch...@trickysolutions.com.au wrote: Thanks for the advice Gabor, I was indeed not starting and finishing with sqldf(). Which was why it was not working for me. Please forgive a blatantly obvious mistake. I have tried what U suggested and

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-15 Thread Chris Howden
7878 ch...@trickysolutions.com.au -Original Message- From: Gabor Grothendieck [mailto:ggrothendi...@gmail.com] Sent: Friday, 15 October 2010 1:03 PM To: Chris Howden Cc: r-help@r-project.org Subject: Re: [R] merging and working with BIG data sets. Is sqldf the best way?? On Thu, Oct 14

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-14 Thread Chris Howden
...@gmail.com] Sent: Thursday, 14 October 2010 9:02 AM To: Chris Howden Cc: r-help@r-project.org Subject: Re: [R] merging and working with BIG data sets. Is sqldf the best way?? On Tue, Oct 12, 2010 at 2:39 AM, Chris Howden ch...@trickysolutions.com.au wrote: I’m working with some very big datasets

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-14 Thread Gabor Grothendieck
On Thu, Oct 14, 2010 at 10:56 PM, Chris Howden ch...@trickysolutions.com.au wrote: Thanks for the suggestion and code Gabor, I've tried creating 2 indices: 1) just for the variable I intend to merge on 2) on the entire data set I am merging (which I think is the one I should be using??)

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-13 Thread Gabor Grothendieck
On Tue, Oct 12, 2010 at 2:39 AM, Chris Howden ch...@trickysolutions.com.au wrote: I’m working with some very big datasets (each dataset has 11 million rows and 2 columns). My first step is to merge all my individual data sets together (I have about 20) I’m using the following command from

[R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-12 Thread Chris Howden
Hi everyone, I’m working with some very big datasets (each dataset has 11 million rows and 2 columns). My first step is to merge all my individual data sets together (I have about 20) I’m using the following command from sqldf data1 - sqldf(select A.*, B.* from A inner join B

Re: [R] merging and working with BIG data sets. Is sqldf the best way??

2010-10-12 Thread Jeff Newmiller
Chris Howden wrote: Hi everyone, I’m working with some very big datasets (each dataset has 11 million rows and 2 columns). My first step is to merge all my individual data sets together (I have about 20) I’m using the following command from sqldf data1 - sqldf(select A.*, B.*

Re: [R] merging and working with big data sets

2010-10-12 Thread Jay Emerson
I can't speak for ff and filehash, but bigmemory's data structure doesn't allow clever merges (for actually good reasons). However, it is still probably less painful (and faster) than other options, though we don't implement it: we leave it to the user because details may vary depending on the