This thread discussed R memory limitations, compared handling with S and SAS.
Since I routinely use R to process multi-gigababyte sets on computers with
sometimes 256mb of memory - here are some comments on that.
Most memory limitations vanish if R is used with any relational database. [My
per
(Ted Harding) wrote:
> On 12-Apr-07 10:14:21, Jim Lemon wrote:
>
>>Charilaos Skiadas wrote:
>>
>>>A new fortune candidate perhaps?
>>>
>>>On Apr 10, 2007, at 6:27 PM, Greg Snow wrote:
>>>
>>>
>>>
Remember, everything is better than everything else given the
right comparison.
>>
>>Only
On 12-Apr-07 10:14:21, Jim Lemon wrote:
> Charilaos Skiadas wrote:
>> A new fortune candidate perhaps?
>>
>> On Apr 10, 2007, at 6:27 PM, Greg Snow wrote:
>>
>>
>>>Remember, everything is better than everything else given the
>>>right comparison.
>>>
> Only if we remove the grammatical blip that
Lucke, Joseph F writes:
> A re-interpretation of Zorn's lemma?
>
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Jim Lemon
> Sent: Thursday, April 12, 2007 5:14 AM
> To: [EMAIL PROTECTED]
> Subject: Re: [R] Reas
A re-interpretation of Zorn's lemma?
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Jim Lemon
Sent: Thursday, April 12, 2007 5:14 AM
To: [EMAIL PROTECTED]
Subject: Re: [R] Reasons to Use R
Charilaos Skiadas wrote:
> A new fortune candidate
Douglas Bates writes:
> One
> can do data analysis by using the computer as a blunt instrument with
> which to bludgeon the problem to death but one can't do elegant data
> analysis like that.
One nice thing about a "blunt instrument" like Stata is the ability to
hold an entire dataset in memo
Charilaos Skiadas wrote:
> A new fortune candidate perhaps?
>
> On Apr 10, 2007, at 6:27 PM, Greg Snow wrote:
>
>
>>Remember, everything is better than everything else given the right
>>comparison.
>>
Only if we remove the grammatical blip that turns it into an infinite
regress, i.e.
"Remember
On Wed, 11 Apr 2007, Alan Zaslavsky wrote:
> I have thought for a long time that a facility for efficient rowwise
> calculations might be a valuable enhancement to S/R. The storage of the
> object would be handled by a database and there would have to be an
> efficient interface for pulling a row
On 4/11/07, Robert Duval <[EMAIL PROTECTED]> wrote:
> So I guess my question is...
>
> Is there any hope of R being modified on its core in order to handle
> more graciously large datasets? (You've mentioned SAS and SPSS, I'd
> add Stata to the list).
>
> Or should we (the users of large datasets)
I think the reason that stata is fast is because it only keeps 1 work
table in ram. if you just keep 1 data frame in R, it will run fast
too. But ...
On 4/11/07, Robert Duval <[EMAIL PROTECTED]> wrote:
> So I guess my question is...
>
> Is there any hope of R being modified on its core in order to
So I guess my question is...
Is there any hope of R being modified on its core in order to handle
more graciously large datasets? (You've mentioned SAS and SPSS, I'd
add Stata to the list).
Or should we (the users of large datasets) expect to keep on working
with the present tools for the time to
On Wed, 2007-04-11 at 11:26 -0500, Marc Schwartz wrote:
> On Wed, 2007-04-11 at 17:56 +0200, Bi-Info
> (http://members.home.nl/bi-info) wrote:
> > I certainly have that idea too. SPSS functions in a way the same,
> > although it specialises in PC applications. Memory addition to a PC is
> > not a
model has been modified in
> later versions.
>
>
>
>
>
> > On 4/10/07, Greg Snow <[EMAIL PROTECTED]> wrote:
> > > > -Original Message-
> > > > From: [EMAIL PROTECTED]
> > > > [mailto:[EMAIL PROTECTED] On Behalf Of Bi-Info
&
thanks, I will take a look.
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible co
On Wed, 2007-04-11 at 17:56 +0200, Bi-Info
(http://members.home.nl/bi-info) wrote:
> I certainly have that idea too. SPSS functions in a way the same,
> although it specialises in PC applications. Memory addition to a PC is
> not a very expensive thing these days. On my first AT some extra memory
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Alan Zaslavsky
> Sent: Wednesday, April 11, 2007 9:07 AM
> To: R-help@stat.math.ethz.ch
> Subject: [R] Reasons to Use R
[snip]
> I have thought for a long time that a f
Rajarshi Guha wrote:
> On Wed, 2007-04-11 at 11:06 -0400, Alan Zaslavsky wrote:
>
> > I have thought for a long time that a facility for efficient rowwise
> > calculations might be a valuable enhancement to S/R. The storage of the
> > object would be handled by a database and there would have t
On Wed, 2007-04-11 at 11:06 -0400, Alan Zaslavsky wrote:
> I have thought for a long time that a facility for efficient rowwise
> calculations might be a valuable enhancement to S/R. The storage of the
> object would be handled by a database and there would have to be an
> efficient interface
I certainly have that idea too. SPSS functions in a way the same,
although it specialises in PC applications. Memory addition to a PC is
not a very expensive thing these days. On my first AT some extra memory
cost 300 dollars or more. These days you get extra memory with a package
of marshmello
cient calculations with large datasets.
Alan Zaslavsky
[EMAIL PROTECTED]
> Date: Tue, 10 Apr 2007 16:27:50 -0600
> From: "Greg Snow" <[EMAIL PROTECTED]>
> Subject: Re: [R] Reasons to Use R
> To: "Wensui Liu" <[EMAIL PROTECTED]>
>
> I think SAS
Certainly true. In particular, SAS was designed from to store
data items on disk, and to read into core memory the minimum
needed for a particular calculation.
The kind of data SAS handles is (for the most part) limited to
rectangular arrays, similar to R data frames. In many procedures
they can
; > > -Original Message-
> > > From: [EMAIL PROTECTED]
> > > [mailto:[EMAIL PROTECTED] On Behalf Of
> > > Bi-Info (http://members.home.nl/bi-info)
> > > Sent: Monday, April 09, 2007 4:23 PM
> > > To: Gabor Grothendieck
> > &g
A new fortune candidate perhaps?
On Apr 10, 2007, at 6:27 PM, Greg Snow wrote:
> Remember, everything is better than everything else given the right
> comparison.
>
> --
> Gregory (Greg) L. Snow Ph.D.
Haris Skiadas
Department of Mathematics and Computer Science
Hanover College
t; Cc: Bi-Info (http://members.home.nl/bi-info); Gabor
> Grothendieck; Lorenzo Isella; r-help@stat.math.ethz.ch
> Subject: Re: [R] Reasons to Use R
>
> Greg,
> As far as I understand, SAS is more efficient handling large
> data probably than S+/R. Do you have an
alf Of
> > Bi-Info (http://members.home.nl/bi-info)
> > Sent: Monday, April 09, 2007 4:23 PM
> > To: Gabor Grothendieck
> > Cc: Lorenzo Isella; r-help@stat.math.ethz.ch
> > Subject: Re: [R] Reasons to Use R
>
> [snip]
>
> > So what's the big deal about S us
Taylor, Z Todd wrote:
> On Monday, April 09, 2007 3:23 PM, someone named Wilfred wrote:
>
>> So what's the big deal about S using files instead of memory
>> like R. I don't get the point. Isn't there enough swap space
>> for S? (Who cares anyway: it works, isn't it?) Or are there
>> any problems w
Hi Todd,
I guess I don't see the difference between that strategy and using
make to look after scripts, raw data, Sweave files, and (if necessary)
images. I find that I can get pretty fine-grained control over what
parts of a project need to be rerun by breaking the analysis into
chapters. I sup
On Monday, April 09, 2007 3:23 PM, someone named Wilfred wrote:
> So what's the big deal about S using files instead of memory
> like R. I don't get the point. Isn't there enough swap space
> for S? (Who cares anyway: it works, isn't it?) Or are there
> any problems with S and large datasets? I do
nfo)
> > > Sent: Monday, April 09, 2007 4:23 PM
> > > To: Gabor Grothendieck
> > > Cc: Lorenzo Isella; r-help@stat.math.ethz.ch
> > > Subject: Re: [R] Reasons to Use R
> >
> > [snip]
> >
> > > So what's the big deal about S using
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of
> Bi-Info (http://members.home.nl/bi-info)
> Sent: Monday, April 09, 2007 4:23 PM
> To: Gabor Grothendieck
> Cc: Lorenzo Isella; r-help@stat.math.ethz.ch
> Subject: Re: [R]
EMAIL PROTECTED]
(801) 408-8111
> -Original Message-
> From: Charilaos Skiadas [mailto:[EMAIL PROTECTED]
> Sent: Monday, April 09, 2007 3:24 PM
> To: Greg Snow
> Cc: Gabor Grothendieck; Lorenzo Isella; R-Help list
> Subject: Re: [R] Reasons to Use R
>
> On Apr 9, 2007
"halldor bjornsson" <[EMAIL PROTECTED]> writes:
> ...
> Now, R does not have everything we want. One thing missing is a decent
> R-DB2 connection, for windows the excellent RODBC works fine, but ODBC
> support on Linux is a hassle.
>
A hassle? I use RODBC on Linux to read data from a mainf
e problem is the DB size.
> >
> > -Mensaje original-
> > De: Gabor Grothendieck [mailto:[EMAIL PROTECTED]
> > Enviado el: Lunes, 09 de Abril de 2007 11:28
> > Para: Jorge Cornejo-Donoso
> > CC: r-help@stat.math.ethz.ch
> > Asunto: Re: [R] Reasons to
e problem is the DB size.
>
> -Mensaje original-
> De: Gabor Grothendieck [mailto:[EMAIL PROTECTED]
> Enviado el: Lunes, 09 de Abril de 2007 11:28
> Para: Jorge Cornejo-Donoso
> CC: r-help@stat.math.ethz.ch
> Asunto: Re: [R] Reasons to Use R
>
> Have you tried 64 bit mac
eg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
[EMAIL PROTECTED]
(801) 408-8111
> -Original Message-
> From: Gabor Grothendieck [mailto:[EMAIL PROTECTED]
> Sent: Monday, April 09, 2007 10:44 AM
> To: Greg Snow
> Cc: Lorenzo Isella; r-help@stat.math.ethz.c
On Apr 9, 2007, at 1:45 PM, Greg Snow wrote:
> The licences keep changing, some have in the past but don't now, some
> you can get an additional licence for home at a discounted price. Some
> it depends on the type of licence you have at work (currently our SAS
> licence is such that the 3 people
Dear Lorenzo,
Thanks for starting a great thread here. Like others, I would like to
hear a summary
if you make one.
My institute uses R for internal data processing and analyzing. Below
are some of our reasons, and yes cost (or lack thereof) is not the
only one.
First, prior to the rise of R we
From: Gabor Grothendieck [mailto:[EMAIL PROTECTED]
> Sent: Monday, April 09, 2007 10:44 AM
> To: Greg Snow
> Cc: Lorenzo Isella; r-help@stat.math.ethz.ch
> Subject: Re: [R] Reasons to Use R
>
> I might be wrong about this but I thought that the licenses
> for at least some o
,
>
>
> --
> Gregory (Greg) L. Snow Ph.D.
> Statistical Data Center
> Intermountain Healthcare
> [EMAIL PROTECTED]
> (801) 408-8111
>
>
>
> > -Original Message-
> > From: [EMAIL PROTECTED]
> > [mailto:[EMAIL PROTECTED] On Behalf Of Lo
CTED]
(801) 408-8111
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Lorenzo Isella
> Sent: Thursday, April 05, 2007 9:02 AM
> To: r-help@stat.math.ethz.ch
> Subject: [R] Reasons to Use R
>
> Dear All,
> The institute
l-
> De: Gabor Grothendieck [mailto:[EMAIL PROTECTED]
> Enviado el: Lunes, 09 de Abril de 2007 11:28
> Para: Jorge Cornejo-Donoso
> CC: r-help@stat.math.ethz.ch
> Asunto: Re: [R] Reasons to Use R
>
> Have you tried 64 bit machines with larger memory or do you mean that you
to Use R
Have you tried 64 bit machines with larger memory or do you mean that you
can't use R on your current machines?
Also have you tried S-Plus? Will that work for you? The transition from
that to R would be less than from SAS to R.
On 4/9/07, Jorge Cornejo-Donoso <[EMAIL PROTECTED
Have you tried 64 bit machines with larger memory or do you mean
that you can't use R on your current machines?
Also have you tried S-Plus? Will that work for you? The transition from
that to R would be less than from SAS to R.
On 4/9/07, Jorge Cornejo-Donoso <[EMAIL PROTECTED]> wrote:
> tha s9z
Dear Johann and Gabor,
It's what amounts to large datasets. There are hundreds of datasets R
can't handle, probably thousands or more. I noticed on my computer
(which is nothing more that an average PC) that R breaks down after 250
MB of memory. I also note that SPSS breaks down, Matlab, etc.
I'm
On 4/8/07, Johann Hibschman <[EMAIL PROTECTED]> wrote:
> R's pass-by-value semantics also make it harder than it should be to
> deal with where it's crucial that you not make a copy of the data
> frame, for fear of running out of memory. Pass-by-reference would
> make implementing data transformat
On 4/6/07, Wilfred Zegwaard <[EMAIL PROTECTED]> wrote:
> I'm not a programmer, but I have the experience that R is good for
> processing large datasets, especially in combination with specialised
> statistics.
This I find a little surprising, but maybe it's just a sign that I'm
not experienced en
Regarding (2),
I wonder if this information is too outdated or not relevant when scaled up
to larger problems...
http://www.sciviews.org/benchmark/index.html
--- Ramon Diaz-Uriarte <[EMAIL PROTECTED]> wrote:
> Dear Lorenzo,
>
> I'll try not to repeat what other have answered before.
>
> On
Dear Lorenzo and Steven,
I'm not a programmer, but I have the experience that R is good for
processing large datasets, especially in combination with specialised
statistics. There are some limits to that, but R handles large datasets
/ complicated computation a lot better that SPSS for example. I
Dear Lorenzo,
I'll try not to repeat what other have answered before.
On 4/5/07, Lorenzo Isella <[EMAIL PROTECTED]> wrote:
> The institute I work for is organizing an internal workshop for High
> Performance Computing (HPC).
(...)
> (1)Institutions (not only academia) using R
You can count my i
Hi Lorenzo,
On 4/5/07, Lorenzo Isella <[EMAIL PROTECTED]> wrote:
>
> I would like to have suggestions about where to collect info about:
> (1)Institutions (not only academia) using R
A starting point might be to look at the R-project homepage and look at the
members and donors list. This is, of
PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Lorenzo Isella
> Sent: Thursday, April 05, 2007 11:02 AM
> To: r-help@stat.math.ethz.ch
> Subject: [R] Reasons to Use R
>
> Dear All,
> The institute I work for is organizing an internal workshop for High
> Performance Comput
Hi Lorenzo,
I don't think I'm qualified to provide solid information on the first
three questions, but I'd like to drop a few thoughts on (4). While
there are no shortage of language advocates out there, I'd like to
join in for this once. My background is in chemical engineering and
atmospheric sc
As to my knowledge the core of R is considered "adequate" and "good" by
the statisticians. That's sufficient isn't it?
Last year I read some documentation about R and most routines were
considered "good", but "some very bad". That is a benchmark somehow.
There must be some benchmarks you want. R i
John Kane wrote:
> --- Lorenzo Isella <[EMAIL PROTECTED]> wrote:
>
>
>> (4)finally, a list of the advantages for using R
>> over commercial
>> statistical packages. The money-saving in itself is
>> not a reason good
>> enough and some people are scared by the lack of
>> professional support,
>>
--- Lorenzo Isella <[EMAIL PROTECTED]> wrote:
>
> (4)finally, a list of the advantages for using R
> over commercial
> statistical packages. The money-saving in itself is
> not a reason good
> enough and some people are scared by the lack of
> professional support,
> though this mailing list is s
Lorenzo Isella writes:
> (4)finally, a list of the advantages for using R over commercial
> statistical packages.
Here's my entry on the list, as this was a topic of conversation over
lunch: it's better than the proprietary statistical software I use
most of the time. By better I mean that the
r.de
-Ursprüngliche Nachricht-
Von: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Im Auftrag von Lorenzo Isella
Gesendet: Donnerstag, 5. April 2007 17:02
An: r-help@stat.math.ethz.ch
Betreff: [R] Reasons to Use R
Dear All,
The institute I work for is organizing an internal workshop for
Dear All,
The institute I work for is organizing an internal workshop for High
Performance Computing (HPC).
I am planning to attend it and talk a bit about fluid dynamics, but
there is also quite a lot of interest devoted to data post-processing
and management of huge data sets.
A lot of people are
58 matches
Mail list logo
