Re: [R] Limit
Às 00:58 de 09/11/2024, Val escreveu: Hi All, I am reading data file ( > 1B rows) and do some date formatting like dat=fread(mydatafile) dat$date1 <- as.Date(ymd(dat$date1)) However, I am getting an error message saying that Error: cons memory exhausted (limit reached?) The script was working when the number rows were around 650M. Is there another way to handle a big data set in R? Thank you. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. Hello, fread works but as.Date(ymd(.)) does not? You probably don't need both date coercion functions, get rid of one of them and try again. dat$date1 <- ymd(dat$date1) or dat$date1 <- as.Date(dat$date1) Hope this helps, Rui Barradas -- Este e-mail foi analisado pelo software antivírus AVG para verificar a presença de vírus. www.avg.com __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
Thank you, I will take a look. On Fri, Nov 8, 2024 at 8:09 PM Ben Bolker wrote: > > Check the "high performance task view" on CRAN ... > https://cran.r-project.org/web/views/HighPerformanceComputing.html > > On Fri, Nov 8, 2024, 7:58 PM Val wrote: >> >> Hi All, >> >> I am reading data file ( > 1B rows) and do some date formatting like >> dat=fread(mydatafile) >> dat$date1 <- as.Date(ymd(dat$date1)) >> >> However, I am getting an error message saying that >> Error: cons memory exhausted (limit reached?) >> >> The script was working when the number rows were around 650M. >> >> Is there another way to handle a big data set in R? >> >> >> Thank you. >> >> __ >> [email protected] mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide https://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
Thank you Jeff for the tip! I don't think I have 4 times as much free memory to process data... . I allocated the max memory of the system.has. On Fri, Nov 8, 2024 at 8:30 PM Jeff Newmiller wrote: > > There is always an implied "and do computations on it before writing the > processed data out" when reading chunks of a file. > > And you would almost certainly not be getting that error if you were not out > of memory. A good rule of thumb is that you need 4 times as much free memory > to process data than you need to read it in. > > On November 8, 2024 6:08:16 PM PST, Val wrote: > >Hi Jeff, > > > >Memory was not an issue. The system only used 75% of the memory > >allocated for the job. > > > > I am trying to understand what "r read large file in chunks" is doing. > > > >On Fri, Nov 8, 2024 at 7:50 PM Jeff Newmiller > >wrote: > >> > >> Then you don't have enough memory to process the whole thing at once. Not > >> unlike stuffing your mouth with cookies and not being able to chew for > >> lack of space to move the food around in your mouth. > >> > >> Now, can you answer my question? > >> > >> On November 8, 2024 5:38:37 PM PST, Val wrote: > >> >The data was read. The problem is with processing. > >> > > >> >On Fri, Nov 8, 2024 at 7:30 PM Bert Gunter wrote: > >> >> > >> >> Is the problem reading the file in or processing it after it has been > >> >> read in? > >> >> > >> >> Bert > >> >> > >> >> On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help > >> >> wrote: > >> >>> > >> >>> Can you tell us what is wrong with the "chunked" package which comes > >> >>> up when you Google "r read large file in chunks"? > >> >>> > >> >>> On November 8, 2024 4:58:18 PM PST, Val wrote: > >> >>> >Hi All, > >> >>> > > >> >>> >I am reading data file ( > 1B rows) and do some date formatting like > >> >>> > dat=fread(mydatafile) > >> >>> > dat$date1 <- as.Date(ymd(dat$date1)) > >> >>> > > >> >>> >However, I am getting an error message saying that > >> >>> >Error: cons memory exhausted (limit reached?) > >> >>> > > >> >>> >The script was working when the number rows were around 650M. > >> >>> > > >> >>> >Is there another way to handle a big data set in R? > >> >>> > > >> >>> > > >> >>> >Thank you. > >> >>> > > >> >>> >__ > >> >>> >[email protected] mailing list -- To UNSUBSCRIBE and more, see > >> >>> >https://stat.ethz.ch/mailman/listinfo/r-help > >> >>> >PLEASE do read the posting guide > >> >>> >https://www.R-project.org/posting-guide.html > >> >>> >and provide commented, minimal, self-contained, reproducible code. > >> >>> > >> >>> -- > >> >>> Sent from my phone. Please excuse my brevity. > >> >>> > >> >>> __ > >> >>> [email protected] mailing list -- To UNSUBSCRIBE and more, see > >> >>> https://stat.ethz.ch/mailman/listinfo/r-help > >> >>> PLEASE do read the posting guide > >> >>> https://www.R-project.org/posting-guide.html > >> >>> and provide commented, minimal, self-contained, reproducible code. > >> > >> -- > >> Sent from my phone. Please excuse my brevity. > > -- > Sent from my phone. Please excuse my brevity. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
There is always an implied "and do computations on it before writing the processed data out" when reading chunks of a file. And you would almost certainly not be getting that error if you were not out of memory. A good rule of thumb is that you need 4 times as much free memory to process data than you need to read it in. On November 8, 2024 6:08:16 PM PST, Val wrote: >Hi Jeff, > >Memory was not an issue. The system only used 75% of the memory >allocated for the job. > > I am trying to understand what "r read large file in chunks" is doing. > >On Fri, Nov 8, 2024 at 7:50 PM Jeff Newmiller wrote: >> >> Then you don't have enough memory to process the whole thing at once. Not >> unlike stuffing your mouth with cookies and not being able to chew for lack >> of space to move the food around in your mouth. >> >> Now, can you answer my question? >> >> On November 8, 2024 5:38:37 PM PST, Val wrote: >> >The data was read. The problem is with processing. >> > >> >On Fri, Nov 8, 2024 at 7:30 PM Bert Gunter wrote: >> >> >> >> Is the problem reading the file in or processing it after it has been >> >> read in? >> >> >> >> Bert >> >> >> >> On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help >> >> wrote: >> >>> >> >>> Can you tell us what is wrong with the "chunked" package which comes up >> >>> when you Google "r read large file in chunks"? >> >>> >> >>> On November 8, 2024 4:58:18 PM PST, Val wrote: >> >>> >Hi All, >> >>> > >> >>> >I am reading data file ( > 1B rows) and do some date formatting like >> >>> > dat=fread(mydatafile) >> >>> > dat$date1 <- as.Date(ymd(dat$date1)) >> >>> > >> >>> >However, I am getting an error message saying that >> >>> >Error: cons memory exhausted (limit reached?) >> >>> > >> >>> >The script was working when the number rows were around 650M. >> >>> > >> >>> >Is there another way to handle a big data set in R? >> >>> > >> >>> > >> >>> >Thank you. >> >>> > >> >>> >__ >> >>> >[email protected] mailing list -- To UNSUBSCRIBE and more, see >> >>> >https://stat.ethz.ch/mailman/listinfo/r-help >> >>> >PLEASE do read the posting guide >> >>> >https://www.R-project.org/posting-guide.html >> >>> >and provide commented, minimal, self-contained, reproducible code. >> >>> >> >>> -- >> >>> Sent from my phone. Please excuse my brevity. >> >>> >> >>> __ >> >>> [email protected] mailing list -- To UNSUBSCRIBE and more, see >> >>> https://stat.ethz.ch/mailman/listinfo/r-help >> >>> PLEASE do read the posting guide >> >>> https://www.R-project.org/posting-guide.html >> >>> and provide commented, minimal, self-contained, reproducible code. >> >> -- >> Sent from my phone. Please excuse my brevity. -- Sent from my phone. Please excuse my brevity. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
Is the problem reading the file in or processing it after it has been read in? Bert On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help < [email protected]> wrote: > Can you tell us what is wrong with the "chunked" package which comes up > when you Google "r read large file in chunks"? > > On November 8, 2024 4:58:18 PM PST, Val wrote: > >Hi All, > > > >I am reading data file ( > 1B rows) and do some date formatting like > > dat=fread(mydatafile) > > dat$date1 <- as.Date(ymd(dat$date1)) > > > >However, I am getting an error message saying that > >Error: cons memory exhausted (limit reached?) > > > >The script was working when the number rows were around 650M. > > > >Is there another way to handle a big data set in R? > > > > > >Thank you. > > > >__ > >[email protected] mailing list -- To UNSUBSCRIBE and more, see > >https://stat.ethz.ch/mailman/listinfo/r-help > >PLEASE do read the posting guide > https://www.R-project.org/posting-guide.html > >and provide commented, minimal, self-contained, reproducible code. > > -- > Sent from my phone. Please excuse my brevity. > > __ > [email protected] mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > https://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
Check the "high performance task view" on CRAN ... https://cran.r-project.org/web/views/HighPerformanceComputing.html On Fri, Nov 8, 2024, 7:58 PM Val wrote: > Hi All, > > I am reading data file ( > 1B rows) and do some date formatting like > dat=fread(mydatafile) > dat$date1 <- as.Date(ymd(dat$date1)) > > However, I am getting an error message saying that > Error: cons memory exhausted (limit reached?) > > The script was working when the number rows were around 650M. > > Is there another way to handle a big data set in R? > > > Thank you. > > __ > [email protected] mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > https://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
Hi Jeff, Memory was not an issue. The system only used 75% of the memory allocated for the job. I am trying to understand what "r read large file in chunks" is doing. On Fri, Nov 8, 2024 at 7:50 PM Jeff Newmiller wrote: > > Then you don't have enough memory to process the whole thing at once. Not > unlike stuffing your mouth with cookies and not being able to chew for lack > of space to move the food around in your mouth. > > Now, can you answer my question? > > On November 8, 2024 5:38:37 PM PST, Val wrote: > >The data was read. The problem is with processing. > > > >On Fri, Nov 8, 2024 at 7:30 PM Bert Gunter wrote: > >> > >> Is the problem reading the file in or processing it after it has been read > >> in? > >> > >> Bert > >> > >> On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help > >> wrote: > >>> > >>> Can you tell us what is wrong with the "chunked" package which comes up > >>> when you Google "r read large file in chunks"? > >>> > >>> On November 8, 2024 4:58:18 PM PST, Val wrote: > >>> >Hi All, > >>> > > >>> >I am reading data file ( > 1B rows) and do some date formatting like > >>> > dat=fread(mydatafile) > >>> > dat$date1 <- as.Date(ymd(dat$date1)) > >>> > > >>> >However, I am getting an error message saying that > >>> >Error: cons memory exhausted (limit reached?) > >>> > > >>> >The script was working when the number rows were around 650M. > >>> > > >>> >Is there another way to handle a big data set in R? > >>> > > >>> > > >>> >Thank you. > >>> > > >>> >__ > >>> >[email protected] mailing list -- To UNSUBSCRIBE and more, see > >>> >https://stat.ethz.ch/mailman/listinfo/r-help > >>> >PLEASE do read the posting guide > >>> >https://www.R-project.org/posting-guide.html > >>> >and provide commented, minimal, self-contained, reproducible code. > >>> > >>> -- > >>> Sent from my phone. Please excuse my brevity. > >>> > >>> __ > >>> [email protected] mailing list -- To UNSUBSCRIBE and more, see > >>> https://stat.ethz.ch/mailman/listinfo/r-help > >>> PLEASE do read the posting guide > >>> https://www.R-project.org/posting-guide.html > >>> and provide commented, minimal, self-contained, reproducible code. > > -- > Sent from my phone. Please excuse my brevity. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
Then you don't have enough memory to process the whole thing at once. Not unlike stuffing your mouth with cookies and not being able to chew for lack of space to move the food around in your mouth. Now, can you answer my question? On November 8, 2024 5:38:37 PM PST, Val wrote: >The data was read. The problem is with processing. > >On Fri, Nov 8, 2024 at 7:30 PM Bert Gunter wrote: >> >> Is the problem reading the file in or processing it after it has been read >> in? >> >> Bert >> >> On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help >> wrote: >>> >>> Can you tell us what is wrong with the "chunked" package which comes up >>> when you Google "r read large file in chunks"? >>> >>> On November 8, 2024 4:58:18 PM PST, Val wrote: >>> >Hi All, >>> > >>> >I am reading data file ( > 1B rows) and do some date formatting like >>> > dat=fread(mydatafile) >>> > dat$date1 <- as.Date(ymd(dat$date1)) >>> > >>> >However, I am getting an error message saying that >>> >Error: cons memory exhausted (limit reached?) >>> > >>> >The script was working when the number rows were around 650M. >>> > >>> >Is there another way to handle a big data set in R? >>> > >>> > >>> >Thank you. >>> > >>> >__ >>> >[email protected] mailing list -- To UNSUBSCRIBE and more, see >>> >https://stat.ethz.ch/mailman/listinfo/r-help >>> >PLEASE do read the posting guide >>> >https://www.R-project.org/posting-guide.html >>> >and provide commented, minimal, self-contained, reproducible code. >>> >>> -- >>> Sent from my phone. Please excuse my brevity. >>> >>> __ >>> [email protected] mailing list -- To UNSUBSCRIBE and more, see >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> https://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. -- Sent from my phone. Please excuse my brevity. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
The data was read. The problem is with processing. On Fri, Nov 8, 2024 at 7:30 PM Bert Gunter wrote: > > Is the problem reading the file in or processing it after it has been read in? > > Bert > > On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help > wrote: >> >> Can you tell us what is wrong with the "chunked" package which comes up when >> you Google "r read large file in chunks"? >> >> On November 8, 2024 4:58:18 PM PST, Val wrote: >> >Hi All, >> > >> >I am reading data file ( > 1B rows) and do some date formatting like >> > dat=fread(mydatafile) >> > dat$date1 <- as.Date(ymd(dat$date1)) >> > >> >However, I am getting an error message saying that >> >Error: cons memory exhausted (limit reached?) >> > >> >The script was working when the number rows were around 650M. >> > >> >Is there another way to handle a big data set in R? >> > >> > >> >Thank you. >> > >> >__ >> >[email protected] mailing list -- To UNSUBSCRIBE and more, see >> >https://stat.ethz.ch/mailman/listinfo/r-help >> >PLEASE do read the posting guide >> >https://www.R-project.org/posting-guide.html >> >and provide commented, minimal, self-contained, reproducible code. >> >> -- >> Sent from my phone. Please excuse my brevity. >> >> __ >> [email protected] mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide https://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit
Can you tell us what is wrong with the "chunked" package which comes up when you Google "r read large file in chunks"? On November 8, 2024 4:58:18 PM PST, Val wrote: >Hi All, > >I am reading data file ( > 1B rows) and do some date formatting like > dat=fread(mydatafile) > dat$date1 <- as.Date(ymd(dat$date1)) > >However, I am getting an error message saying that >Error: cons memory exhausted (limit reached?) > >The script was working when the number rows were around 650M. > >Is there another way to handle a big data set in R? > > >Thank you. > >__ >[email protected] mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide https://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code. -- Sent from my phone. Please excuse my brevity. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit bar graph output
Bert Jeff I just resorted and took the top 30 and then reordered again in the geom_bar function – below ggplot(data=st.cnt)+ geom_bar(aes(x=reorder(CourseName, -n), y=n),fill = "dark blue", stat="identity")+ theme(axis.text.x = element_text(angle = 60, hjust = 1)) Jeff From: Bert Gunter Sent: Sunday, October 14, 2018 10:51 PM To: [email protected] Cc: R-help Subject: Re: [R] limit bar graph output If I understand correctly, just subset your sorted data. e.g. : x <- runif(50) ## 50 unsorted values sort(x, dec = TRUE)[1:10] ## the 10 biggest -- Bert Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Sun, Oct 14, 2018 at 7:13 PM Jeff Reichman mailto:[email protected]> > wrote: R-Help Forum I'm using the following code to reorder (from highest to lowest) my miRNA counts. But there are 500 plus and I only need the first (say) 15-20. How do I limit ggplot to only the first 20 miRNA counts ggplot(data = corr.m, aes(x = reorder(miRNA, -value), y = value, fill = variable)) + geom_bar(stat = "identity") Jeff __ [email protected] <mailto:[email protected]> mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit bar graph output
A reproducible example would help here (you cannot assume we know what type "miRNA" is) but guessing from the use of "reorder" I suspect it is a factor. In which case after you subset you will need to use the droplevels function to remove the unused levels, and then plot that prepared data. On October 14, 2018 8:51:29 PM PDT, Bert Gunter wrote: >If I understand correctly, just subset your sorted data. > >e.g. : > >x <- runif(50) >## 50 unsorted values > >sort(x, dec = TRUE)[1:10] >## the 10 biggest > > >-- Bert > > > >Bert Gunter > >"The trouble with having an open mind is that people keep coming along >and >sticking things into it." >-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) > > >On Sun, Oct 14, 2018 at 7:13 PM Jeff Reichman >wrote: > >> R-Help Forum >> >> I'm using the following code to reorder (from highest to lowest) my >miRNA >> counts. But there are 500 plus and I only need the first (say) >15-20. How >> do I limit ggplot to only the first 20 miRNA counts >> >> ggplot(data = corr.m, aes(x = reorder(miRNA, -value), y = value, fill >= >> variable)) + >> geom_bar(stat = "identity") >> >> Jeff >> >> __ >> [email protected] mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > [[alternative HTML version deleted]] > >__ >[email protected] mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code. -- Sent from my phone. Please excuse my brevity. __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit bar graph output
If I understand correctly, just subset your sorted data. e.g. : x <- runif(50) ## 50 unsorted values sort(x, dec = TRUE)[1:10] ## the 10 biggest -- Bert Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Sun, Oct 14, 2018 at 7:13 PM Jeff Reichman wrote: > R-Help Forum > > I'm using the following code to reorder (from highest to lowest) my miRNA > counts. But there are 500 plus and I only need the first (say) 15-20. How > do I limit ggplot to only the first 20 miRNA counts > > ggplot(data = corr.m, aes(x = reorder(miRNA, -value), y = value, fill = > variable)) + > geom_bar(stat = "identity") > > Jeff > > __ > [email protected] mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ [email protected] mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit in Directory Hierarchy?
This is the wrong place to ask what RStudio can or cannot do. However, if your
question is about R you should try invoking your reproducible example in RGui
or the command line R.exe before posting here.
R has no directory depth limit. There is an operating system limit on returning
paths more than 260 characters long, and this seems to create problems
sometimes (Google it). You might also look at operating system permissions for
the "0318.data" directory.
--
Sent from my phone. Please excuse my brevity.
On March 25, 2018 12:55:33 PM PDT, WRAY NICHOLAS via R-help
wrote:
>A quick question - is there a limit to the number of levels one can go
>down when setting the directory in R studio?
>I ask because I have been trying to set the directory to a folder 8
>levels down which R studio won't allow, and when I try to set the
>directory through Session/Set Working Directory, it won't even show the
>lowest level folder:
>
>setwd("C:/Users/supermollusc/Desktop/151017 Shellfish/Mussel
>Months/Mussel Mar 18/0318.data")
>
>but it's quite happy to set the directory at 7 levels
>
>setwd("C:/Users/supermollusc/Desktop/151017 Shellfish/Mussel
>Months/Mussel Mar 18")
>
>I can't see why it won't do this if there isn't a hierarchy level limit
>but I can't find reference to one on the net
>
>Thanks
>
>Nick Wray
>
> [[alternative HTML version deleted]]
>
>__
>[email protected] mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
__
[email protected] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit the y-axis line with ggplot2 (not the axis itself, but the line used at the left of the graph)
> Hello everybody,
>
> Using ggplot2 package, is there a way to force to stop the y-axis line
> at a specified point ? (not using ylim because I want that some text
> written using annotate() at the top of the graph is
> still shown).
>
> Bellow is a simple example to show what I would like do:
>
> Thanks a lot
>
> Marc
>
>
>
>
>
> library("ggplot2")
>
> g <- ggplot()+
> geom_point(aes(x=c(20, 29, 32), y=c(0, 0.4, 1)))+
> scale_y_continuous(breaks=c(0, 0.25, 0.5, 0.75, 1))+
> labs(x="Label for x axis")+
> labs(y="Label for y axis") +
> annotate("text", x = 25 , y=1.2, label="Text 1")+
> annotate("text", x = 22 , y=1.0, label="How to stop the y-axis line
>here !")+
> geom_segment(aes(x=25, xend=25, y=0, yend=1.1), linetype=2)+
> > This part is just to make it more nice
> theme(panel.background = element_rect(fill = 'white'),
> panel.grid.major = element_blank(),
> panel.grid.minor = element_blank(),
> plot.margin = unit(c(0.5, 1, 0.5, 0.5), "cm"),
> axis.text.x=element_text(size=14),
> axis.text.y=element_text(size=14),
> axis.title.x=element_text(size=18),
> axis.title.y=element_text(size=18),
> axis.ticks.length=unit(0.3,"cm"),
> panel.border = element_blank(),
> axis.line.x = element_line(),
> axis.line.y = element_line())
> g
Hey Marc,
I feel there's a better solution out there, but I think I got what
you're after by replacing the y-axis with a
`annotate(geom="segment")`, which I could manipulate using the y and
yend arguments.
It took some other finagling of the other elements though.
#after already defining g per the OP
g + scale_x_continuous(limits = c(19,32.5), expand = c(0,0)) +
scale_y_continuous(limits=c(-0.05,1.25), breaks=c(0, 0.25, 0.5,
0.75, 1), expand = c(0,0))+
annotate(geom = "segment", x=19, xend = 19, y = -0.05, yend = 1)+
theme(axis.line.y = element_blank())
Cheers,
Zach
__
[email protected] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit the y-axis line with ggplot2 (not the axis itself, but the line used at the left of the graph) [SOLVED]
Hi
Here are some alternatives that involve messing about with the grobs and
viewports after the plot has been drawn. The percentage optimality of
these solutions is up for debate ...
###
# Edit the y-axis line after drawing
library("ggplot2"); library("grid")
g <- ggplot()+
geom_point(aes(x=c(20, 29, 32), y=c(0, 0.4, 1)))+
scale_y_continuous(breaks=c(0, 0.25, 0.5, 0.75, 1))+
labs(x="Label for x axis")+
labs(y="Label for y axis") +
annotate("text", x = 25 , y=1.2, label="Text 1")+
annotate("text", x = 22 , y=1.0, label="How to stop the y-axis line
here !")+
geom_segment(aes(x=25, xend=25, y=0, yend=1.1), linetype=2)+
# This part is just to make it more nice
theme(panel.background = element_rect(fill = 'white'),
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
plot.margin = unit(c(0.5, 1, 0.5, 0.5), "cm"),
axis.text.x=element_text(size=14),
axis.text.y=element_text(size=14),
axis.title.x=element_text(size=18),
axis.title.y=element_text(size=18),
axis.ticks.length=unit(0.3,"cm"),
panel.border = element_blank(),
axis.line.x = element_line(),
axis.line.y = element_line())
g
# Force creation of all grobs
grid.force()
# List all grobs
grid.ls()
# Get y-axis line y-locations
grid.get("axis.line.y..polyline", grep=TRUE)$y
# Edit y-axis line y-locations (trial-and-error to get 0.81)
grid.edit("axis.line.y..polyline", grep=TRUE, y=unit(c(0, 0.81), "npc"))
###
# Insert marker to work off and edit the y-axis line after drawing
library("ggplot2"); library("grid")
g <- ggplot()+
geom_point(aes(x=c(20, 29, 32), y=c(0, 0.4, 1)))+
scale_y_continuous(breaks=c(0, 0.25, 0.5, 0.75, 1))+
labs(x="Label for x axis")+
labs(y="Label for y axis") +
annotate("text", x = 25 , y=1.2, label="Text 1")+
annotate("text", x = 22 , y=1.0, label="How to stop the y-axis line
here !")+
geom_segment(aes(x=25, xend=25, y=0, yend=1.1), linetype=2)+
# This part is just to make it more nice
theme(panel.background = element_rect(fill = 'white'),
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
plot.margin = unit(c(0.5, 1, 0.5, 0.5), "cm"),
axis.text.x=element_text(size=14),
axis.text.y=element_text(size=14),
axis.title.x=element_text(size=18),
axis.title.y=element_text(size=18),
axis.ticks.length=unit(0.3,"cm"),
panel.border = element_blank(),
axis.line.x = element_line(),
axis.line.y = element_line()) +
# marker
geom_point(aes(x=20, y=1), col="red")
g
# Force creation of all grobs
grid.force()
# List all grobs
grid.ls()
# Navigate to panel viewport
downViewport("panel.6-4-6-4")
# Get marker
marker <- grid.get("geom_point", grep=TRUE, global=TRUE)[[2]]
# Edit y-axis line y-locations (based on marker)
grid.edit("axis.line.y..polyline", grep=TRUE,
y=convertY(unit(0:1, c("npc", "groby"), list(NULL, marker)),
"npc"))
# grid.edit("axis.line.y..polyline", grep=TRUE,
# y=unit(c(0, convertY(grobY(marker, 0), "npc", valueOnly=TRUE)),
# "npc"))
# Remove marker
grid.remove(marker$name)
###
# Just draw the plot, with margin above, vertical dash line clipped
library("ggplot2"); library("grid")
g <- ggplot()+
geom_point(aes(x=c(20, 29, 32), y=c(0, 0.4, 1)))+
scale_y_continuous(breaks=c(0, 0.25, 0.5, 0.75, 1),
limits=c(-.05, 1), expand=c(0, 0),
oob=function(x, range) x)+
labs(x="Label for x axis")+
labs(y="Label for y axis") +
annotate("text", x = 25 , y=1.2, label="Text 1")+
annotate("text", x = 22 , y=1.0, label="How to stop the y-axis line
here !")+
geom_segment(aes(x=25, xend=25, y=0, yend=1.1), linetype=2)+
# This part is just to make it more nice
theme(panel.background = element_rect(fill = 'white'),
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
plot.margin = unit(c(3, 1, 0.5, 0.5), "cm"),
axis.text.x=element_text(size=14),
axis.text.y=element_text(size=14),
axis.title.x=element_text(size=18),
axis.title.y=element_text(size=18),
axis.ticks.length=unit(0.3,"cm"),
panel.border = element_blank(),
axis.line.x = element_line(),
axis.line.y = element_line())
g
# Force creation of all grobs
grid.force()
# List all grobs
grid.ls()
# Navigate to panel viewport
downViewport("panel.6-4-6-4")
# Get the line segment and label
seg <- grid.get("segments", grep=TRUE)
lab <- grid.get("text", grep=TRUE)
# Push new viewport with clipping off
pushViewport(viewport(clip="off"))
# Draw line segment and text
grid.draw(seg)
grid.draw(lab)
# Clean up
upViewport()
Paul
On 25/
Re: [R] Limit the y-axis line with ggplot2 (not the axis itself, but the line used at the left of the graph) [SOLVED]
After many tries, here is a solution using grob.
I post here in case it could help someone.
Note that this solution is not 100% optimal as it uses a trick (limits =
c(-0.05, 1.02)) to show fully the points.
Marc
library("ggplot2"); require("gtable"); require("grid")
p <- ggplot()+
geom_point(aes(x=c(20, 29, 32), y=c(0, 0.4, 1)))+
scale_y_continuous(breaks=c(0, 0.25, 0.5, 0.75, 1),
expand = c(0, 0), limits = c(-0.05, 1.02))+
xlim(20, 32) +
labs(x="Label for x axis")+
labs(y="Label for y axis") +
geom_segment(aes(x=25, xend=25, y=0, yend=1), linetype=2)+
# This part is just to make it more nice
theme(panel.background = element_rect(fill = 'white'),
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
plot.margin = unit(c(0.5, 1, 0.5, 0.5), "cm"),
axis.text.x=element_text(size=14),
axis.text.y=element_text(size=14),
axis.title.x=element_text(size=18),
axis.title.y=element_text(size=18),
axis.ticks.length=unit(0.3,"cm"),
panel.border = element_blank(),
axis.line.x = element_line(),
axis.line.y = element_line())
# I prepare the legend to be shown at the top
plegend <- ggplot() +
geom_blank() +
geom_segment(aes(x=25, xend=25, y=0, yend=0.4), linetype=2) +
annotate("text", x = 25 , y=0.5, label="Text 1")+
scale_y_continuous(expand=c(0,0), limits = c(0,1)) +
scale_x_continuous(limits = c(20, 32)) +
theme_minimal() + theme(line=element_blank(),
text=element_blank(),
panel.background=element_blank())
# extract the panel only
gl <- gtable_filter(ggplotGrob(plegend), "panel")
# And draw both
g <- ggplotGrob(p)
g <- gtable_add_rows(x = g, heights = unit(2, "cm"), pos = 0)
g <- gtable_add_grob(g, gl, t = 2, l=4, b=1, r=4)
grid.newpage()
grid.draw(g)
Le 24/10/2016 à 13:08, Marc Girondot via R-help a écrit :
Hello everybody,
Using ggplot2 package, is there a way to force to stop the y-axis line
at a specified point ? (not using ylim because I want that some text
written using annotate() at the top of the graph is still shown).
Bellow is a simple example to show what I would like do:
Thanks a lot
Marc
library("ggplot2")
g <- ggplot()+
geom_point(aes(x=c(20, 29, 32), y=c(0, 0.4, 1)))+
scale_y_continuous(breaks=c(0, 0.25, 0.5, 0.75, 1))+
labs(x="Label for x axis")+
labs(y="Label for y axis") +
annotate("text", x = 25 , y=1.2, label="Text 1")+
annotate("text", x = 22 , y=1.0, label="How to stop the y-axis line
here !")+
geom_segment(aes(x=25, xend=25, y=0, yend=1.1), linetype=2)+
# This part is just to make it more nice
theme(panel.background = element_rect(fill = 'white'),
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
plot.margin = unit(c(0.5, 1, 0.5, 0.5), "cm"),
axis.text.x=element_text(size=14),
axis.text.y=element_text(size=14),
axis.title.x=element_text(size=18),
axis.title.y=element_text(size=18),
axis.ticks.length=unit(0.3,"cm"),
panel.border = element_blank(),
axis.line.x = element_line(),
axis.line.y = element_line())
g
__
[email protected] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
__
Marc Girondot, Pr
Laboratoire Ecologie, Systématique et Evolution
Equipe de Conservation des Populations et des Communautés
CNRS, AgroParisTech et Université Paris-Sud 11 , UMR 8079
Bâtiment 362
91405 Orsay Cedex, France
Tel: 33 1 (0)1.69.15.72.30 Fax: 33 1 (0)1.69.15.73.53
e-mail: [email protected]
Web: http://www.ese.u-psud.fr/epc/conservation/Marc.html
Skype: girondot
__
[email protected] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit of cmdscale function
You avoid the call to cmdscale() by supplying your own starting configuration (see the manual page for the y= argument). You could still hit other barriers within isoMDS() or insufficient memory on your computer. - David L Carlson Department of Anthropology Texas A&M University College Station, TX 77840-4352 -Original Message- From: [email protected] [mailto:[email protected]] On Behalf Of Kawashima, Masayuki Sent: Wednesday, November 5, 2014 10:51 PM To: [email protected] Subject: [R] limit of cmdscale function Hi We have a few questions regarding the use of the "isoMDS" function. When we run "isoMDS" function using 60,000 x 60,000 data matrix, we get the following error message: cmdscale(d, k) : invalid value of 'n' Calls: isoMDS -> cmdscale We checked the source code of "cmdscale" and found the following limitation: ## we need to handle nxn internally in dblcen if(is.na(n) || n > 46340) stop("invalid value of 'n'") 1. This cmdscale limitation ('n > 46340') is due to the limitation of BLAS and LAPACK variables(int4) which can only handle '2^31-1' amount of data? 2. Is there any workaround to run isoMDS using large data (i.e. greater than 46340)? We would like to run isoMDS using a maximum of 150,000x150,000 data matrix. Best regards Masayuki Kawashima Email: [email protected] __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit on vector size allocation
On Aug 29, 2012, at 10:23 AM, Siddeek, Shareef (DFG) wrote: Hi, Can someone help me on the following problem? I have nearly 103,000 records with 22 variables (some are factors). When I ran a stepwise glm on the data set, the R program stops with an error message Stepwise methods are statistically suspect. The fact that they are easy to do in SAS or SPSS is not evidence against the prior sentence. "Error: cannot allocate vector of size 171.3Mb." I have an R 2.12.0 on my PC (Windows XP) with a maximum 4Gb RAM. You appear to have the machine resources to handle the problem, but probably have too much other "stuff" (running programs, open windows, etc) in your system at this time that is occupying memory, this preventing R from having contiguous memory that can hold your data object. You should exit R, update R to a current version, restart your computer, do not run any other applications, ... restart R and redo this analysis. -- David Winsemius, MD Alameda, CA, USA __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit of detection (LOD) by logistic regression
Dear Ellison, You are right, now the figure is good! Question solved. Thank you very much! Best wishes, Luigi -Original Message- From: S Ellison [mailto:[email protected]] Sent: 24 July 2012 10:20 To: Luigi; [email protected] Subject: RE: [R] limit of detection (LOD) by logistic regression > set 1; however the figure obtained from the sample set 2 > shows that interpolation is not correct. I don't think the interpolation is incorrect; what is making it look incorrect is using a straight line to represent a logistic regression. Try adding the predicted values for the line to your plot: lines( dil <- 10^seq(-1, 6, 0.05), predict(model, newdata=data.frame(dilution=dil), type="response")) S Ellison *** This email and any attachments are confidential. Any use...{{dropped:8}} __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit of detection (LOD) by logistic regression
> set 1; however the figure obtained from the sample set 2
> shows that interpolation is not correct.
I don't think the interpolation is incorrect; what is making it look incorrect
is using a straight line to represent a logistic regression.
Try adding the predicted values for the line to your plot:
lines( dil <- 10^seq(-1, 6, 0.05), predict(model,
newdata=data.frame(dilution=dil), type="response"))
S Ellison
***
This email and any attachments are confidential. Any use...{{dropped:8}}
__
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit when reading in file?
On Aug 16, 2011, at 1:33 PM, Sarah Goslee wrote: Hi Noah, On Tue, Aug 16, 2011 at 1:25 PM, Noah Silverman > wrote: Hello, I'm trying to read in a fairly large file into R, and am getting an odd error (65000 rows, 37 columns) Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings, : line 25628 did not have 37 elements The single most common cause of this kind of error is a single or double quote somewhere in that or the surrounding lines. ... but # is another cause of obscure input errors That line DOES have 37 elements. As A test, I tried deleting it, and a few surrounding lines. Same error occurs with a different line number. Is there some hard limit as to how large a file I can import? If so, any suggested work around? Check for quotes. Check your quote option in whatever command you're using to import (you didn't tell us). ... and you can suppress the handling of comments with ..., comment.char="", ... If neither of those work, then offer the list your actual command, and a small reproducible example (eg the lines right around where the error appears). Sarah -- David Winsemius, MD West Hartford, CT __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit when reading in file?
Hi Noah, On Tue, Aug 16, 2011 at 1:25 PM, Noah Silverman wrote: > Hello, > > I'm trying to read in a fairly large file into R, and am getting an odd error > (65000 rows, 37 columns) > > Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings, : > line 25628 did not have 37 elements The single most common cause of this kind of error is a single or double quote somewhere in that or the surrounding lines. > That line DOES have 37 elements. As A test, I tried deleting it, and a few > surrounding lines. Same error occurs with a different line number. > > Is there some hard limit as to how large a file I can import? If so, any > suggested work around? Check for quotes. Check your quote option in whatever command you're using to import (you didn't tell us). If neither of those work, then offer the list your actual command, and a small reproducible example (eg the lines right around where the error appears). Sarah -- Sarah Goslee http://www.stringpage.com __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit on read.socket?
On Sat, 4 Sep 2010, [email protected] wrote: Hi, I have the following piece of code, repeat{ ss<-read.socket(sockfd); if(ss=="") break output<-paste(output,ss) } but somehow, output is not receiving all the data that is coming through the socket.My suspicion is on the if statement. what happens if a white space occurs in between the string arriving over the socket? That's not the problem. The problem occurs when R reads faster than the data are provided -- R will read and there will not be any new data available, so ss will be the empty string and the program will end. In general you can't rely on the sender transmitting data fast enough to keep ahead of R. The example in make.socket() is reading a very small amount of data, so it worked reasonably well back in the days when the finger daemon was more widely active. You need some more precise way of knowing when the data stream is over. This could be some sort of 'end of transmission' marker or a count of the number of bytes or lines to be expected. -thomas Thomas Lumley Professor of Biostatistics University of Washington, Seattle __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] limit to p-value using t.test()
On Sat, Feb 6, 2010 at 8:53 AM, Pete Shepard wrote: > I am using t-test to check if the difference between two populations is > significant. I have a large N=20,000, 10,000 in each population. I compare a > few different populations with each other and though I get different t-scores, > I get the same p-value of 10^-16 which seems like the limit for this > function. Is this true and is so, is there a workaround to get a more > sensitive/accurate p-value? Three comments -- First, with a given value of t and the df for your test, you can get p-values smaller than 2.2e-16 by plugging that information into pt(). > pt(500, df=10, lower.tail=FALSE) [1] 1.259769e-23 > pt(1500, df=10, lower.tail=FALSE) [1] 2.133778e-28 Second, if these are *populations* then a t-test is inappropriate. Just compute the means, and if they do not equal one another, then the population means are different. All the statistical tests that I can think of try to make and place bounds on inferences about the population based upon samples drawn from those populations. If you have the populations, this makes no sense. It seems like you need to decide what kinds of differences are meaningful, and then check to see if the population differences meet those criteria. Third, why do you want a more accurate p-value? The only reason I can think of is using Rosenthal & Rubin's method to compute effect sizes from a p-value, but again, if you have the populations, you can compute effect sizes directly. Good luck! Michael __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit on number of times Realloc can be called?
Adam Waldemar Kowalewski wrote:
Hello,
I've been writing a program in C that will be called by R. I seem to have
stumbled upon an odd error that seems to suggest there is a limit on the
number of times "Realloc" (the R version as defined in the manual
"R-extenstions" not the C version "realloc") when I try to use the
following program:
#include
#include
SEXP test_mem_alloc(SEXP z) {
double *t = Calloc(sizeof(double), double);
*t = 2;
SEXP end_product;
int i = 0;
for(i=1; i < 20; i++) {
t = Realloc(t, sizeof(t) + sizeof(double), double);
The second argument to Realloc is supposed to be the number of elements
to allocate. sizeof(t) is 4 or 8 (32 bit or 64 bit), sizeof double is
8, so you always allocate 12 or 16 elements. Then in the next line you
write out of bounds.
Duncan Murdoch
t[i] = i;
}
PROTECT(end_product = allocVector(REALSXP,6));
for(i = 0; i < 20; i++) {
REAL(end_product)[i] = t[i];
}
UNPROTECT(1);
Free(t);
return end_product;
}
I call it from R using the following script:
z <- 1
test_mem_alloc <- function(z) {
if(!(is.loaded("test_mem_alloc_v6"))) dyn.load("test_mem_alloc_v6.dll")
out <- .Call("test_mem_alloc",
as.double(z))
return(out)
dyn.unload("test_mem_alloc_v6.dll")
}
Basically I get the following error messages:
First:
"Runtime Error!
Program: C:\Program Files\R\R-2.10.0\bin\Rgui.exe
This application has requested the Runtime to terminate it in an unusual
way. Please contact the application's support team for more information."
The second error message is:
"The instruction at "0x002c". The memory could not be "read"."
Now, if change the number of times the program goes through the for loop
from 20 to say 6, and hence calls Realloc fewer times, then the program
runs without any problems. It does not seem to have anything to do with
the size of the memory being allocated as I changed the size of the total
memory being allocated with a "for" loop with only 6 iterations to
something that is substantially larger than the memory being allocated in
the "for" loop when calling Realloc twenty times and the program ran
successfully. Has anyone else come across this problem or knows about some
sort of limitation on using "Realloc" that is not specified in the R
documentation?
Any help would be greatly appreciated.
Yours sincerely
Adam Kowalewski
__
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit on number of Rows when reading R dataset
Thank you Uwe and Prof. Ripley. The problem was solved. The row in question indeed have garbage data, which possibly was truncating the number of lines read. I apologise for the oversight. Thank you once again. Regards Harsh Singhal Bangalore, India On Tue, Dec 2, 2008 at 2:50 PM, Prof Brian Ripley <[EMAIL PROTECTED]> wrote: > Take a look at your dataset at around that row. Perhaps you have an > unmatched quote? > > The limit on the number of rows of a data frame is far larger than 100,000 > (2^31-1, but you will run out of address space on a 32-bit platform before > that - see ?"Memory-limits"). > > On Tue, 2 Dec 2008, Harsh wrote: > >> Hello, >> I am trying to read a dataset with 100,000 rows and around 365 columns >> into R, using read.table/read.csv. >> In Windows XP, with R 32 bit, I am able to read only 15266 rows and >> not more than that. >> I tried the same in R running in Ubuntu and it does the same and reads >> only 15266 rows. >> Using the nrows paramter i can read rows less than 15266, but when i >> used a value larger than 15266, it reads only 15266 nevertheless. >> >> Thank you for your patience and responses. >> >> Regards >> Harsh Singhal >> Bangalore, India > > -- > Brian D. Ripley, [EMAIL PROTECTED] > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ > University of Oxford, Tel: +44 1865 272861 (self) > 1 South Parks Road, +44 1865 272866 (PA) > Oxford OX1 3TG, UKFax: +44 1865 272595 > __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit on number of Rows when reading R dataset
Take a look at your dataset at around that row. Perhaps you have an unmatched quote? The limit on the number of rows of a data frame is far larger than 100,000 (2^31-1, but you will run out of address space on a 32-bit platform before that - see ?"Memory-limits"). On Tue, 2 Dec 2008, Harsh wrote: Hello, I am trying to read a dataset with 100,000 rows and around 365 columns into R, using read.table/read.csv. In Windows XP, with R 32 bit, I am able to read only 15266 rows and not more than that. I tried the same in R running in Ubuntu and it does the same and reads only 15266 rows. Using the nrows paramter i can read rows less than 15266, but when i used a value larger than 15266, it reads only 15266 nevertheless. Thank you for your patience and responses. Regards Harsh Singhal Bangalore, India -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit on number of Rows when reading R dataset
Harsh wrote: Hello, I am trying to read a dataset with 100,000 rows and around 365 columns into R, using read.table/read.csv. In Windows XP, with R 32 bit, I am able to read only 15266 rows and not more than that. I tried the same in R running in Ubuntu and it does the same and reads only 15266 rows. Using the nrows paramter i can read rows less than 15266, but when i used a value larger than 15266, it reads only 15266 nevertheless. What happens exactly? Error message? How does you file look like in rows 15260-15270? Uwe Ligges Thank you for your patience and responses. Regards Harsh Singhal Bangalore, India __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Limit distribution of continuous-time Markov process
On Thu, 5 Jun 2008, [EMAIL PROTECTED] wrote:
I have (below) an attempt at an R script to find the limit distribution
of
a continuous-time Markov process, using the formulae outlined at
http://www.uwm.edu/~ziyu/ctc.pdf, page 5.
First, is there a better exposition of a practical algorithm for doing
this?
The exposition there seemed pretty clear.
Maybe you need to brush up on algebra - particularly matrix
decompositions. I'd go to Rao's book (Linear Statistical Inference and Its
Applications,1973) and look at the early chapter (sorry I can't say which
one - I am at home now and do not have a copy at hand) that covers matrix
decomposition/diagonalization and work a few exercises to get up to speed.
But there are lots of texts that would cover this stuff, so pick one and
have at it.
The point of that exposition is that you can get the matrix exponential of
the transition matrix from the matrix exponential of the diagonal matrix
of eigenvalues and the eigenvectors of the rate matrix.
So, eigen() does most of the work.
Once you realize what $\lim_{ t \to \infty }{ E^{tD }$ is (using the
notation of the link you provided), you will see the computation is
trivial given the results of eigen().
HTH,
Chuck
I have not found an R package that does this specifically, nor
anything on the web.
Second, the script below will give the right answer, _if_ I "normalize"
the rate matrix, so that the average rate is near 1.0, and _if_ I
tweak the multiplier below (see **), and then watch for the Answer to
converge to a matrix where the rows to sum to 1.0. (This multiplier
is "t" in the PDF whose URL is above.) Is there a known way to get
this to converge?
Thank you.
---R script:--
# The rate matrix:
Q <- matrix(c(-1, 1, 0, 1, -2, 1, 0, 1, -1), ncol=3, byrow=T);
M <- eigen(Q)$vectors # diagonalizer matrix
D <- ginv(eigen(Q)$vectors) %*% Q %*% eigen(Q)$vectors # Diagonalized
form
Sum <- matrix(c(rep(0, 9)), ncol=3, byrow=T);
for (i in 0:60)
{ # Naive, Taylor series summation:
Temp <- D;
diag(Temp) <- (4 * diag(D)) ^ i; # ** =4
Sum <- Sum + Temp / factorial(i);
}
Answer <- M %*% Sum %*% ginv(M);
Answer;
# (Right answer for this example is a matrix with all values = 1/3.)
Grant D. Schultz
__
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Charles C. Berry(858) 534-2098
Dept of Family/Preventive Medicine
E mailto:[EMAIL PROTECTED] UC San Diego
http://famprevmed.ucsd.edu/faculty/cberry/ La Jolla, San Diego 92093-0901
__
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
