Hi Jeff, Memory was not an issue. The system only used 75% of the memory allocated for the job.
I am trying to understand what "r read large file in chunks" is doing. On Fri, Nov 8, 2024 at 7:50 PM Jeff Newmiller <jdnew...@dcn.davis.ca.us> wrote: > > Then you don't have enough memory to process the whole thing at once. Not > unlike stuffing your mouth with cookies and not being able to chew for lack > of space to move the food around in your mouth. > > Now, can you answer my question? > > On November 8, 2024 5:38:37 PM PST, Val <valkr...@gmail.com> wrote: > >The data was read. The problem is with processing. > > > >On Fri, Nov 8, 2024 at 7:30 PM Bert Gunter <bgunter.4...@gmail.com> wrote: > >> > >> Is the problem reading the file in or processing it after it has been read > >> in? > >> > >> Bert > >> > >> On Fri, Nov 8, 2024 at 5:13 PM Jeff Newmiller via R-help > >> <r-help@r-project.org> wrote: > >>> > >>> Can you tell us what is wrong with the "chunked" package which comes up > >>> when you Google "r read large file in chunks"? > >>> > >>> On November 8, 2024 4:58:18 PM PST, Val <valkr...@gmail.com> wrote: > >>> >Hi All, > >>> > > >>> >I am reading data file ( > 1B rows) and do some date formatting like > >>> > dat=fread(mydatafile) > >>> > dat$date1 <- as.Date(ymd(dat$date1)) > >>> > > >>> >However, I am getting an error message saying that > >>> > Error: cons memory exhausted (limit reached?) > >>> > > >>> >The script was working when the number rows were around 650M. > >>> > > >>> >Is there another way to handle a big data set in R? > >>> > > >>> > > >>> >Thank you. > >>> > > >>> >______________________________________________ > >>> >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > >>> >https://stat.ethz.ch/mailman/listinfo/r-help > >>> >PLEASE do read the posting guide > >>> >https://www.R-project.org/posting-guide.html > >>> >and provide commented, minimal, self-contained, reproducible code. > >>> > >>> -- > >>> Sent from my phone. Please excuse my brevity. > >>> > >>> ______________________________________________ > >>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > >>> https://stat.ethz.ch/mailman/listinfo/r-help > >>> PLEASE do read the posting guide > >>> https://www.R-project.org/posting-guide.html > >>> and provide commented, minimal, self-contained, reproducible code. > > -- > Sent from my phone. Please excuse my brevity. ______________________________________________ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide https://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
