try this:
input <- file("yourLargeCSV", "r")
fileNo <- 1
repeat{
myLines <- readLines(input, n=100000) # 100K lines / file
if (length(myLines) == 0) break
writeLines(myLines, sprintf("output%03d.csv", fileNo))
fileNo <- fileNo + 1
}
close(input)
On Tue, Jul 24, 2012 at 9:45 AM, Rantony <[email protected]> wrote:
> Hi,
>
> Here in R, I need to load a huge file(.csv) , its size is 200MB. [may come
> more than 1GB sometimes].
> When i tried to load into a variable it taking too much of time and after
> that when i do cbind by groups,
> getting an error like this
>
> " Error: cannot allocate vector of size 82.4 Mb "
>
> My requirement is, spilt data from Huge-size-file(.csv) to no. of small csv
> files.
> Here i will give no of lines to be 'split by' as input.
>
> Below i give my code
> -------------------------------
> SplitLargeCSVToMany <-
> function(DataMatrix,Destination,NoOfLineToGroup)
> {
> test <- data.frame(read.csv(DataMatrix))
>
> # create groups No.of rows
> group <- rep(1:NROW(test), each=NoOfLineToGroup)
> new.test <- cbind(test, group=group)
> new.test2 <- new.test
> new.test2[,ncol(new.test2)] <- NULL
>
> # now get indices to write out
> indices <- split(seq(nrow(test)), new.test[, 'group'])
>
> # now write out the files
> for (i in names(indices))
> {
> write.csv(new.test2[indices[[i]],],
> file=paste(Destination,"data.", i,
> ".csv", sep=""),row.names=FALSE)
> }
> }
>
> -----------------------------------------------------
> My system Configuration is,
> Intel Core2 Duo
> speed : 3GHz
> 2 GB RAM
> OS: Windows-XP [ServicePack-3]
> ---------------------------------------------------
>
> Any hope to solve this issue ?
>
> Thanks in advance,
> Antony.
>
>
>
>
> --
> View this message in context:
> http://r.789695.n4.nabble.com/ERROR-cannot-allocate-vector-of-size-in-MB-GB-tp4637597.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [email protected] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
--
Jim Holtman
Data Munger Guru
What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
