Re: [R] help usin scan on large matrix (caveats to what has been discussed before)

Thu, 12 Aug 2010 06:36:46 -0700 

Hi baptiste,
thanks a lot. Could you please comment on that code, I cannto figure out what it does. Appart from the file name, what parameters does it need? Seems to me like you need to know the size of the table a priori. Is that right? Do you have to set up the block size depending on that (so that you get full multiples of the block to form the resulting frame)? 
Cheers
Martin

On 8/12/2010 2:45 PM, baptiste Auguié wrote:

Hi,

I don't know if this can be useful to you, but I recently wrote a small 
function to read a large datafile like yours in a number of steps, with the 
possibility to save each intermediate block as .Rdata. This is based on 
read.table --- not as efficient as lower-level scan() but it might be good 
enough,

file<- 'test.txt'
## write.table(matrix(rnorm(1e6*14), ncol=14), file=file,row.names = F,
##             col.names = F )

n<- as.numeric(gsub("[^0123456789]","", system(paste("wc -l ", file), 
int=TRUE)))
n

blocks<- function(n=18, size=5){
res<- c(replicate(n%/%size, size))
if(n%%size) res<- c(res, n%%size)
if(!sum(res) == n) stop("ERROR!!!")
res
}
## blocks(1003, 500)


readBlocks<- function(file, nbk=1e5, out="tmp", save.inter=TRUE,
                        classes= c("numeric", "numeric", rep("NULL", 6),
                          "numeric", "numeric", rep("NULL", 4))){

   n<- as.numeric(gsub("[^0123456789]","", system(paste("wc -l ", file), 
int=TRUE)))

   ncols<- length(grep("NULL", classes, invert=TRUE))
   results<- matrix(0, nrow=n, ncol=ncols)
   Nb<- blocks(n, nbk)
   skip<- c(0, cumsum(Nb))
   for(ii in seq_along(Nb)){
     d<- read.table(file, colClasses = classes, nrows=Nb[ii], skip=skip[ii], comment.char 
= "")
     if(save.inter){
       save(d, file=paste(out, ".", ii, ".rda", sep=""))
       }
     print(ii)
     results[seq(1+skip[ii], skip[ii]+Nb[ii]), ]<- as.matrix(d)
     rm(d) ; gc()
   }
   save(results, file=paste(out, ".rda", sep=""))
   invisible(results)
}

## test<- readBlocks(file)

HTH,

baptiste



On Aug 12, 2010, at 1:34 PM, Martin Tomko wrote:


Hi Peter,
thank you for your reply. I still cannot get it to work.
I have modified your code as follows:
rows<-length(R)
cols<- max(unlist(lapply(R,function(x) length(unlist(gregexpr(" 
",x,fixed=TRUE,useBytes=TRUE))))))
c<-scan(file=f,what=rep(c(list(NULL),rep(list(0L),cols-1),rows-1)), skip=1)
m<-matrix(c, nrow = rows-1, ncol=cols+1,byrow=TRUE);

the list c seems ok, with all the values I would expect. Still, length(c) gives 
me a value = cols+1, which I find odd (I would expect =cols).
I thine repeated it rows-1 times (to account for the header row). The values 
seem ok.
Anyway, I tried to construct the matrix, but when I print it, the values are 
odd:

m[1:10,1:10]

      [,1] [,2]       [,3]       [,4]       [,5]       [,6]       [,7]
[1,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[2,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[3,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[4,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[5,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[6,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[7,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[8,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[9,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
[10,] NULL Integer,15 Integer,15 Integer,15 Integer,15 Integer,15 Integer,15
....

Any idea where the values are gone?
Thanks
Martin

Hence, I filled it into the matrix of dimensions

On 8/12/2010 12:24 PM, peter dalgaard wrote:

On Aug 12, 2010, at 11:30 AM, Martin Tomko wrote:



c<-scan(file=f,what=list(c("",(rep(integer(0),cols)))), skip=1)
m<-matrix(c, nrow = rows, ncol=cols,byrow=TRUE);

for some reason I end up with a character matrix, which I don't want. Is this the proper 
way to skip the first column (this is not documented anywhere - how does one skip the 
first column in scan???). is my way of specifying "integer(0)" correct?


No. Well, integer(0) is just superfluous where 0L would do, since scan only 
looks at the types not the contents, but more importantly, what= wants a list 
of as many elements as there are columns and you gave it



list(c("",(rep(integer(0),5))))


[[1]]
[1] ""

I think what you actually meant was

c(list(NULL),rep(list(0L),5))





And finally - would any sparse matrix package be more appropriate, and can I 
use a sparse matrix for the image() function producing typical heat,aps? I have 
seen that some sparse matrix packages produce different looking outputs, which 
would not be appropriate.

Thanks
Martin

______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.





--
Martin Tomko
Postdoctoral Research Assistant

Geographic Information Systems Division
Department of Geography
University of Zurich - Irchel
Winterthurerstr. 190
CH-8057 Zurich, Switzerland

email:  martin.to...@geo.uzh.ch
site:   http://www.geo.uzh.ch/~mtomko
mob:    +41-788 629 558
tel:    +41-44-6355256
fax:    +41-44-6356848

______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.






--
Martin Tomko
Postdoctoral Research Assistant

Geographic Information Systems Division
Department of Geography
University of Zurich - Irchel
Winterthurerstr. 190
CH-8057 Zurich, Switzerland

email:  martin.to...@geo.uzh.ch
site:   http://www.geo.uzh.ch/~mtomko
mob:    +41-788 629 558
tel:    +41-44-6355256
fax:    +41-44-6356848

______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Reply via email to