calc() is great for map algebra type stuff, but for anything more complicated,
such as your favorite clustering or regression routine in R, then you sometimes
need the entire image data as an R data.frame to feed to whatever function
expects a data.frame.
Perhaps a solution to your needs is to then use as.matrix or as.array or
sampleRegular(). These work on Raster Stacks. And, you can specify the
maximum number of pixels. The data are then in a single R object, which you
can then reshape and feed to whatever R function you want.
as.array is nice, because it preserves the three-dimensional nature of the
stack.
Here is a cut and paste of code I have used. It is non-reproducible because it
points to my own data store, but hopefully you get the idea.
# I have a folder called modis_data. Each file is a single date and band,
created by the MRT tool.
# Get the list of all dates within my data folder with the TIF files.
dates <- sort(unique(substr(list.files('modis_data/'), 10, 16)))
###
band.names <- c('b01', 'b02', 'b03', 'b04', 'b05', 'b06', 'b07')
modis <- list()
for(f in band.names){
# Construct a list of all TIF filenames for this band that I have
regexp.str <- paste('MOD09.*(', paste(dates, collapse='|'), ').*sur_refl_',
f, '\\.tif', sep='')
files.list <- list.files('modis_data/', full.names=TRUE)[grep(regexp.str,
list.files('modis_data/'))]
# Assemble the MODIS object in R. It is a list, with one element for each band
# Each list element is a stack of all the TIF files for that band.
modis[[f]] <- stack(lapply(files.list, raster))
}
# Now the data are all accessible to R (but not in memory, they're still the
TIFs!)
# Now we get to do analysis
# Get data from the raster files for band 01
dat <- as.array(modis[['b01']])
# dat is now a big 3D R array, each slice is a different MODIS image for band
01, but it's all in R now
# If that is too big, maybe call sampleRegular() instead. It works on stacks,
too.
# Do something on that array
# For example, reshape to a vector and put in a data.frame.
On May 8, 2013, at 11:07 AM, Francisco Zambrano
<[email protected]<mailto:[email protected]>> wrote:
I need apply the function "lowess" and that (from what I understand) it take
values from all layers of the stack at the same time. But, "lapply or loop" run
function for each layer not for one cell and all the layers, And is for that
I'm think need a rasterBrick instead a rasterStack
The code is something like:
out<-stack(list.files(pattern='*.tiff$)
filename<-rasterTmpFile()
out<-writeStart(out,filename,overwrite=TRUE)
bs<-blockSize(out)
pb<-pbCreate(bs$n)
for (i in 1:bs$n){
v<-getValues(out,row=bs$row[i],nrows=bs$nrows[i])
s<-t(sapply(apply(v,1,lowess,f=7/n,iter=10),"[[","y"))
out<-writeValues(out,s,bs$row[i])
pbStep(pb,i)
}
2013/5/8 Nagle, Nicholas <[email protected]<mailto:[email protected]>>
Stacks can be treated as lists of RasterLayers. So use writeStart on lists of
rasters.
Options to consider:
1) You could either loop (or lapply) through list, modifying the original TIFs
with new data.
2) Create a new stack (not in memory) of RasterLayers with empty values. You
can use one of the MODIS stack layers as a template for these new layers, and
then loop (lapply) through the stack, using writeStart on each Layer, or
3) create a new brick (not in memory!) with the necessary dimensions, and then
use writeStart on this with the calculated values. Again, you should be able
to use one of the stack layers as a template to create this brick.
Only method 1) will modify the original TIFs and put the new values alongside
the original MODIS values. I don't know how important it is to duplicate the
original MODIS pixel values along with whatever values you are creating.
Method 2) will create lots of individual, manageable files. You could modify
the naming convention of the original MODIS files.
Method 3) will keep all of the new values together in one big file.
Sometimes I am doing analysis for myself, and don't need to pass the data along
to someone else. In these instances, I often just save the new values as R
data.frames, and just inject subsets of the data.frame into RasterLayers at the
time of plotting.
Nicholas
On May 8, 2013, at 9:23 AM, Francisco Zambrano
<[email protected]<mailto:[email protected]>>
wrote:
Howdy Nicholas,
Thanks for your answer. In the beginning I did the stack, but to make a
function using "writeStart", this function works with "rasterBrick" objects and
not with 'rasterStack'. That is the reason why I need to make a brick.
I don't know what could be the most efficent solution.
My solution in this moment is as follow:
1.- Create the 'rasterStack'
2.- Save with 'writeRaster(rasterStack,filename)
3.- Open it with "brick(filenanme)"
4.- Use the function writeStart
Thats is the better way or exist another option?
2013/5/8 Nagle, Nicholas <[email protected]<mailto:[email protected]>>
Dear Francisco,
I am working with the same time series of MOD09 data, but I leave it as a
stack, rather than create the brick.
Why do you need it as a brick? The problem with brick is that it needs the
data to be in one file. That's a big file in this instance (Tb scale?). And,
since you didn't specify a filename, R is going to try and do this in memory,
and will need to create virtual memory for this. 11 hours doesn't sound
unreasonable for this.
Why not just leave it as a stack? I've found it to work pretty well.
Otherwise, be sure to specify the filename so that R doesn't try to to this in
memory. But that still might take time, and is probably a needless recopy of a
large amount of data.
Nicholas
Nicholas Nagle, Assistant Professor
Department of Geography
Burchfiel Geography Building 307
University of Tennessee
Knoxville, TN 37996
> Date: Tue, 7 May 2013 10:35:00 -0400
> From: Francisco Zambrano <[email protected]<mailto:[email protected]>>
> To: [email protected]<mailto:[email protected]>
> Subject: [R-sig-Geo] How to write functions with mutiple large raster
> data file (MOD13Q1, 2000-2012)
> Message-ID:
>
> <CANE=RDO9N-WmnjKTbqnzkeFN=tsfogxwwykqo5jrx4ejsmg...@mail.gmail.com<mailto:[email protected]>>
> Content-Type: text/plain
>
> Hi everybody,
>
> I'm working with MODIS data MOD13Q1 and the NDVI index between 2000- today.
> I have been trying to do a function, but I have a problem when stack the
> 310 files and then I tried to use the writeStart function. I had an error
> because the startFuction work with rasterLayer or RasterBrick. But, when I
> tried to make the brick with the rasterStack (>brick(rasterStack)) the
> operation took too much time (11 hours)
>
> My question, is what is the most efficiently way to make a rasterBrick from
> multiple raster (geoTIFF) files?
>
> save the rasterStack and then open it with brick?
>
> My code is something like:
>
>> out<-stack(list.files(pattern=*..tif$))
>> out<-brick(mod) #too much time (11 hours appox)
>
>> out<-writeStart(out,filename,overwrite=TRUE)
>> bs<-blockSize(out)
>> pb<-pbCreate(bs$n)
>> for (i in 1:bs$n){
> v<-getValues(out,row=bs$row[i],nrows=bs$nrows[i])
> s<-t(sapply(apply(v,1,lowess,f=7/n,iter=10),"[[","y"))
> out<-writeValues(out,s,bs$row[i])
> pbStep(pb,i)
> }
>
> Kind Regards
>
> Francisco Zambrano Bigiarini
>
> from Chile, latin america
--
El que te roba una idea se queda con un pedazo de tí, pero pierde dos pedazos
de sí: la imaginación y el honor
--
El que te roba una idea se queda con un pedazo de tí, pero pierde dos pedazos
de sí: la imaginación y el honor
[[alternative HTML version deleted]]
_______________________________________________
R-sig-Geo mailing list
[email protected]
https://stat.ethz.ch/mailman/listinfo/r-sig-geo