Re: [GRASS-user] new GRASS virtual raster in trunk
(cc grass-dev for the record) On Mon, Jun 4, 2018 at 4:59 PM, Markus Metz wrote: > On Mon, Jun 4, 2018 at 3:36 PM, Nikos Alexandris > wrote: ... >>> On Mon, 4 Jun 2018, Markus Metz wrote: A new GRASS virtual raster format has been added to trunk in r72761-4, together with a new module to build a GRASS VRT: r.buildvrt. >>> ... >> if I may, virtual raster maps allow to treat an arbitrary number of >> raster maps as one map. Hence, it is not required to patch hundreds or >> thousands or more (?) raster map tiles, to build a single map. > > Technically, there is no limit on the number of map tiles combined in a VRT. > However, reading should become a bit slower with more tiles, independent of > the size of the individual tiles. >> >> Why build a single map? Because the one or the other module or workflow >> need it, in order to be able to process the entire computational extent >> of interest. >> >> >> In my case, I treat 177 tiles that cover European territories, >> for each of which there exists a raster map (of 1 x 1 pixels). >> Having one raster map, for the whole of Europe, will make it easier to >> extract zonal statistics, for example. >> >> >> So, in the general case of many raster maps, that form say, a "canonical" >> tile grid, zones of interest (say vector boundaries), for which is asked >> to extract statistics, are possibly split over two to up to four raster >> map tiles. It takes some respectable amount of time of handcrafting, to >> get stats for each tile, assembly then results for zones that don't >> entirely fall inside one tile. > > In this case, using a huge single raster map should be slower, if only up to > four small tiles are needed. Here, a VRT is faster. > >> >> I am sure there are better explanations or even more complicated >> problems for which a Virtual raster is the answer. In short, however, >> and the way I understand things, it is a "work-around" to avoid dealing >> with huge raster data sets. Huge as in file size. And, while the total >> processing time, might not necessarily drop down, depending on the >> work-flow, the time to script a solution decreases dramatically. > > Thanks for the explanation Nikos! > > There are more sophisticated use cases: you can distribute tiles over > different mapsets which may reside on different physical storage devices. > Simultaneous access to different parts of the VRT raster should now become > much faster. This can be useful for e.g. web processing services or also > parallel processing on a HPC. > > Markus M So cool, thanks Markus!! markusN ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] new GRASS virtual raster in trunk
On Mon, Jun 4, 2018 at 3:36 PM, Nikos Alexandris wrote: > > * Rich Shepard [2018-06-04 05:56:16 -0700]: > >> On Mon, 4 Jun 2018, Markus Metz wrote: >> >>> A new GRASS virtual raster format has been added to trunk in r72761-4, >>> together with a new module to build a GRASS VRT: r.buildvrt. >> >> >> Markus, >> >> For folks like me who have not followed this thread please explain the >> problem for which a raster VRT is the solution. >> >> Best regards, >> >> Rich > > > Dear Rich, > > (sorry Markus for my intervention -- just so excited about this new > creation of yours!) > > if I may, virtual raster maps allow to treat an arbitrary number of > raster maps as one map. Hence, it is not required to patch hundreds or > thousands or more (?) raster map tiles, to build a single map. Technically, there is no limit on the number of map tiles combined in a VRT. However, reading should become a bit slower with more tiles, independent of the size of the individual tiles. > > Why build a single map? Because the one or the other module or workflow > need it, in order to be able to process the entire computational extent > of interest. > > > In my case, I treat 177 tiles that cover European territories, > for each of which there exists a raster map (of 1 x 1 pixels). > Having one raster map, for the whole of Europe, will make it easier to > extract zonal statistics, for example. > > > So, in the general case of many raster maps, that form say, a "canonical" > tile grid, zones of interest (say vector boundaries), for which is asked > to extract statistics, are possibly split over two to up to four raster > map tiles. It takes some respectable amount of time of handcrafting, to > get stats for each tile, assembly then results for zones that don't > entirely fall inside one tile. In this case, using a huge single raster map should be slower, if only up to four small tiles are needed. Here, a VRT is faster. > > I am sure there are better explanations or even more complicated > problems for which a Virtual raster is the answer. In short, however, > and the way I understand things, it is a "work-around" to avoid dealing > with huge raster data sets. Huge as in file size. And, while the total > processing time, might not necessarily drop down, depending on the > work-flow, the time to script a solution decreases dramatically. Thanks for the explanation Nikos! There are more sophisticated use cases: you can distribute tiles over different mapsets which may reside on different physical storage devices. Simultaneous access to different parts of the VRT raster should now become much faster. This can be useful for e.g. web processing services or also parallel processing on a HPC. Markus M > > > Nikos, happy to be a GRASS GIS user. > > ___ > grass-user mailing list > grass-user@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/grass-user ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] new GRASS virtual raster in trunk
On Mon, 4 Jun 2018, Nikos Alexandris wrote: Why build a single map? Because the one or the other module or workflow need it, in order to be able to process the entire computational extent of interest. Nikos, Thanks for the cogent explanation. Regards, Rich ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] new GRASS virtual raster in trunk
* Markus Metz [2018-06-04 14:37:41 +0200]: A new GRASS virtual raster format has been added to trunk in r72761-4, together with a new module to build a GRASS VRT: r.buildvrt. *r.buildvrt* builds a virtual raster (VRT) that is a mosaic of the list of input raster maps. The purpose of such a VRT is to provide fast access to small subsets of the VRT, also with multiple simultaneous read requests. *r.buildvrt* creates a list of raster maps that can be located in different mapsets. The ouput is a read-only link to the original raster maps which is only valid if the original raster maps remain in the originally indicated mapset. A VRT can also be built from raster maps registered with *r.external*. Reading the whole VRT is slower than reading the equivalent single raster map. Only reading small parts of the VRT provides a performance benefit. Please test! Markus M Speechless! Respect Mr. Metz. A working test: # get extents g.region -ec raster=L5192028_02820090723_Rad_ref_TRC_BYTE_FROMGLCaggV1@PERMANENT res=1000 north-south extent: 214230.00 east-west extent: 246330.00 center easting: 741150.00 center northing:510.00 # set east to center easting of landcover map g.region -a e=738150.00 res=1000 # build a "half" map r.mapcalc "WestHalf = 1" --o # set west to center easting of land cover map, reset east g.region w=738150.00 e=$(echo "246330/2 + 741150" |bc) res=1000 -a # build other half r.mapcalc "EastHalf = 2" --o # combine r.buildvrt input=WestHalf,EastHalf out=fIRSTgRASSYvRT --o title="First GRASS GIS Virtual Raster Map" # ? r.info -ge fIRSTgRASSYvRT north=5208000 south=4992000 east=865000 west=617000 nsres=1000 ewres=1000 rows=216 cols=248 cells=53568 datatype=CELL ncats=0 map=fIRSTgRASSYvRT mapset=lst location=lst_time_series database=/geo37/grassdb date="Mon Jun 4 16:02:58 2018" creator="nik" title="First GRASS GIS Virtual Raster Map" timestamp="none" units="none" vdatum="none" source1="" source2="" description="generated by r.buildvrt" comments="r.buildvrt --overwrite input="WestHalf,EastHalf" output="fIRSTgRASSY\vRT" title="First GRASS GIS Virtual Raster Map"" # see in-terminal g.region -a raster=L5192028_02820090723_Rad_ref_TRC_BYTE_FROMGLCaggV1@PERMANENT res=1000 ty.d.rast fIRSTgRASSYvRT # terminology's `tycat` Thank you so very much Markus, Nikos signature.asc Description: PGP signature ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] new GRASS virtual raster in trunk
* Nikos Alexandris [2018-06-04 15:36:14 +0200]: * Rich Shepard [2018-06-04 05:56:16 -0700]: On Mon, 4 Jun 2018, Markus Metz wrote: A new GRASS virtual raster format has been added to trunk in r72761-4, together with a new module to build a GRASS VRT: r.buildvrt. Markus, For folks like me who have not followed this thread please explain the problem for which a raster VRT is the solution. Best regards, Rich Dear Rich, (sorry Markus for my intervention -- just so excited about this new creation of yours!) if I may, virtual raster maps allow to treat an arbitrary number of raster maps as one map. Hence, it is not required to patch hundreds or thousands or more (?) raster map tiles, to build a single map. Why build a single map? Because the one or the other module or workflow need it, in order to be able to process the entire computational extent of interest. In my case, I treat 177 tiles that cover European territories, for each of which there exists a raster map (of 1 x 1 pixels). Having one raster map, for the whole of Europe, will make it easier to extract zonal statistics, for example. So, in the general case of many raster maps, that form say, a "canonical" tile grid, zones of interest (say vector boundaries), for which is asked to extract statistics, are possibly split over two to up to four raster map tiles. It takes some respectable amount of time of handcrafting, to get stats for each tile, assembly then results for zones that don't entirely fall inside one tile. Not to forget: somewhere before the processing, of course, a `g.region` to set the extent over a zone of interest, is required. With A VRT, the whole "split-up in different raster maps" problem goes away. Nikos I am sure there are better explanations or even more complicated problems for which a Virtual raster is the answer. In short, however, and the way I understand things, it is a "work-around" to avoid dealing with huge raster data sets. Huge as in file size. And, while the total processing time, might not necessarily drop down, depending on the work-flow, the time to script a solution decreases dramatically. Nikos, happy to be a GRASS GIS user. -- Nikos Alexandris | Remote Sensing & Geomatics GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3 signature.asc Description: PGP signature ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] new GRASS virtual raster in trunk
* Rich Shepard [2018-06-04 05:56:16 -0700]: On Mon, 4 Jun 2018, Markus Metz wrote: A new GRASS virtual raster format has been added to trunk in r72761-4, together with a new module to build a GRASS VRT: r.buildvrt. Markus, For folks like me who have not followed this thread please explain the problem for which a raster VRT is the solution. Best regards, Rich Dear Rich, (sorry Markus for my intervention -- just so excited about this new creation of yours!) if I may, virtual raster maps allow to treat an arbitrary number of raster maps as one map. Hence, it is not required to patch hundreds or thousands or more (?) raster map tiles, to build a single map. Why build a single map? Because the one or the other module or workflow need it, in order to be able to process the entire computational extent of interest. In my case, I treat 177 tiles that cover European territories, for each of which there exists a raster map (of 1 x 1 pixels). Having one raster map, for the whole of Europe, will make it easier to extract zonal statistics, for example. So, in the general case of many raster maps, that form say, a "canonical" tile grid, zones of interest (say vector boundaries), for which is asked to extract statistics, are possibly split over two to up to four raster map tiles. It takes some respectable amount of time of handcrafting, to get stats for each tile, assembly then results for zones that don't entirely fall inside one tile. I am sure there are better explanations or even more complicated problems for which a Virtual raster is the answer. In short, however, and the way I understand things, it is a "work-around" to avoid dealing with huge raster data sets. Huge as in file size. And, while the total processing time, might not necessarily drop down, depending on the work-flow, the time to script a solution decreases dramatically. Nikos, happy to be a GRASS GIS user. signature.asc Description: PGP signature ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] new GRASS virtual raster in trunk
On Mon, 4 Jun 2018, Markus Metz wrote: A new GRASS virtual raster format has been added to trunk in r72761-4, together with a new module to build a GRASS VRT: r.buildvrt. Markus, For folks like me who have not followed this thread please explain the problem for which a raster VRT is the solution. Best regards, Rich ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] new GRASS virtual raster in trunk
A new GRASS virtual raster format has been added to trunk in r72761-4, together with a new module to build a GRASS VRT: r.buildvrt. *r.buildvrt* builds a virtual raster (VRT) that is a mosaic of the list of input raster maps. The purpose of such a VRT is to provide fast access to small subsets of the VRT, also with multiple simultaneous read requests. *r.buildvrt* creates a list of raster maps that can be located in different mapsets. The ouput is a read-only link to the original raster maps which is only valid if the original raster maps remain in the originally indicated mapset. A VRT can also be built from raster maps registered with *r.external*. Reading the whole VRT is slower than reading the equivalent single raster map. Only reading small parts of the VRT provides a performance benefit. Please test! Markus M ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
