On Wed, Mar 22, 2017 at 9:52 PM, Markus Neteler <[email protected]> wrote: > > On Wed, Mar 22, 2017 at 9:28 PM, Markus Metz > <[email protected]> wrote: > > On Wed, Mar 22, 2017 at 8:12 PM, Markus Neteler <[email protected]> wrote: > ... > >> Nikos, for an even bigger map try > >> > >> Global Surface Water (2000-2012, 30 m, Data coverage is from 80° north > >> to 60° south): > >> http://landcover.usgs.gov/glc/WaterDescriptionAndDownloads.php > >> by USGS. 1.6GB in size. > >> > >> Using gdalbuildvrt I created a VRT from the 504 GeoTIFF files. > >> > >> After import into GRASS GIS, here the timings: > >> > >> # final map size: > >> g.region -p > >> ... > >> rows: 493200 > >> cols: 1296001 > >> cells: 639187693200 > >> > >> (handling only works in GRASS GIS 7.3.svn since Markus Metz's recent > >> improvements on global data import are needed). > > > > (my changes were bug fixes, not improvements) > > > >> > >> Benchmarks: > >> - Import took 2h while reading the data from a CIFS mounted storage > >> box (slow) and writing on SSD. > >> - Displaying the entire map (639 giga-pixel) in GRASS GIS' display > >> (d.mon) took ~15 sec over a ssh tunnel from my laptop to the server, > >> since I am at a conference. > >> > >> Fair deal I would say :-) > > > > A bit more information would help to compare: > > - what is your GDAL version? > > GDAL 2.1.2 > > > - are 504 GeoTIFF files compressed? If yes, which method? > > Yes, COMPRESSION=LZW > > > - what are the block dimensions of the input GeoTIFFs? > > Size is 36001, 36001 - Block=36001x1
This is row by row compression as in GRASS. That could help import with r.in.gdal which also reads and writes row by row. > Type=Byte > > > - what kind of GRASS compression did you use? > > Default raster + NULL compression enabled. I.e., > > r.compress -p watermask2010 > <watermask2010> is compressed (method 2: ZLIB). Data type: CELL You might save disk space at the cost of longer reading times with BZIP2. > <watermask2010> has a compressed NULL file > > Again, the fact that I had to read from an attached storage box likely > slowed down the import. > Just thought to post these numbers here. Impressive that such a large raster can be imported at all, and relatively fast! Reading about 1.6 GB (also from an attached storage box) should not take 2 hours, therefore I think the limit is software input decompression and output compression. Markus M
_______________________________________________ grass-user mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/grass-user
