Re: [GRASS-user] Itzi output: several questions

[Rich Shepard]

On Tue, 14 Mar 2017, Anna Petrášová wrote:


look at second 41 of the video, that adds vector


  Thanks, Anna.

Rich
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Itzi output: several questions

[Anna Petrášová]

On Tue, Mar 14, 2017 at 6:34 PM, Rich Shepard  wrote:
> On Mon, 13 Mar 2017, Anna Petrášová wrote:
>
>> perhaps this video will help:
>> https://www.youtube.com/watch?v=Pi1ESk9iPwk
>
>
> Anna,
>
>   It played too quickly for me to see where you clicked in the anaimation
> window to add maps to the strds.
>
>   I have the strds animated and I want to underlay it with vector contour
> and road maps. What icon allows me to do this?

look at second 41 of the video, that adds vector

Anna

>
>
> TIA,
>
> Rich
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Itzi output: several questions

[Laurent C.]

2017-03-14 16:01 GMT-06:00 Rich Shepard :
> On Mon, 13 Mar 2017, Laurent C. wrote:
>
>> The water level of 3m seems to be due to accumulation on the edge of the
>> domain. The water flows according to topography. If there is a lower point
>> without an exit, there will be an accumulation of water. If you want a map
>> in cm or mm, you can convert it using r.mapcalc.
>
>
> Laurent,
>
>   If I copy the strds can I run r.mapcalc on all the maps in it to convert
> from meters to centimeters? Have not before worked with s-t data sets so I'm
> unfamiliar with applying r.mapcalc collectively to a set of raster maps.

You might try t.rast.mapcalc.

Laurent
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Itzi output: several questions

[Rich Shepard]

On Mon, 13 Mar 2017, Anna Petrášová wrote:


perhaps this video will help:
https://www.youtube.com/watch?v=Pi1ESk9iPwk


Anna,

  It played too quickly for me to see where you clicked in the anaimation
window to add maps to the strds.

  I have the strds animated and I want to underlay it with vector contour
and road maps. What icon allows me to do this?

TIA,

Rich
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

[GRASS-user] t.rast.colors [was] Itzi output: several questions

[Rich Shepard]

On Mon, 13 Mar 2017, Laurent C. wrote:


This seems to be a color table problem. You can change it with r.colors or
t.rast.colors.


  There's an issue with t.rast.colors in grass-7.3.svn (updated yesterday
but I did not write down the build number and cannot find it in the
directory).

  Using the GUI and the Layer Manager, the Temporal drop down menu includes
the item, "Raster data set color table [t.rast.colors]". When clicked,
nothing happens except there's a brief vertical space in the menu display
before it closes.

Rich
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Slow import of GHSL

[Nikos Alexandris]

[all deleted]

Here's a non-elegant way, derived out of tests. Maybe a starter for the
Wiki. Elegant would be scripted, no need to manually enter any GRASS
session.


```
# Get Eurostat's NUTS_2013_01M_SH.zip vector map
unzip NUTS_2013_01M_SH.zip && cd NUTS_2013_01M_SH/data/
grass73 -c NUTS_2013_01M_SH.shp /geo/grassdb/europe/etrs89
v.import in=NUTS_RG_01M_2013.shp out=NUTS_RG_01M_2013

# draw, view, pick & set computational region of interest, create a vector map
v.in.region out=europe_less_box

# Clip original VRT to Europe’s extent, output as VRT
# and successively add overviews !Adding overviews takes time!

# 1990
gdal_translate -projwin -3480828.507849 11465936.382472 4989400.357796 
3203413.703282 GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt 
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt -of VRT
gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE 
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt 2 4 8 16

# 2000
gdal_translate -projwin -3480828.507849 11465936.382472 4989400.357796 
3203413.703282 GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0.vrt 
GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt -of VRT
gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE 
GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0_Europe.tif 2 4 8 16

# 2014
gdal_translate -projwin -3480828.507849 11465936.382472 4989400.357796 
3203413.703282 GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38_v1_0.vrt 
GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt -of VRT
gdaladdo -ro --config COMPRESS_OVERVIEW DEFLATE 
GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt  2 4 8 16

# Create 'epsg:3857' location for the 1990 data, set region &
 # work this in three different terminals
# resolution (v.proj-ing existing "box" map), import raster
grass72 -c "epsg:3857" /geo/grassdb/global/wgs84_3857_1990
v.proj dbase=/geo/grassdb/europe/ location=etrs89 mapset=PERMANENT 
in=europe_less_box out=europe_less_box
g.region -p vect=europe_less_box_epsg_3857 ewres=38.218470987084757 
nsres=38.218446797782505
r.import input=GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt 
out=GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_Europe_Less memory=2047 
extent=region

# Repeat for 2000
grass72 -c "epsg:3857" /geo/grassdb/global/wgs84_3857_2000
v.proj dbase=/geo/grassdb/europe/ location=etrs89 mapset=PERMANENT 
in=europe_less_box out=europe_less_box
g.region -p vect=europe_less_box_epsg_3857 ewres=38.218470987084757 
nsres=38.218446797782505
r.import input=GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt 
out=GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38_v1_0_Europe_Less memory=2047 
extent=region

# Repeat for 2014
grass72 -c "epsg:3857" /geo/grassdb/global/wgs84_3857_2014
v.proj dbase=/geo/grassdb/europe/ location=etrs89 mapset=PERMANENT 
in=europe_less_box out=europe_less_box
g.region -p vect=europe_less_box_epsg_3857 ewres=38.218470987084757 
nsres=38.218446797782505
r.import input=GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38_v1_0_Europe_Less.vrt 
out=GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38_v1_0_Europe_Less memory=2047 
extent=region
```

Nikos
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Slow import of GHSL

[Nikos Alexandris]

* Markus Metz  [2017-03-14 15:02:30 +0100]:


On Tue, Mar 14, 2017 at 10:01 AM, Nikos Alexandris 
wrote:


Nikos Alexandris


Why does (attempting to) import a 38m pixel resolution GHSL [0]

GeoTIFF

layer, ie GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif, in

GRASS'

db progress slow?



Markus M



because it is a very large raster map: Size is 507904, 647168




(Apologies for cross-posting to gdal-dev)



Markus Neteler:


Can you elaborate a bit more? I have downloaded and checked:

That is 9835059101  bytes in 19885 files or I downloaded the wrong one
(please post an URL).



For example ,

see

GHS_BUILT_LDS1975_GLOBE_R2016A_3857_38 (768MB)


GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38 (854MB)
GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38 (892MB)
GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38 (900MB)



"3857" is the EPSG code.  They are split in two GeoTIFFs (p1, p2) and
there is a VRT along with overviews for it.  No overviews for the TIFFs.

For example:

GHSL_data_access_v1.3.pdf
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.clr
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt.ovr
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p2.tif


Even trying to clip, with gdal_translate, might create file(s) of
hundreds of GBs. This might be due to missing compression.




then use compression. The source tiffs use LZW with blocks of 4096x4096
cells.





The import of p1 or p2 or of the VRT file in GRASS' data base, via
r.in.gdal/r.import, does not progress at all.




Importing GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif with

r.in.gdal

took 1:31 hours on a laptop with SSD. The resultant cell file was 1.5 GB.

Recompressing with BZIP2 took 2:20 hours and the size of the cell file

was

reduced to a mere 143 MB.



Some messy rough timings:

1) i7, 8 cores, 32GB RAM, Base OS: CentOS -> Three r.in.gdal processes
for "p2.tif", each stuck at 3% for almost 14h

2) Xeon, 24 Cores, 32GB RAM, Base OS: Windows -> Three gdal_translate
processes with -projwin, the VRT file as an input and GeoTIFF as output,
at 40% since yesterday afternoon

3) Xeon, 12 Cores, ? RAM, Base OS: CentOS.jpg -> Same processes as in
1), stuck at 0% of progress for more than 16h.

SSD can be seen as a "necessity".


Hmm, not really. With the p1 tif and GRASS db on the same spinning HDD, and
6 other heavy processes constantly reading from and writing to that same
HDD, r.in.gdal took 2h 13min to import the p1 tif. 360 MB as input and 1.5
GB as output is not that heavy on disk IO. Most of the time is spent
decompressing input and compressing output.

Are your r.in.gdal and gdal_translate processes running at nearly 100% CPU?
Anything slowing down the HDD(s)?

Markus M


Ehm, maybe GDAL version 1.11.4? Just realised!
Working in restricted environment, time spent to configure things.
Will update...

Nikos
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Slow import of GHSL

[Markus Metz]

On Tue, Mar 14, 2017 at 10:01 AM, Nikos Alexandris 
wrote:
>
> Nikos Alexandris
>
> Why does (attempting to) import a 38m pixel resolution GHSL [0]
GeoTIFF
> layer, ie GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif, in
GRASS'
> db progress slow?
>
>
> Markus M
>
>
>> because it is a very large raster map: Size is 507904, 647168
>
>
>>> (Apologies for cross-posting to gdal-dev)
>
>
> Markus Neteler:
>
 Can you elaborate a bit more? I have downloaded and checked:

 That is 9835059101  bytes in 19885 files or I downloaded the wrong one
 (please post an URL).
>>>
>>>
>>> For example ,
>>>
>>> see
>>>
>>> GHS_BUILT_LDS1975_GLOBE_R2016A_3857_38 (768MB)
>>
>> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38 (854MB)
>> GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38 (892MB)
>> GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38 (900MB)
>>>
>>>
>>> "3857" is the EPSG code.  They are split in two GeoTIFFs (p1, p2) and
>>> there is a VRT along with overviews for it.  No overviews for the TIFFs.
>>>
>>> For example:
>>>
>>> GHSL_data_access_v1.3.pdf
>>> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.clr
>>> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt
>>> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt.ovr
>>> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif
>>> GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p2.tif
>>>
>>>
>>> Even trying to clip, with gdal_translate, might create file(s) of
>>> hundreds of GBs. This might be due to missing compression.
>
>
>> then use compression. The source tiffs use LZW with blocks of 4096x4096
>> cells.
>
>
>
>>> The import of p1 or p2 or of the VRT file in GRASS' data base, via
>>> r.in.gdal/r.import, does not progress at all.
>
>
>> Importing GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif with
r.in.gdal
>> took 1:31 hours on a laptop with SSD. The resultant cell file was 1.5 GB.
>>
>> Recompressing with BZIP2 took 2:20 hours and the size of the cell file
was
>> reduced to a mere 143 MB.
>
>
> Some messy rough timings:
>
> 1) i7, 8 cores, 32GB RAM, Base OS: CentOS -> Three r.in.gdal processes
> for "p2.tif", each stuck at 3% for almost 14h
>
> 2) Xeon, 24 Cores, 32GB RAM, Base OS: Windows -> Three gdal_translate
> processes with -projwin, the VRT file as an input and GeoTIFF as output,
> at 40% since yesterday afternoon
>
> 3) Xeon, 12 Cores, ? RAM, Base OS: CentOS.jpg -> Same processes as in
> 1), stuck at 0% of progress for more than 16h.
>
> SSD can be seen as a "necessity".

Hmm, not really. With the p1 tif and GRASS db on the same spinning HDD, and
6 other heavy processes constantly reading from and writing to that same
HDD, r.in.gdal took 2h 13min to import the p1 tif. 360 MB as input and 1.5
GB as output is not that heavy on disk IO. Most of the time is spent
decompressing input and compressing output.

Are your r.in.gdal and gdal_translate processes running at nearly 100% CPU?
Anything slowing down the HDD(s)?

Markus M
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Slow import of GHSL

[Nikos Alexandris]

Nikos Alexandris


Why does (attempting to) import a 38m pixel resolution GHSL [0] GeoTIFF
layer, ie GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif, in GRASS'
db progress slow?


Markus M


because it is a very large raster map: Size is 507904, 647168



(Apologies for cross-posting to gdal-dev)


Markus Neteler:


Can you elaborate a bit more? I have downloaded and checked:

That is 9835059101  bytes in 19885 files or I downloaded the wrong one
(please post an URL).


For example ,

see

GHS_BUILT_LDS1975_GLOBE_R2016A_3857_38 (768MB)

GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38 (854MB)
GHS_BUILT_LDS2000_GLOBE_R2016A_3857_38 (892MB)
GHS_BUILT_LDS2014_GLOBE_R2016A_3857_38 (900MB)


"3857" is the EPSG code.  They are split in two GeoTIFFs (p1, p2) and
there is a VRT along with overviews for it.  No overviews for the TIFFs.

For example:

GHSL_data_access_v1.3.pdf
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.clr
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0.vrt.ovr
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif
GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p2.tif


Even trying to clip, with gdal_translate, might create file(s) of
hundreds of GBs. This might be due to missing compression.



then use compression. The source tiffs use LZW with blocks of 4096x4096
cells.




The import of p1 or p2 or of the VRT file in GRASS' data base, via
r.in.gdal/r.import, does not progress at all.



Importing GHS_BUILT_LDS1990_GLOBE_R2016A_3857_38_v1_0_p1.tif with r.in.gdal
took 1:31 hours on a laptop with SSD. The resultant cell file was 1.5 GB.

Recompressing with BZIP2 took 2:20 hours and the size of the cell file was
reduced to a mere 143 MB.


Some messy rough timings:

1) i7, 8 cores, 32GB RAM, Base OS: CentOS -> Three r.in.gdal processes
for "p2.tif", each stuck at 3% for almost 14h

2) Xeon, 24 Cores, 32GB RAM, Base OS: Windows -> Three gdal_translate
processes with -projwin, the VRT file as an input and GeoTIFF as output,
at 40% since yesterday afternoon

3) Xeon, 12 Cores, ? RAM, Base OS: CentOS.jpg -> Same processes as in
1), stuck at 0% of progress for more than 16h.

SSD can be seen as a "necessity".

Nikos

[rest deleted]
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user