Re: [GRASS-user] importing Mars DEM takes too long

2016-11-23 Thread Markus Neteler 
Hi Even,

On Mon, Nov 21, 2016 at 12:42 AM, Helmut Kudrnovsky wrote:
> Even Rouault wrote
...
>> I've committed in GDAL trunk r36357 "OpenJPEG: add a USE_TILE_AS_BLOCK=YES
>> open option that can help with whole image conversion"
>> thats add the open option :
...
>> USE_TILE_AS_BLOCK=YES/NO: (GDAL > 2.2) Whether to always use the
>> JPEG-2000 block size as the GDAL block size
>> Defaults to NO.
...
>
> Even, thanks for the insight and improvements on GDAL's level!

This is really great! And thanks for directly reading the grass-user
list :-) Luckily this change will also help other projects depending
on GDAL.

Is there a chance that you backport r36357 to GDAL 2.1 which will be
the version distributed through the various (Linux distro) channels
for a while?

thanks again,
Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Helmut Kudrnovsky 
Even Rouault-2 wrote
> On dimanche 20 novembre 2016 17:38:19 CET Anna Petrášová wrote:
>> On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky 

> hellik@

>  wrote:
>> >>11 seconds can be pretty long depending on the size of the file. Try
>> >>to convert it to tif and load again, it will be probably much faster.
>> >>
>> > the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
>> > 
>> > the import takes about 7 seconds.
>> > 
>> > there is some difference in importing time ; maybe cause of compressed
>> vs.
>> > uncompressed.
>> > 
>> > it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
>> > driver.
>> 
>> I just tested it on in wingrass both standalone and from OSGeo4W and
>> it's very slow with ~50MB JP2 file. I tested this one:
>> http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_02
>> 0528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2
>> 
>> How could you import it in 11 s? Maybe the problem appears only with
>> certain files?
> 
> When enabling GDAL debug messages (CPL_DEBUG=ON), one can see:
> 
> OPENJPEG: nX0 = 0
> OPENJPEG: nY0 = 0
> OPENJPEG: nTileW = 7552
> OPENJPEG: nTileH = 19113
> OPENJPEG: nTilesX = 1
> OPENJPEG: nTilesY = 1
> OPENJPEG: mct = 0
> OPENJPEG: psImage->x0 = 0
> OPENJPEG: psImage->y0 = 0
> OPENJPEG: psImage->x1 = 7552
> OPENJPEG: psImage->y1 = 19113
> OPENJPEG: psImage->numcomps = 1
> 
> The tile dimensions are the same as the image, meaning the image is single 
> tiled.
> The OpenJPEG library must thus decompress the whole time into memory since
> it 
> doesn't support decompressing a region of interest within a tile. To avoid 
> exposing a block size so large, GDAL exposes a 1024x1024 artificial
> blocking, 
> which will in fact the image to be decompressed each time one of those
> blocks 
> is read.
> 
> I've committed in GDAL trunk r36357 "OpenJPEG: add a USE_TILE_AS_BLOCK=YES 
> open option that can help with whole image conversion"
> thats add the open option :
> 
> USE_TILE_AS_BLOCK=YES/NO
*
> : (GDAL  2.2) Whether to always use the 
> JPEG-2000 block size as the GDAL block size
> Defaults to NO. Setting this option can be useful when doing whole image
> decompression and the image is single-tiled. Note however that the tile
> size
> must not exceed 2 GB since that's the limit supported by GDAL.
> 
> (note: this limitation of 2GB is reached unfortunately with other images
> as 
> ESP_020528_1750_IRB_A_01_ORTHO.JP2)
> 
> $ time GDAL_CACHEMAX=150 gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2  
> out.tif -co TILED=YES -oo USE_TILE_AS_BLOCK=YES -co COMPRESS=DEFLATE
> real  0m22.981s
> user  0m22.756s
> sys   0m0.220s
> 
> GDAL_CACHEMAX must be larger than width * height * data_type_size * 
> number_of_bands = 7552 * 19113 * 1 * 1 = 144 MB, so as to avoid the block 
> corresponding to the JPEG-2000 image to be evicted from the GDAL block
> cache.

Even, thanks for the insight and improvements on GDAL's level!



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296689.html
Sent from the Grass - Users mailing list archive at Nabble.com.
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Even Rouault 
On dimanche 20 novembre 2016 17:38:19 CET Anna Petrášová wrote:
> On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky  wrote:
> >>11 seconds can be pretty long depending on the size of the file. Try
> >>to convert it to tif and load again, it will be probably much faster.
> >>
> > the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
> > 
> > the import takes about 7 seconds.
> > 
> > there is some difference in importing time ; maybe cause of compressed vs.
> > uncompressed.
> > 
> > it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
> > driver.
> 
> I just tested it on in wingrass both standalone and from OSGeo4W and
> it's very slow with ~50MB JP2 file. I tested this one:
> http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_02
> 0528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2
> 
> How could you import it in 11 s? Maybe the problem appears only with
> certain files?

When enabling GDAL debug messages (CPL_DEBUG=ON), one can see:

OPENJPEG: nX0 = 0
OPENJPEG: nY0 = 0
OPENJPEG: nTileW = 7552
OPENJPEG: nTileH = 19113
OPENJPEG: nTilesX = 1
OPENJPEG: nTilesY = 1
OPENJPEG: mct = 0
OPENJPEG: psImage->x0 = 0
OPENJPEG: psImage->y0 = 0
OPENJPEG: psImage->x1 = 7552
OPENJPEG: psImage->y1 = 19113
OPENJPEG: psImage->numcomps = 1

The tile dimensions are the same as the image, meaning the image is single 
tiled.
The OpenJPEG library must thus decompress the whole time into memory since it 
doesn't support decompressing a region of interest within a tile. To avoid 
exposing a block size so large, GDAL exposes a 1024x1024 artificial blocking, 
which will in fact the image to be decompressed each time one of those blocks 
is read.

I've committed in GDAL trunk r36357 "OpenJPEG: add a USE_TILE_AS_BLOCK=YES 
open option that can help with whole image conversion"
thats add the open option :

USE_TILE_AS_BLOCK=YES/NO: (GDAL  2.2) Whether to always use the 
JPEG-2000 block size as the GDAL block size
Defaults to NO. Setting this option can be useful when doing whole image
decompression and the image is single-tiled. Note however that the tile size
must not exceed 2 GB since that's the limit supported by GDAL.

(note: this limitation of 2GB is reached unfortunately with other images as 
ESP_020528_1750_IRB_A_01_ORTHO.JP2)

$ time GDAL_CACHEMAX=150 gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2  
out.tif -co TILED=YES -oo USE_TILE_AS_BLOCK=YES -co COMPRESS=DEFLATE
real0m22.981s
user0m22.756s
sys 0m0.220s

GDAL_CACHEMAX must be larger than width * height * data_type_size * 
number_of_bands = 7552 * 19113 * 1 * 1 = 144 MB, so as to avoid the block 
corresponding to the JPEG-2000 image to be evicted from the GDAL block cache.

Even

> 
> Anna
> 
> > -
> > best regards
> > Helmut
> > --
> > View this message in context:
> > http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5
> > 296530p5296685.html Sent from the Grass - Users mailing list archive at
> > Nabble.com.
> > ___
> > grass-user mailing list
> > grass-user@lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/grass-user
> 
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Helmut Kudrnovsky 
Anna Petrášová wrote
> On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky 

> hellik@

>  wrote:
>>>11 seconds can be pretty long depending on the size of the file. Try
>>>to convert it to tif and load again, it will be probably much faster.
>>
>> the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
>>
>> the import takes about 7 seconds.
>>
>> there is some difference in importing time ; maybe cause of compressed
>> vs.
>> uncompressed.
>>
>> it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
>> driver.
> 
> I just tested it on in wingrass both standalone and from OSGeo4W and
> it's very slow with ~50MB JP2 file. I tested this one:
> http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_020528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2
> 
> How could you import it in 11 s? Maybe the problem appears only with
> certain files?

tested your file with
--
System Info 
GRASS version: 7.3.svn  
GRASS SVN revision: r69851  
Build date: 2016-11-20  
Build platform: x86_64-w64-mingw32  
GDAL: 2.1.2 
PROJ.4: 4.9.3   
GEOS: 3.5.0 
SQLite: 3.14.1  
Python: 2.7.5   
wxPython: 2.8.12.1  
Platform: Windows-8-6.2.9200 (OSGeo4W) 
--
gdalinfo --formats | grep JPEG2000
  JP2ECW -raster,vector- (rov): ERDAS JPEG2000 (SDK 5.0)
  JP2MrSID -raster- (rov): MrSID JPEG2000
--
gdal_translate -co COMPRESS=LZW -co PREDICTOR=2
ESP_020528_1750_RED_C_01_ORTHO.JP2
ESP_020528_1750_RED_C_01_ORTHO_compressed.tif
Input file size is 7552, 19113
0...10...20...30...40...50...60...70...80...90...100 - done.
--
gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2
ESP_020528_1750_RED_C_01_ORTHO_uncompressed.tif
Input file size is 7552, 19113
0...10...20...30...40...50...60...70...80...90...100 - done.
--
ls -l
total 268597
-rw-r--r-- 1  49609275 Nov 20 22:58 ESP_020528_1750_RED_C_01_ORTHO.JP2
-rw-r--r-- 1  80937523 Nov 20 23:00
ESP_020528_1750_RED_C_01_ORTHO_compressed.tif
-rw-r--r-- 1 144495106 Nov 20 23:01
ESP_020528_1750_RED_C_01_ORTHO_uncompressed.tif
--
r.import input=D:\dl\mars\ESP_020528_1750_RED_C_01_ORTHO.JP2
output=ESP_020528_1750_RED_C_01_ORTHO
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map ...
(Sun Nov 20 23:59:03 2016) Command finished (11 sec)  
--
r.import input=D:\dl\mars\ESP_020528_1750_RED_C_01_ORTHO_compressed.tif
output=ESP_020528_1750_RED_C_01_ORTHO_compressed
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map ...
(Mon Nov 21 00:04:39 2016) Command finished (6 sec)
--
r.import input=D:\dl\mars\ESP_020528_1750_RED_C_01_ORTHO_uncompressed.tif
output=ESP_020528_1750_RED_C_01_ORTHO_uncompressed
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map ...
(Mon Nov 21 00:05:41 2016) Command finished (5 sec) 
--

no special tuning here on my side, just used the OSGeo4W settings on a 8GB
RAM win 10 box.

there are differences in file size (compressed vs. uncompresssed) and
importing time (also compressed vs. uncompressed).

just have a look at GDAL website regarding jpeg2000 driver implementations
(http://www.gdal.org/formats_list.html):

JPEG2000 (.jp2, .j2k)   JPEG2000Yes Yes 2GiBNo, needs 
libjasper
JPEG2000 (.jp2, .j2k)   JP2ECW  Yes Yes 500MB   No, needs ECW SDK
JPEG2000 (.jp2, .j2k)   JP2KAK  Yes Yes No limits   No, needs 
Kakadu library
JPEG2000 (.jp2, .j2k)   JP2MrSIDYes Yes No, needs MrSID 
SDK
JPEG2000

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Anna Petrášová 
On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky  wrote:
>>11 seconds can be pretty long depending on the size of the file. Try
>>to convert it to tif and load again, it will be probably much faster.
>
> the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
>
> the import takes about 7 seconds.
>
> there is some difference in importing time ; maybe cause of compressed vs.
> uncompressed.
>
> it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
> driver.

I just tested it on in wingrass both standalone and from OSGeo4W and
it's very slow with ~50MB JP2 file. I tested this one:
http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_020528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2

How could you import it in 11 s? Maybe the problem appears only with
certain files?

Anna

>
>
>
> -
> best regards
> Helmut
> --
> View this message in context: 
> http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296685.html
> Sent from the Grass - Users mailing list archive at Nabble.com.
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Helmut Kudrnovsky 
>11 seconds can be pretty long depending on the size of the file. Try 
>to convert it to tif and load again, it will be probably much faster. 

the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).

the import takes about 7 seconds. 

there is some difference in importing time ; maybe cause of compressed vs.
uncompressed. 

it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
driver. 



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296685.html
Sent from the Grass - Users mailing list archive at Nabble.com.
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Markus Neteler 
On Sun, Nov 20, 2016 at 9:37 PM, Markus Neteler  wrote:
> On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová  wrote:
>> On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler  wrote:
>>> On Nov 19, 2016 11:55 PM, "Anna Petrášová"  wrote:

 On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler  wrote:
 > On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
>>> ...
 See above, I already tested with 2.1.2 (released in October).

FWIW: I have recompiled GDAL 2.1 for Fedora (extra repo):
https://copr.fedorainfracloud.org/coprs/neteler/GDAL/

At least Sentinel-2 is easier to read now :)

The Mars DEM remains slow also then (as you observed, too) with the
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Anna Petrášová 
On Sun, Nov 20, 2016 at 4:13 PM, Helmut Kudrnovsky  wrote:
> Markus Neteler wrote
>> On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová 
>
>> kratochanna@
>
>>  wrote:
>>> On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler 
>
>> neteler@
>
>>  wrote:
 On Nov 19, 2016 11:55 PM, "Anna Petrášová" 
>
>> kratochanna@
>
>>  wrote:
>
> On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler 
>
>> neteler@
>
>>  wrote:
> > On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
 ...
> See above, I already tested with 2.1.2 (released in October).

 Ok. And gdal_translate is equally slow with this dataset with respect to
 r.in.gdal?
>>>
>>> I haven't measured it, but it seems very slow as well. Do you think I
>>> should make a ticket for that for GDAL?
>>
>> This would be good, to shed some light here.
>>
>> If needed, here a Mars DEM for the ticket to simplify things for them:
>> http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750
>
> just tested one of the smaller ones:
>
> ESP_020673_1750_RED_C_01_ORTHO.JP2
>
> r.import input=D:\dl\mars\ESP_020673_1750_RED_C_01_ORTHO.JP2
> output=ESP_020673_1750_RED_C_01_ORTHO
> WARNING: Datum  not recognised by GRASS and no parameters found
> WARNING: Datum  not recognised by GRASS and no parameters found
> WARNING: Datum  not recognised by GRASS and no parameters found
> Proceeding with import of 1 raster bands...
> Importing raster map ...
> (Sun Nov 20 22:09:07 2016) Command finished (11 sec)
>
> by
>
> System Info
> GRASS version: 7.3.svn
> GRASS SVN revision: r69851
> Build date: 2016-11-20
> Build platform: x86_64-w64-mingw32
> GDAL: 2.1.2
> PROJ.4: 4.9.3
> GEOS: 3.5.0
> SQLite: 3.14.1
> Python: 2.7.5
> wxPython: 2.8.12.1
> Platform: Windows-8-6.2.9200 (OSGeo4W)
>
> no issues (slow or other) here.

11 seconds can be pretty long depending on the size of the file. Try
to convert it to tif and load again, it will be probably much faster.

>
>
>
> -
> best regards
> Helmut
> --
> View this message in context: 
> http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296680.html
> Sent from the Grass - Users mailing list archive at Nabble.com.
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Helmut Kudrnovsky 
Markus Neteler wrote
> On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová 

> kratochanna@

>  wrote:
>> On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler 

> neteler@

>  wrote:
>>> On Nov 19, 2016 11:55 PM, "Anna Petrášová" 

> kratochanna@

>  wrote:

 On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler 

> neteler@

>  wrote:
 > On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
>>> ...
 See above, I already tested with 2.1.2 (released in October).
>>>
>>> Ok. And gdal_translate is equally slow with this dataset with respect to
>>> r.in.gdal?
>>
>> I haven't measured it, but it seems very slow as well. Do you think I
>> should make a ticket for that for GDAL?
> 
> This would be good, to shed some light here.
> 
> If needed, here a Mars DEM for the ticket to simplify things for them:
> http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750

just tested one of the smaller ones: 

ESP_020673_1750_RED_C_01_ORTHO.JP2

r.import input=D:\dl\mars\ESP_020673_1750_RED_C_01_ORTHO.JP2
output=ESP_020673_1750_RED_C_01_ORTHO
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map ...
(Sun Nov 20 22:09:07 2016) Command finished (11 sec) 

by

System Info 
GRASS version: 7.3.svn  
GRASS SVN revision: r69851  
Build date: 2016-11-20  
Build platform: x86_64-w64-mingw32  
GDAL: 2.1.2 
PROJ.4: 4.9.3   
GEOS: 3.5.0 
SQLite: 3.14.1  
Python: 2.7.5   
wxPython: 2.8.12.1  
Platform: Windows-8-6.2.9200 (OSGeo4W) 

no issues (slow or other) here.



-
best regards
Helmut
--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296680.html
Sent from the Grass - Users mailing list archive at Nabble.com.
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Anna Petrášová 
On Sun, Nov 20, 2016 at 3:37 PM, Markus Neteler  wrote:
> On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová  wrote:
>> On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler  wrote:
>>> On Nov 19, 2016 11:55 PM, "Anna Petrášová"  wrote:

 On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler  wrote:
 > On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
>>> ...
 See above, I already tested with 2.1.2 (released in October).
>>>
>>> Ok. And gdal_translate is equally slow with this dataset with respect to
>>> r.in.gdal?
>>
>> I haven't measured it, but it seems very slow as well. Do you think I
>> should make a ticket for that for GDAL?
>
> This would be good, to shed some light here.
>
> If needed, here a Mars DEM for the ticket to simplify things for them:
> http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750

Yes, I got that file from somebody, but I just found these too. Thank you

>
> Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Markus Neteler 
On Sun, Nov 20, 2016 at 9:17 PM, Anna Petrášová  wrote:
> On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler  wrote:
>> On Nov 19, 2016 11:55 PM, "Anna Petrášová"  wrote:
>>>
>>> On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler  wrote:
>>> > On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
>> ...
>>> See above, I already tested with 2.1.2 (released in October).
>>
>> Ok. And gdal_translate is equally slow with this dataset with respect to
>> r.in.gdal?
>
> I haven't measured it, but it seems very slow as well. Do you think I
> should make a ticket for that for GDAL?

This would be good, to shed some light here.

If needed, here a Mars DEM for the ticket to simplify things for them:
http://www.uahirise.org/dtm/dtm.php?ID=ESP_020673_1750

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Anna Petrášová 
On Sun, Nov 20, 2016 at 3:02 AM, Markus Neteler  wrote:
> On Nov 19, 2016 11:55 PM, "Anna Petrášová"  wrote:
>>
>> On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler  wrote:
>> > On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
> ...
>> See above, I already tested with 2.1.2 (released in October).
>
> Ok. And gdal_translate is equally slow with this dataset with respect to
> r.in.gdal?

I haven't measured it, but it seems very slow as well. Do you think I
should make a ticket for that for GDAL?

>
> Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-20 Thread Markus Neteler 
On Nov 19, 2016 11:55 PM, "Anna Petrášová"  wrote:
>
> On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler  wrote:
> > On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová
...
> See above, I already tested with 2.1.2 (released in October).

Ok. And gdal_translate is equally slow with this dataset with respect to
r.in.gdal?

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-19 Thread Anna Petrášová 
On Sat, Nov 19, 2016 at 2:20 PM, Markus Neteler  wrote:
> On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová  wrote:
>> On Sat, Nov 19, 2016 at 10:55 AM, Markus Neteler  wrote:
>>> On Nov 19, 2016 2:08 PM, "Anna Petrášová"  wrote:
> ...
>>> Did you set the cache? Eg
>>>
>>> export GDAL_CACHEMAX=2000
>>
>> I installed GDAL 2.1.2 and set the cachemax, but it doesn't help, it's
>> still slow. I might try the ECW then.
>
> (got the file offlist)
>
> So, the file size is 154M. My test:
>
> gdal-config --version
> 2.0.2
>
> gdalinfo ESP_035583_2060_RED.JP2 | grep driver
> Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
>
>
> time -p grass72 -c ESP_035583_2060_RED.JP2 ~/grassdata/mars_dem --exec
> r.import input=ESP_035583_2060_RED.JP2 output=ESP_035583_2060_RED
> Cleaning up temporary files...
> Creating new GRASS GIS location/mapset...
> Executing  output=ESP_035583_2060_RED> ...
> WARNING: Datum  not recognised by GRASS and no parameters found
> WARNING: Datum  not recognised by GRASS and no parameters found
> WARNING: Datum  not recognised by GRASS and no parameters found
> Proceeding with import of 1 raster bands...
> Importing raster map ...
>
>  it takes forever.
>
> Note:
> The difference with QGIS is here that we import while QGIS is simply
> viewing it using the image pyramids which are useless when it comes to
> import.
>
> Can you make a test with GDAL 2.1?

See above, I already tested with 2.1.2 (released in October).

>
> Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-19 Thread Markus Neteler 
On Sat, Nov 19, 2016 at 6:17 PM, Anna Petrášová  wrote:
> On Sat, Nov 19, 2016 at 10:55 AM, Markus Neteler  wrote:
>> On Nov 19, 2016 2:08 PM, "Anna Petrášová"  wrote:
...
>> Did you set the cache? Eg
>>
>> export GDAL_CACHEMAX=2000
>
> I installed GDAL 2.1.2 and set the cachemax, but it doesn't help, it's
> still slow. I might try the ECW then.

(got the file offlist)

So, the file size is 154M. My test:

gdal-config --version
2.0.2

gdalinfo ESP_035583_2060_RED.JP2 | grep driver
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library


time -p grass72 -c ESP_035583_2060_RED.JP2 ~/grassdata/mars_dem --exec
r.import input=ESP_035583_2060_RED.JP2 output=ESP_035583_2060_RED
Cleaning up temporary files...
Creating new GRASS GIS location/mapset...
Executing  ...
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
WARNING: Datum  not recognised by GRASS and no parameters found
Proceeding with import of 1 raster bands...
Importing raster map ...

 it takes forever.

Note:
The difference with QGIS is here that we import while QGIS is simply
viewing it using the image pyramids which are useless when it comes to
import.

Can you make a test with GDAL 2.1?

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-19 Thread Anna Petrášová 
On Sat, Nov 19, 2016 at 10:55 AM, Markus Neteler  wrote:
> On Nov 19, 2016 2:08 PM, "Anna Petrášová"  wrote:
>>
>> On Sat, Nov 19, 2016 at 7:07 AM, Markus Neteler  wrote:
>> > On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth 
>> > wrote:
>> >> Hi Anna
>> >>
>> >> I am not sure if it is related, but I had similar issues with Sentinel
>> >> JP2
>> >> files.
>> >
>> > .. me too, but...:
>> >
>> >> The solution was to use the ECW library to read the JP2 files instead
>> >> of
>> >> JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.
>> >>
>> >> I had to recompile gdal to add the ECW support.
>> >
>> > I did that for some weeks, then dropped it (also because of the
>> > license situation and the packaging pain).
>> > Recent GDAL seems to be faster due to driver improvements.
>> >
>> > Anna: which GAL version do you use?
>> >
>> > Markus
>>
>> GDAL 1.11.3, released 2015/09/16, from Ubuntu 16.04 packages
>
> That's a bit oldish for JP2000.
>
>> I wonder why QGIS was able to read it without problems? I would think
>> they use GDAL too?
>
> Did you set the cache? Eg
>
> export GDAL_CACHEMAX=2000

I installed GDAL 2.1.2 and set the cachemax, but it doesn't help, it's
still slow. I might try the ECW then.

>
> Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-19 Thread Markus Neteler 
On Nov 19, 2016 2:08 PM, "Anna Petrášová"  wrote:
>
> On Sat, Nov 19, 2016 at 7:07 AM, Markus Neteler  wrote:
> > On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth 
wrote:
> >> Hi Anna
> >>
> >> I am not sure if it is related, but I had similar issues with Sentinel
JP2
> >> files.
> >
> > .. me too, but...:
> >
> >> The solution was to use the ECW library to read the JP2 files instead
of
> >> JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.
> >>
> >> I had to recompile gdal to add the ECW support.
> >
> > I did that for some weeks, then dropped it (also because of the
> > license situation and the packaging pain).
> > Recent GDAL seems to be faster due to driver improvements.
> >
> > Anna: which GAL version do you use?
> >
> > Markus
>
> GDAL 1.11.3, released 2015/09/16, from Ubuntu 16.04 packages

That's a bit oldish for JP2000.

> I wonder why QGIS was able to read it without problems? I would think
> they use GDAL too?

Did you set the cache? Eg

export GDAL_CACHEMAX=2000

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-19 Thread Anna Petrášová 
On Sat, Nov 19, 2016 at 7:07 AM, Markus Neteler  wrote:
> On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth  wrote:
>> Hi Anna
>>
>> I am not sure if it is related, but I had similar issues with Sentinel JP2
>> files.
>
> .. me too, but...:
>
>> The solution was to use the ECW library to read the JP2 files instead of
>> JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.
>>
>> I had to recompile gdal to add the ECW support.
>
> I did that for some weeks, then dropped it (also because of the
> license situation and the packaging pain).
> Recent GDAL seems to be faster due to driver improvements.
>
> Anna: which GAL version do you use?
>
> Markus

GDAL 1.11.3, released 2015/09/16, from Ubuntu 16.04 packages

I wonder why QGIS was able to read it without problems? I would think
they use GDAL too?

I'll try to get the latest GDAL and test again. Thanks both of you.

Anna
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-19 Thread Markus Neteler 
On Sat, Nov 19, 2016 at 12:06 PM, Sajid Pareeth  wrote:
> Hi Anna
>
> I am not sure if it is related, but I had similar issues with Sentinel JP2
> files.

.. me too, but...:

> The solution was to use the ECW library to read the JP2 files instead of
> JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.
>
> I had to recompile gdal to add the ECW support.

I did that for some weeks, then dropped it (also because of the
license situation and the packaging pain).
Recent GDAL seems to be faster due to driver improvements.

Anna: which GAL version do you use?

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] importing Mars DEM takes too long

2016-11-19 Thread Sajid Pareeth 
Hi Anna

I am not sure if it is related, but I had similar issues with Sentinel JP2
files.
The solution was to use the ECW library to read the JP2 files instead of
JP2OpenJPEG/JPEG-2000 driver with OpenJPEG library.

I had to recompile gdal to add the ECW support.

## Add ECW support to GDAL
./configure --with-ecw=/usr/local
...
## To read a jp2 file using ecw plugin, switch off the other two drivers
export GDAL_SKIP=JP2OpenJPEG,JPEG2000


Ofcourse you have to install libecw before recompiling gdal.

Hope it helps.

Sajid



On Fri, Nov 18, 2016 at 10:08 PM, Anna Petrášová 
wrote:

> Hi,
>
> I was trying to import Mars DEM with r.in.gdal, and it takes forever
> to finish it (It showed 3% after 10 minutes). This is what gdalinfo
> says:
>
> > gdalinfo /tmp/ESP_035583_2060_RED.JP2
> Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
> Files: /tmp/ESP_035583_2060_RED.JP2
>/tmp/ESP_035583_2060_RED.JP2.aux.xml
> Size is 13094, 26340
> Coordinate System is:
> PROJCS["Equirectangular MARS",
> GEOGCS["GCS_MARS",
> DATUM["unknown",
> SPHEROID["unnamed",3392593.6110435,0]],
> PRIMEM["Greenwich",0],
> UNIT["degree",0.0174532925199433]],
> PROJECTION["Equirectangular"],
> PARAMETER["latitude_of_origin",25],
> PARAMETER["central_meridian",180],
> PARAMETER["standard_parallel_1",0],
> PARAMETER["false_easting",0],
> PARAMETER["false_northing",0],
> UNIT["metre",1,
> AUTHORITY["EPSG","9001"]]]
> Origin = (3889293.250,1519463.750)
> Pixel Size = (0.500,-0.500)
> Corner Coordinates:
> Upper Left  ( 3889293.250, 1519463.750) (114d18'56.57"W, 50d39'41.21"N)
> Lower Left  ( 3889293.250, 1506293.750) (114d18'56.57"W, 50d26'20.49"N)
> Upper Right ( 3895840.250, 1519463.750) (114d12'18.52"W, 50d39'41.21"N)
> Lower Right ( 3895840.250, 1506293.750) (114d12'18.52"W, 50d26'20.49"N)
> Center  ( 3892566.750, 1512878.750) (114d15'37.55"W, 50d33' 0.85"N)
> Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Undefined
>   Overviews: 6547x13170, 3273x6585, 1636x3292, 818x1646, 409x823,
> 204x411, 102x205
>   Overviews: arbitrary
>   Image Structure Metadata:
> NBITS=10
>
> GRASS location was created from this file:
>
> projection: 99 (Equirectangular MARS)
> zone:   0
> datum:  ** unknown (default: WGS84) **
> ellipsoid:  a=3392593.6110435 es=0
> north:  1519463.75
> south:  1506293.75
> west:   3889293.25
> east:   3895840.25
> nsres:  0.5
> ewres:  0.5
> rows:   26340
> cols:   13094
> cells:  344895960
>
> I tried to open it in QGIS and it appeared instantly. Then, I tried to
> export it from QGIS as Geotiff, but that took again very long (I
> didn't wait for it to finish). Running gdal_translate in command line
> can't finish either. I tried to debug r.in.gdal and it seems to be
> stuck on line 1103 in  GDALRasterIO call. Any advice? I can send the
> data privately.
>
> Thank you,
>
> Anna
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

[GRASS-user] importing Mars DEM takes too long

2016-11-18 Thread Anna Petrášová 
Hi,

I was trying to import Mars DEM with r.in.gdal, and it takes forever
to finish it (It showed 3% after 10 minutes). This is what gdalinfo
says:

> gdalinfo /tmp/ESP_035583_2060_RED.JP2
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
Files: /tmp/ESP_035583_2060_RED.JP2
   /tmp/ESP_035583_2060_RED.JP2.aux.xml
Size is 13094, 26340
Coordinate System is:
PROJCS["Equirectangular MARS",
GEOGCS["GCS_MARS",
DATUM["unknown",
SPHEROID["unnamed",3392593.6110435,0]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433]],
PROJECTION["Equirectangular"],
PARAMETER["latitude_of_origin",25],
PARAMETER["central_meridian",180],
PARAMETER["standard_parallel_1",0],
PARAMETER["false_easting",0],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]]]
Origin = (3889293.250,1519463.750)
Pixel Size = (0.500,-0.500)
Corner Coordinates:
Upper Left  ( 3889293.250, 1519463.750) (114d18'56.57"W, 50d39'41.21"N)
Lower Left  ( 3889293.250, 1506293.750) (114d18'56.57"W, 50d26'20.49"N)
Upper Right ( 3895840.250, 1519463.750) (114d12'18.52"W, 50d39'41.21"N)
Lower Right ( 3895840.250, 1506293.750) (114d12'18.52"W, 50d26'20.49"N)
Center  ( 3892566.750, 1512878.750) (114d15'37.55"W, 50d33' 0.85"N)
Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Undefined
  Overviews: 6547x13170, 3273x6585, 1636x3292, 818x1646, 409x823,
204x411, 102x205
  Overviews: arbitrary
  Image Structure Metadata:
NBITS=10

GRASS location was created from this file:

projection: 99 (Equirectangular MARS)
zone:   0
datum:  ** unknown (default: WGS84) **
ellipsoid:  a=3392593.6110435 es=0
north:  1519463.75
south:  1506293.75
west:   3889293.25
east:   3895840.25
nsres:  0.5
ewres:  0.5
rows:   26340
cols:   13094
cells:  344895960

I tried to open it in QGIS and it appeared instantly. Then, I tried to
export it from QGIS as Geotiff, but that took again very long (I
didn't wait for it to finish). Running gdal_translate in command line
can't finish either. I tried to debug r.in.gdal and it seems to be
stuck on line 1103 in  GDALRasterIO call. Any advice? I can send the
data privately.

Thank you,

Anna
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user