Re: [gdal-dev] ElasticSearch and native geometry

[Jonathan Moules]

Hi Jukka,
I'm still very new to ElasticSearch (so take with plenty of salt!), but 
if you want to keep the geom in _source and calculate on the fly, I 
guess you could maybe try it with one of the scripting languages: 
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html 
- Python's on there so then you'd have access to all of the py-geo stuff.


Of course, assuming it worked, the trade-off would be reduced storage 
for (considerably?) increase processing usage and complexity.


The other option would be to store the native geometry as a "binary" 
datatype - 
https://www.elastic.co/guide/en/elasticsearch/reference/current/binary.html


Cheers,
Jonathan


On 23/11/2016 15:41, Rahkonen Jukka (MML) wrote:


Hi,

What if somebody would like to use ElasticSearch for queries but still 
somehow get the native geometries included in the output?  The 
geometry must be re-projected into EPSG:4326 for creating either 
geo_point or geo_shape, but can anybody suggest a clever way for 
saving the native geometry? It is of course possible to save it into a 
text field for example as WKT of JSON but could it be possible to keep 
the original geometry in “_source” and just create the geo_point or 
geo_shape on-the-fly or the index?


-Jukka Rahkonen-



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Unable to extract features from KML

[Michael Z Freeman]

Hi Even Rouault,

Thanks. I had not found those SQL dialect docs for some reason so no
wonder it was not making sense.

I am now using ...

ogr2ogr -overwrite -f PostgreSQL PG:"dbname=original_hirise"
Beyer_Hirise_Areas_KML/MC-01-0-0-0.kml -nln gdal_hirise_from_kml -a_srs
"EPSG:4326" -dialect SQLite -sql "SELECT ST_CollectionExtract(geometry, 1)
AS postgis_point, ST_CollectionExtract(geometry, 3) AS postgis_polygon, *
FROM 'MC-1, Mare Boreum (North Pole), 0-0-0'"

I can access the database using QGIS and all the Polygons and Markers
appear in the correct locations. Now to import the entire HiRise feature
set :)

Michael Z Freeman

On Thu, November 24, 2016 5:03 pm, Even Rouault wrote:
> On jeudi 24 novembre 2016 16:54:31 CET Michael Z Freeman wrote:
>
>> Hi,
>>
>>
>> Still having trouble with this after updating to ogr2ogr/gdal 2.1.1.
>>
>>
>> Some of it is undoubtedly my unfamiliarity with GDAL. I'm on the steep
>> part of the learning curve ! But fun.
>>
>> "ogr2ogr -f PostgreSQL PG:"dbname=original_hirise" -a_srs "EPSG:4326"
>> -lco
>> OVERWRITE=YES -explodecollections -nlt geometry
>> Beyer_Hirise_Areas_KML/MC-01-0-0-0.kml"
>>
>>
>> ... creates (an edited CSV exported from a PGAdmin4 query) ...
>>
>>
>> "ogc_fid","name","description","wkb_geometry"
>> 1,"ESP_018108_2680","[...]","010120E6100AD7A3703D6A3EC0744694F60
>> 6035
>> 640"
>> 2,"ESP_018108_2680","[...]","010320E61001000500A4703D0AD
>> 723
>>
>>
> 3CC079E9263108FC554000C03FC0F4FDD478E9FA5540F0A7C64B377940C0992A1
>  895
>
>>
> D4095640FCA9F1D24DE23CC002BC0512140B5640A4703D0AD7233CC079E9263108FC5540
> "
>
>> 3,"ESP_018318_2675","[...]","010120E610A245B6F3FD543EC0728A8EE4
>> F2DF
>> 5540"
>> 4,"ESP_018318_2675","[...]","010320E610010005004260E5D02
>> 23B
>>
>>
> 3BC00C022B8716D95540AAF1D24D62503DC0FED478E926D55540448B6CE7FBD940C0C36
> 4AA60
>
>>
> 54E6554039B4C876BE7F3FC06D567DAEB6EA55404260E5D0223B3BC00C022B8716D95540
> "
>
>>
>> As you can see the geometries from the KML are exploded ... the short
>> WKB
>> is a point, the longer one POLYGON geometry. However two database
>> entries are created for each feature ("name"). I could just merge these
>> but I feel that OGR can create two geometry columns each for POINT and
>> POLYGON, in
>> fact this seems to be explicitly stated that it can do this in the news
>> release for GDAL version 2.
>
> What you observe is the intended behaviour of -explodecollections, but
> not what you want.
>
>>
>> Should I be using some kind of SQL statement on the OGR command line to
>>  make it do this ?
>
> You can indeed use Spatialite SQL to do what you want if you specify the
> SQLite dialect
> (http://gdal.org/ogr_sql_sqlite.html). If you look at
> http://www.gaia-gis.it/gaia-sins/
> spatialite-sql-4.3.0.html, you can see there's a ST_CollectionExtract
> function as well.
>
> So something like
>
>
> ogr2ogr -overwrite PG:x your.kml -dialect SQLite -sql "SELECT
> ST_CollectionExtract(geometry, 1) AS the_point,
> ST_COLLECTIONExtract(geometry, 3) AS
> the_polygon, * FROM the_layer"
>
> Even
>
>
>> Michael Z Freeman
>>
>>
>> On Mon, November 21, 2016 7:30 am, jratike80 wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> The main problem is that KML support such data models which do not
>>> suit well with the simple feature model of GDAL. You can get some
>>> information about that by reading http://www.gdal.org/drv_libkml.html.
>>>
>>>
>>> In your case each feature has two geometries: one point and one
>>> polygon. Best that can be done with the simple feature model it to
>>> push them both into one geometry which in that case must be a geometry
>>> collection. Here is one example from your data
>>>
>>> GEOMETRYCOLLECTION (POINT (-1.282 87.1064),POLYGON ((3.02457
>>> 87.0792,2.22294
>>> 86.9806,-5.655 87.1168,-5.098 87.2201,3.02457 87.0792)))
>>>
>>>
>>>
>>> You can save geometry collections into PostGIS if you use ogr2ogr
>>> with switch "-nlt geometry". However, saving geometry collections into
>>> PostGIS
>>> may not solve your problem because only few GIS programs can handle
>>> geometry collections well.
>>>
>>> A better option for you might be to explode the collections and save
>>> points and polygons as separate features into PostGIS. The ogr2ogr
>>> command to use is like
>>>
>>> ogr2ogr -f PGDump output_pgdump -nlt geometry -explodecollections
>>> Beyer_Hirise_Areas_KML/MC-01-0-0-0.kml
>>>
>>>
>>>
>>> If you want to have a quick look at your data, use JML as
>>> outputformat and open the result with OpenJUMP. It shows both the
>>> geometry collections and exploded collections fine for me with names
>>> and descriptions.
>>>
>>> -Jukka Rahkonen-
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://osgeo-org.1560.x6.nabble.com/gdal-dev-Unable-to-extract-feature
>>> s-f rom-KML-tp5296670p5296708.html Sent from the GDAL - Dev mailing
>>> list archive at Nabble.com.
>>> ___
>>> gdal-dev mailing list 

Re: [gdal-dev] Unable to extract features from KML

[Even Rouault]

On jeudi 24 novembre 2016 16:54:31 CET Michael Z Freeman wrote:
> Hi,
> 
> Still having trouble with this after updating to ogr2ogr/gdal 2.1.1.
> 
> Some of it is undoubtedly my unfamiliarity with GDAL. I'm on the steep
> part of the learning curve ! But fun.
> 
> "ogr2ogr -f PostgreSQL PG:"dbname=original_hirise" -a_srs "EPSG:4326" -lco
> OVERWRITE=YES -explodecollections -nlt geometry
> Beyer_Hirise_Areas_KML/MC-01-0-0-0.kml"
> 
> ... creates (an edited CSV exported from a PGAdmin4 query) ...
> 
> "ogc_fid","name","description","wkb_geometry"
> 1,"ESP_018108_2680","[...]","010120E6100AD7A3703D6A3EC0744694F606035
> 640"
> 2,"ESP_018108_2680","[...]","010320E61001000500A4703D0AD723
> 
3CC079E9263108FC554000C03FC0F4FDD478E9FA5540F0A7C64B377940C0992A1
895
> 
D4095640FCA9F1D24DE23CC002BC0512140B5640A4703D0AD7233CC079E9263108FC5540
"
> 3,"ESP_018318_2675","[...]","010120E610A245B6F3FD543EC0728A8EE4F2DF
> 5540"
> 4,"ESP_018318_2675","[...]","010320E610010005004260E5D0223B
> 
3BC00C022B8716D95540AAF1D24D62503DC0FED478E926D55540448B6CE7FBD940C0C36
4AA60
> 
54E6554039B4C876BE7F3FC06D567DAEB6EA55404260E5D0223B3BC00C022B8716D95540
"
> 
> As you can see the geometries from the KML are exploded ... the short WKB
> is a point, the longer one POLYGON geometry. However two database entries
> are created for each feature ("name"). I could just merge these but I feel
> that OGR can create two geometry columns each for POINT and POLYGON, in
> fact this seems to be explicitly stated that it can do this in the news
> release for GDAL version 2.

What you observe is the intended behaviour of -explodecollections, but not what 
you want.

> 
> Should I be using some kind of SQL statement on the OGR command line to
> make it do this ?

You can indeed use Spatialite SQL to do what you want if you specify the SQLite 
dialect 
(http://gdal.org/ogr_sql_sqlite.html). If you look at 
http://www.gaia-gis.it/gaia-sins/
spatialite-sql-4.3.0.html, you can see there's a ST_CollectionExtract function 
as well.

So something like

ogr2ogr -overwrite PG:x your.kml -dialect SQLite -sql "SELECT 
ST_CollectionExtract(geometry, 1) AS the_point, ST_COLLECTIONExtract(geometry, 
3) AS 
the_polygon, * FROM the_layer"

Even

> Michael Z Freeman
> 
> On Mon, November 21, 2016 7:30 am, jratike80 wrote:
> > Hi,
> > 
> > 
> > The main problem is that KML support such data models which do not suit
> > well with the simple feature model of GDAL. You can get some information
> > about that by reading http://www.gdal.org/drv_libkml.html.
> > 
> > In your case each feature has two geometries: one point and one polygon.
> > Best that can be done with the simple feature model it to push them both
> > into one geometry which in that case must be a geometry collection. Here
> > is one example from your data
> > 
> > GEOMETRYCOLLECTION (POINT (-1.282 87.1064),POLYGON ((3.02457
> > 87.0792,2.22294
> > 86.9806,-5.655 87.1168,-5.098 87.2201,3.02457 87.0792)))
> > 
> > 
> > You can save geometry collections into PostGIS if you use ogr2ogr with
> > switch "-nlt geometry". However, saving geometry collections into PostGIS
> > may not solve your problem because only few GIS programs can handle
> > geometry collections well.
> > 
> > A better option for you might be to explode the collections and save
> > points and polygons as separate features into PostGIS. The ogr2ogr command
> > to use is like
> > 
> > ogr2ogr -f PGDump output_pgdump -nlt geometry -explodecollections
> > Beyer_Hirise_Areas_KML/MC-01-0-0-0.kml
> > 
> > 
> > If you want to have a quick look at your data, use JML as outputformat
> > and open the result with OpenJUMP. It shows both the geometry collections
> > and exploded collections fine for me with names and descriptions.
> > 
> > -Jukka Rahkonen-
> > 
> > 
> > 
> > 
> > --
> > View this message in context:
> > http://osgeo-org.1560.x6.nabble.com/gdal-dev-Unable-to-extract-features-f
> > rom-KML-tp5296670p5296708.html Sent from the GDAL - Dev mailing list
> > archive at Nabble.com. ___
> > gdal-dev mailing list gdal-dev@lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> 
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Unable to extract features from KML

[Michael Z Freeman]

Hi,

Still having trouble with this after updating to ogr2ogr/gdal 2.1.1.

Some of it is undoubtedly my unfamiliarity with GDAL. I'm on the steep
part of the learning curve ! But fun.

"ogr2ogr -f PostgreSQL PG:"dbname=original_hirise" -a_srs "EPSG:4326" -lco
OVERWRITE=YES -explodecollections -nlt geometry
Beyer_Hirise_Areas_KML/MC-01-0-0-0.kml"

... creates (an edited CSV exported from a PGAdmin4 query) ...

"ogc_fid","name","description","wkb_geometry"
1,"ESP_018108_2680","[...]","010120E6100AD7A3703D6A3EC0744694F606035640"
2,"ESP_018108_2680","[...]","010320E61001000500A4703D0AD7233CC079E9263108FC554000C03FC0F4FDD478E9FA5540F0A7C64B377940C0992A1895D4095640FCA9F1D24DE23CC002BC0512140B5640A4703D0AD7233CC079E9263108FC5540"
3,"ESP_018318_2675","[...]","010120E610A245B6F3FD543EC0728A8EE4F2DF5540"
4,"ESP_018318_2675","[...]","010320E610010005004260E5D0223B3BC00C022B8716D95540AAF1D24D62503DC0FED478E926D55540448B6CE7FBD940C0C364AA6054E6554039B4C876BE7F3FC06D567DAEB6EA55404260E5D0223B3BC00C022B8716D95540"

As you can see the geometries from the KML are exploded ... the short WKB
is a point, the longer one POLYGON geometry. However two database entries
are created for each feature ("name"). I could just merge these but I feel
that OGR can create two geometry columns each for POINT and POLYGON, in
fact this seems to be explicitly stated that it can do this in the news
release for GDAL version 2.

Should I be using some kind of SQL statement on the OGR command line to
make it do this ?

Michael Z Freeman

On Mon, November 21, 2016 7:30 am, jratike80 wrote:
> Hi,
>
>
> The main problem is that KML support such data models which do not suit
> well with the simple feature model of GDAL. You can get some information
> about that by reading http://www.gdal.org/drv_libkml.html.
>
> In your case each feature has two geometries: one point and one polygon.
> Best that can be done with the simple feature model it to push them both
> into one geometry which in that case must be a geometry collection. Here
> is one example from your data
>
> GEOMETRYCOLLECTION (POINT (-1.282 87.1064),POLYGON ((3.02457
> 87.0792,2.22294
> 86.9806,-5.655 87.1168,-5.098 87.2201,3.02457 87.0792)))
>
>
> You can save geometry collections into PostGIS if you use ogr2ogr with
> switch "-nlt geometry". However, saving geometry collections into PostGIS
> may not solve your problem because only few GIS programs can handle
> geometry collections well.
>
> A better option for you might be to explode the collections and save
> points and polygons as separate features into PostGIS. The ogr2ogr command
> to use is like
>
> ogr2ogr -f PGDump output_pgdump -nlt geometry -explodecollections
> Beyer_Hirise_Areas_KML/MC-01-0-0-0.kml
>
>
> If you want to have a quick look at your data, use JML as outputformat
> and open the result with OpenJUMP. It shows both the geometry collections
> and exploded collections fine for me with names and descriptions.
>
> -Jukka Rahkonen-
>
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/gdal-dev-Unable-to-extract-features-f
> rom-KML-tp5296670p5296708.html Sent from the GDAL - Dev mailing list
> archive at Nabble.com. ___
> gdal-dev mailing list gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Troubleshooting palette management. Potential bug in palette for overviews?

[Daniele Romagnoli]

On Thu, Nov 24, 2016 at 5:01 PM, Even Rouault 
wrote:

> On jeudi 24 novembre 2016 16:51:59 CET Daniele Romagnoli wrote:
>
> > Doh!
>
> > Thanks for the feedback, Even.
>


> (Indeed it seemed strange to me that nobody had ever noticed before)
>
Sorry, what I meant with that was that I didn't see that ticket before
sending my email so it seemed strange to me that nobody noticed the palette
math weirdness :)


>
>
> Not that much regarding software using GDAL since if you don't explicitly
> request the overview bands/datasets but rather use the subsampling
> RasterIO() calls, you'll never work directly with the color table of the
> overviews but still use the one of the main dataset (or even if you request
> the overviews directly I guess in most cases applications don't bother
> checking if the color table of the overview).
>

As far as I can see, the TIFFImageReader parses the metadata information
based on imageIndex (so overview number). I will double check if things
were working differently in the colormap topic before our
changes/optimizations to this class.


> In GDAL, overviews conceptually share the same palette than the main
> image. Otherwise subsampling RasterIO() will return values of the overview
> palette that will be interpreted with the palette of the main image. You
> can imagine the fun visual effects if the palette are not the same... So
> this is assumed, but not checked.
>
>
>
> Could be worth fixing that if that cause issues (apparently in your use
> case at least), but I'd be interested by information of what is the
> standard/recommanded/best-practice-in-the-industry/etc... formulas to use
> for the mapping between the [0,255] and [0,65535] ranges.
>
>
>

Cheers,
Daniele

> >
>
> > Best Regards,
>
> > Daniele
>
> >
>
> > On Thu, Nov 24, 2016 at 1:35 PM, Even Rouault <
> even.roua...@spatialys.com>
>
> >
>
> > wrote:
>
> > > Daniele,
>
> > >
>
> > >
>
> > >
>
> > > This is indeed an inconsistency I spotted a long time ago :
>
> > >
>
> > > https://trac.osgeo.org/gdal/ticket/2213
>
> > >
>
> > >
>
> > >
>
> > > Nobody was apparently sufficiently annoyed by this to fix it. Actually
> I
>
> > > see I proposed a patch. Was probably waiting for feedback. I'm not
> sure if
>
> > > there's a standardized way of converting between the [0,255] and
> [0,65535]
>
> > > ranges.
>
> > >
>
> > >
>
> > >
>
> > > Even
>
> > >
>
> > > > Hi List,
>
> > > >
>
> > > > I'm investigating on a issue I have with a paletted image with
>
> > > > overviews,
>
> > > >
>
> > > > using java ImageIO tiff reader (wait... I know on this list we
> discuss
>
> > > >
>
> > > > about GDAL topics :) ).
>
> > > >
>
> > > > I'm trying to better understand palette interpretation and palette
>
> > >
>
> > > storing
>
> > >
>
> > > > on tags and how they are handled by GDAL.
>
> > > >
>
> > > >
>
> > > >
>
> > > > From TIFF specifications, colormap entries are stored as short
> values.
>
> > > >
>
> > > > I have a sample file with this color tab (as reported by gdalinfo).
>
> > > >
>
> > > >
>
> > > >
>
> > > > Color Table (RGB with 256 entries)
>
> > > >
>
> > > > 0: *164*,164,164,255
>
> > > >
>
> > > > 1: *255*,255,255,255
>
> > > >
>
> > > > 2: 0,0,0,255
>
> > > >
>
> > > > 3: 0,0,0,255
>
> > > >
>
> > > >  repeat 0,0,0,255 to the end.
>
> > > >
>
> > > >
>
> > > >
>
> > > > Using an Hex editor or AsTiffTagViewer, I can see the byte values of
> the
>
> > > >
>
> > > > colormap as stored on the file.
>
> > > >
>
> > > > For the first 4 mappings of the colortable the Red components are:
>
> > > >
>
> > > > *A4 A4* *FF FF* 00 00 00 00
>
> > > >
>
> > > > (8 bytes = 4 shorts)
>
> > > >
>
> > > >
>
> > > >
>
> > > > As far as I have understood, this is related to that part of the
>
> > > >
>
> > > > specification (quoting it, section 5):
>
> > > >
>
> > > >
>
> > > >
>
> > > >
>
> > > >
>
> > > > *In a TIFF ColorMap, all the Red values come first, followed by the
>
> > > > Green
>
> > > >
>
> > > > values, then the Blue values. In the ColorMap, black is represented
> by
>
> > > >
>
> > > > 0,0,0 and white is represented by 65535, 65535, 65535*
>
> > > >
>
> > > >
>
> > > >
>
> > > > So the second entry of the map, being white (255, 255, 255) has its
> red
>
> > > >
>
> > > > value stored as 65535 = *FF FF *whilst the first color (164,164,164)
> has
>
> > > >
>
> > > > its red value stored as 42148 = *A4 A4*
>
> > > >
>
> > > >
>
> > > >
>
> > > > I see this behavior is confirmed by the way the geotiff.cpp create
> the
>
> > > >
>
> > > > colortable:
>
> > > >
>
> > > > https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/
>
> > >
>
> > > geotiff.cpp#L5043
>
> > >
>
> > > > panTRed[iColor] = static_cast(257 * sRGB.c1);
>
> > > >
>
> > > > panTGreen[iColor] = static_cast(257 * sRGB.c2);
>
> > > >
>
> > > > panTBlue[iColor] = static_cast(257 * sRGB.c3)
>
> > > >
>
> > > >
>
> > > >
>
> > > > Applying a color*257 multiplication allows to have its 8 bits color
>
> > > >
>
> > > > representation 

Re: [gdal-dev] Troubleshooting palette management. Potential bug in palette for overviews?

[Even Rouault]

On jeudi 24 novembre 2016 16:51:59 CET Daniele Romagnoli wrote:
> Doh!
> Thanks for the feedback, Even. (Indeed it seemed strange to me that nobody
> had ever noticed before)

Not that much regarding software using GDAL since if you don't explicitly 
request the 
overview bands/datasets but rather use the subsampling RasterIO() calls, you'll 
never work 
directly with the color table of the overviews but still use the one of the 
main dataset (or 
even if you request the overviews directly I guess in most cases applications 
don't bother 
checking if the color table of the overview). In GDAL, overviews conceptually 
share the same 
palette than the main image. Otherwise subsampling RasterIO() will return 
values of the 
overview palette that will be interpreted with the palette of the main image. 
You can imagine 
the fun visual effects if the palette are not the same... So this is assumed, 
but not checked. 

Could be worth fixing that if that cause issues (apparently in your use case at 
least), but I'd be 
interested by information of what is the 
standard/recommanded/best-practice-in-the-
industry/etc... formulas to use for the mapping between the [0,255] and 
[0,65535] ranges.

> 
> Best Regards,
> Daniele
> 
> On Thu, Nov 24, 2016 at 1:35 PM, Even Rouault 
> 
> wrote:
> > Daniele,
> > 
> > 
> > 
> > This is indeed an inconsistency I spotted a long time ago :
> > 
> > https://trac.osgeo.org/gdal/ticket/2213
> > 
> > 
> > 
> > Nobody was apparently sufficiently annoyed by this to fix it. Actually I
> > see I proposed a patch. Was probably waiting for feedback. I'm not sure if
> > there's a standardized way of converting between the [0,255] and [0,65535]
> > ranges.
> > 
> > 
> > 
> > Even
> > 
> > > Hi List,
> > > 
> > > I'm investigating on a issue I have with a paletted image with
> > > overviews,
> > > 
> > > using java ImageIO tiff reader (wait... I know on this list we discuss
> > > 
> > > about GDAL topics :) ).
> > > 
> > > I'm trying to better understand palette interpretation and palette
> > 
> > storing
> > 
> > > on tags and how they are handled by GDAL.
> > > 
> > > 
> > > 
> > > From TIFF specifications, colormap entries are stored as short values.
> > > 
> > > I have a sample file with this color tab (as reported by gdalinfo).
> > > 
> > > 
> > > 
> > > Color Table (RGB with 256 entries)
> > > 
> > > 0: *164*,164,164,255
> > > 
> > > 1: *255*,255,255,255
> > > 
> > > 2: 0,0,0,255
> > > 
> > > 3: 0,0,0,255
> > > 
> > >  repeat 0,0,0,255 to the end.
> > > 
> > > 
> > > 
> > > Using an Hex editor or AsTiffTagViewer, I can see the byte values of the
> > > 
> > > colormap as stored on the file.
> > > 
> > > For the first 4 mappings of the colortable the Red components are:
> > > 
> > > *A4 A4* *FF FF* 00 00 00 00
> > > 
> > > (8 bytes = 4 shorts)
> > > 
> > > 
> > > 
> > > As far as I have understood, this is related to that part of the
> > > 
> > > specification (quoting it, section 5):
> > > 
> > > 
> > > 
> > > 
> > > 
> > > *In a TIFF ColorMap, all the Red values come first, followed by the
> > > Green
> > > 
> > > values, then the Blue values. In the ColorMap, black is represented by
> > > 
> > > 0,0,0 and white is represented by 65535, 65535, 65535*
> > > 
> > > 
> > > 
> > > So the second entry of the map, being white (255, 255, 255) has its red
> > > 
> > > value stored as 65535 = *FF FF *whilst the first color (164,164,164) has
> > > 
> > > its red value stored as 42148 = *A4 A4*
> > > 
> > > 
> > > ___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Troubleshooting palette management. Potential bug in palette for overviews?

[Daniele Romagnoli]

Doh!
Thanks for the feedback, Even. (Indeed it seemed strange to me that nobody
had ever noticed before)

Best Regards,
Daniele

On Thu, Nov 24, 2016 at 1:35 PM, Even Rouault 
wrote:

> Daniele,
>
>
>
> This is indeed an inconsistency I spotted a long time ago :
>
> https://trac.osgeo.org/gdal/ticket/2213
>
>
>
> Nobody was apparently sufficiently annoyed by this to fix it. Actually I
> see I proposed a patch. Was probably waiting for feedback. I'm not sure if
> there's a standardized way of converting between the [0,255] and [0,65535]
> ranges.
>
>
>
> Even
>
>
>
> > Hi List,
>
> > I'm investigating on a issue I have with a paletted image with overviews,
>
> > using java ImageIO tiff reader (wait... I know on this list we discuss
>
> > about GDAL topics :) ).
>
> > I'm trying to better understand palette interpretation and palette
> storing
>
> > on tags and how they are handled by GDAL.
>
> >
>
> > From TIFF specifications, colormap entries are stored as short values.
>
> > I have a sample file with this color tab (as reported by gdalinfo).
>
> >
>
> > Color Table (RGB with 256 entries)
>
> > 0: *164*,164,164,255
>
> > 1: *255*,255,255,255
>
> > 2: 0,0,0,255
>
> > 3: 0,0,0,255
>
> >  repeat 0,0,0,255 to the end.
>
> >
>
> > Using an Hex editor or AsTiffTagViewer, I can see the byte values of the
>
> > colormap as stored on the file.
>
> > For the first 4 mappings of the colortable the Red components are:
>
> > *A4 A4* *FF FF* 00 00 00 00
>
> > (8 bytes = 4 shorts)
>
> >
>
> > As far as I have understood, this is related to that part of the
>
> > specification (quoting it, section 5):
>
> >
>
> >
>
> > *In a TIFF ColorMap, all the Red values come first, followed by the Green
>
> > values, then the Blue values. In the ColorMap, black is represented by
>
> > 0,0,0 and white is represented by 65535, 65535, 65535*
>
> >
>
> > So the second entry of the map, being white (255, 255, 255) has its red
>
> > value stored as 65535 = *FF FF *whilst the first color (164,164,164) has
>
> > its red value stored as 42148 = *A4 A4*
>
> >
>
> > I see this behavior is confirmed by the way the geotiff.cpp create the
>
> > colortable:
>
> > https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/
> geotiff.cpp#L5043
>
> >
>
> > panTRed[iColor] = static_cast(257 * sRGB.c1);
>
> > panTGreen[iColor] = static_cast(257 * sRGB.c2);
>
> > panTBlue[iColor] = static_cast(257 * sRGB.c3)
>
> >
>
> > Applying a color*257 multiplication allows to have its 8 bits color
>
> > representation stored into a short where intensity goes from 0 to 65535.
>
> >
>
> > The issue I have is that when using gdaladdo, the overviews are generated
>
> > with a different color table.
>
> > So that the first 8 bytes of the red component stored in the colormap,as
>
> > reported by AsTiffTagViewer are:
>
> > *A4* 00 *FF* 00 00 00 00 00
>
> >
>
> > When converting it back to a 0-255 color component with the formula
>
> > R*255/65535 it become *163* instead of 164.
>
> >
>
> > I see in the GDAL code that both the IBuildOverviews and
>
> > CreateOverviewsFromSrcOverviews call the CreateTIFFColorTable function
>
> > which multiplies the colorMapEntry by 256 instead of 257.
>
> >
>
> > https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/
> geotiff.cpp#L9137
>
> >
>
> > This may explain the *A4* 00 *FF* 00 entries in the overview with respect
>
> > to the *A4 A4 FF FF *entries in the main level.
>
> >
>
> > Do you have any feedback on this?
>
> > Please, let me know what do you think about it.
>
> >
>
> > Best Regards,
>
> > Daniele
>
>
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:  +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message 

Re: [gdal-dev] Troubleshooting palette management. Potential bug in palette for overviews?

[Even Rouault]

Daniele,

This is indeed an inconsistency I spotted a long time ago :
https://trac.osgeo.org/gdal/ticket/2213

Nobody was apparently sufficiently annoyed by this to fix it. Actually I see I 
proposed a patch. 
Was probably waiting for feedback. I'm not sure if there's a standardized way 
of converting 
between the [0,255] and [0,65535] ranges.

Even

> Hi List,
> I'm investigating on a issue I have with a paletted image with overviews,
> using java ImageIO tiff reader (wait... I know on this list we discuss
> about GDAL topics :) ).
> I'm trying to better understand palette interpretation and palette storing
> on tags and how they are handled by GDAL.
> 
> From TIFF specifications, colormap entries are stored as short values.
> I have a sample file with this color tab (as reported by gdalinfo).
> 
> Color Table (RGB with 256 entries)
>   0: *164*,164,164,255
>   1: *255*,255,255,255
>   2: 0,0,0,255
>   3: 0,0,0,255
>  repeat 0,0,0,255 to the end.
> 
> Using an Hex editor or AsTiffTagViewer, I can see the byte values of the
> colormap as stored on the file.
> For the first 4 mappings of the colortable the Red components are:
> *A4 A4* *FF FF* 00 00 00 00
> (8 bytes = 4 shorts)
> 
> As far as I have understood, this is related to that part of the
> specification (quoting it, section 5):
> 
> 
> *In a TIFF ColorMap, all the Red values come first, followed by the Green
> values, then the Blue values. In the ColorMap, black is represented by
> 0,0,0 and white is represented by 65535, 65535, 65535*
> 
> So the second entry of the map, being white (255, 255, 255) has its red
> value stored as 65535 = *FF FF *whilst the first color (164,164,164) has
> its red value stored as 42148 = *A4 A4*
> 
> I see this behavior is confirmed by the way the geotiff.cpp create the
> colortable:
> https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/geotiff.cpp#L5043
> 
> panTRed[iColor] = static_cast(257 * sRGB.c1);
> panTGreen[iColor] = static_cast(257 * sRGB.c2);
> panTBlue[iColor] = static_cast(257 * sRGB.c3)
> 
> Applying a color*257 multiplication allows to have its 8 bits color
> representation stored into a short where intensity goes from 0 to 65535.
> 
> The issue I have is that when using gdaladdo, the overviews are generated
> with a different color table.
> So that the first 8 bytes of the red component stored in the colormap,as
> reported by AsTiffTagViewer are:
> *A4* 00 *FF* 00 00 00 00 00
> 
> When converting it back to a 0-255 color component with the formula
> R*255/65535 it become *163* instead of 164.
> 
> I see in the GDAL code that both the IBuildOverviews and
> CreateOverviewsFromSrcOverviews call the CreateTIFFColorTable function
> which multiplies the colorMapEntry by 256 instead of 257.
> 
> https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/geotiff.cpp#L9137
> 
> This may explain the *A4* 00 *FF* 00 entries in the overview with respect
> to the *A4 A4 FF FF *entries in the main level.
> 
> Do you have any feedback on this?
> Please, let me know what do you think about it.
> 
> Best Regards,
> Daniele


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Troubleshooting palette management. Potential bug in palette for overviews?

[Daniele Romagnoli]

Hi List,
I'm investigating on a issue I have with a paletted image with overviews,
using java ImageIO tiff reader (wait... I know on this list we discuss
about GDAL topics :) ).
I'm trying to better understand palette interpretation and palette storing
on tags and how they are handled by GDAL.

>From TIFF specifications, colormap entries are stored as short values.
I have a sample file with this color tab (as reported by gdalinfo).

Color Table (RGB with 256 entries)
  0: *164*,164,164,255
  1: *255*,255,255,255
  2: 0,0,0,255
  3: 0,0,0,255
 repeat 0,0,0,255 to the end.

Using an Hex editor or AsTiffTagViewer, I can see the byte values of the
colormap as stored on the file.
For the first 4 mappings of the colortable the Red components are:
*A4 A4* *FF FF* 00 00 00 00
(8 bytes = 4 shorts)

As far as I have understood, this is related to that part of the
specification (quoting it, section 5):


*In a TIFF ColorMap, all the Red values come first, followed by the Green
values, then the Blue values. In the ColorMap, black is represented by
0,0,0 and white is represented by 65535, 65535, 65535*

So the second entry of the map, being white (255, 255, 255) has its red
value stored as 65535 = *FF FF *whilst the first color (164,164,164) has
its red value stored as 42148 = *A4 A4*

I see this behavior is confirmed by the way the geotiff.cpp create the
colortable:
https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/geotiff.cpp#L5043

panTRed[iColor] = static_cast(257 * sRGB.c1);
panTGreen[iColor] = static_cast(257 * sRGB.c2);
panTBlue[iColor] = static_cast(257 * sRGB.c3)

Applying a color*257 multiplication allows to have its 8 bits color
representation stored into a short where intensity goes from 0 to 65535.

The issue I have is that when using gdaladdo, the overviews are generated
with a different color table.
So that the first 8 bytes of the red component stored in the colormap,as
reported by AsTiffTagViewer are:
*A4* 00 *FF* 00 00 00 00 00

When converting it back to a 0-255 color component with the formula
R*255/65535 it become *163* instead of 164.

I see in the GDAL code that both the IBuildOverviews and
CreateOverviewsFromSrcOverviews call the CreateTIFFColorTable function
which multiplies the colorMapEntry by 256 instead of 257.

https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/gtiff/geotiff.cpp#L9137

This may explain the *A4* 00 *FF* 00 entries in the overview with respect
to the *A4 A4 FF FF *entries in the main level.

Do you have any feedback on this?
Please, let me know what do you think about it.

Best Regards,
Daniele










-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:  +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Mass addition of 'override' keyword ?

[Even Rouault]

Hi,

I've a branch where I've added the 'override' keyword to virtual methods that
are an override of a virtual method declared in a base class.

https://github.com/OSGeo/gdal/compare/trunk...rouault:add_override?expand=1

Of course, not completely manually, but at >90% with the help of clang-tidy.  
GCC >= 5.1 has a -Wsuggest-override warning that I've enabled
in configure to find all places missed by tidy-clang (at least with the drivers 
I've
enabled in that branch). And recent clang versions have a less powerful
-Winconsistent-missing-override (enabled with -Wall I think) that warns when
you have explicitly tagged an overriden method with 'override' but neglected
to do so for other overriden methods of the same class. So new code should
be override friendly.

As we still support C++03 compilers, the following tricks are used :

1) If the compiler doesn't advertize C++11 support, we #define override to 
empty in
cpl_port.h. But that only if -DGDAL_COMPILATION is defined (which is the case
in the Unix and Windows makefiles used to build GDAL), so as to avoid messing 
the
namespace of code that includes GDAL.

2) As we don't want 'override' to leak into installed headers to keep C++03
compat, then CPL_OVERRIDE is used in those files, instead of plain 'override'.
It is a #define to override is the compiler is C++11 enabled, or #define to
empty otherwise.

As words may poorly reflect the reality of the code, here's the code:
{{{
#ifdef __cplusplus

#if HAVE_CXX11 || _MSC_VER >= 1600

/** To be used in public headers only. For non-public headers or .cpp files,
 * use override directly. */
#  define CPL_OVERRIDE override

#else

/** To be used in public headers only. For non-public headers or .cpp files,
 * use override directly. */
#  define CPL_OVERRIDE

/* For GDAL source compilation only, ignore override if non C++11 compiler */
#ifdef GDAL_COMPILATION
#  define override
#endif

#endif /* __cpluscplus */

}}}

So in summary: use override in all .cpp files and non-installed .h headers.
Use CPL_OVERRIDE in installed .h headers. Will make C++11 compilers enforce
overriding, and C++03 ones ignore it.


Interestingly in the process, I found a rather odd bug which pre-existed
unnoticed. It is Windows specific. In VSIFilesystemHandler
we have a virtual method GetDiskFreeSpace(). But on Windows, if you include
, GetDiskFreeSpace is a #define to the ANSI Win32 function
GetDiskFreeSpaceA() ... So the VSIWin32FilesystemHandler::GetDiskFreeSpace()
method was actually evaluated as 
VSIWin32FilesystemHandler::GetDiskFreeSpaceA() by the preprocessor in
port/cpl_vsil_win32.cpp, and not as an override of the base
VSIFilesystemHandler::GetDiskFreeSpace()... Not critical since this is rarely
used, and code that use this method should be ready to receive the default -1
value for filesystems that don't implement it.

Thoughts regarding committing this?

Windows and Unix builds are happy:
* https://ci.appveyor.com/project/rouault/gdal2/build/1.0.38 (this one is
C++11. Well actually VS12 is partially C++11, but sufficiently to understand 
override)
* https://travis-ci.org/rouault/gdal2/builds/178478253 (mix of C++11 and C++03 
configs)

Even

~

PS. For reference, here's the method I've rougly used to convert
(with clang-tidy of clang 3.8)

Create in GDAL root directory a ".clang-tidy" file with the following content:

{
Checks: '-*,modernize-use-override'
}


Then create a Python script "apply.py" that will use the output of clang-tidy to
patch files with override. There is
some complication to avoid it to remove the "virtual" keyword while adding
the "override" one (and also avoiding tagging virtual destructors as override,
which doesn't bring anything interesting. And actually is disliked by VS 2010).
I think that for some analyzers like cppcheck, the virtual keyword
might help them better understand the logic when they don't understand the
override keyword (cppcheck 1.72 is in fact confused by the override keyword and
produces a lot of false positives, so in the cppcheck.sh script, override is
forced to be an empty #define).

{{{
import yaml
import sys

root = yaml.load( open(sys.argv[1], 'rb').read() )
replacements = root['Replacements']

files = {}

for rep in replacements:
f = rep['FilePath']
if not f in files:
files[f] = []
files[f].append(rep)

for f in files:
content = open(f).read()
shift = 0
for rep in files[f]:
# Do not apply replacements that will remove the 'virtual' keyword
if rep['Length'] == 0 and rep['ReplacementText'] == ' override':
insert_pos = rep['Offset'] + shift
# Adding override to a virtual destructor is quite ugly. Don't do it
is_destructor = False
for i in range(80):
if i > insert_pos:
break
if content[insert_pos-1-i] == ';':
break
if content[insert_pos-1-i] == '~':

Re: [gdal-dev] "Very" long path name support under windows

[ianinKL]

http://https://msdn.microsoft.com/en-us/library/windows/desktop/aa364963(v=vs.85).aspx

 
Microsoft has a very succinct summary of the issue and how to address it:



--
View this message in context: 
http://osgeo-org.1560.x6.nabble.com/Very-long-path-name-support-under-windows-tp5297074p5297231.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev