Re: [gdal-dev] Writing descriptions to GeoTiff bands

[Even Rouault]

Le mardi 20 septembre 2016 17:13:44, Andrew Bell a écrit :
> On Sun, Sep 18, 2016 at 5:49 AM, Even Rouault 
> 
> wrote:
> > Le dimanche 18 septembre 2016 11:02:20, Sean Gillies a écrit :
> > > Hi Andrew, Even,
> > > 
> > > I am so grateful you asked this question, Andrew.
> > 
> > UPDATE: Andrew, after digging, I believe you have hit
> > https://trac.osgeo.org/gdal/ticket/6592 whose fix hasn't yet reached any
> > released version.
> 
> Thanks Even,
> 
> It seems like doing something else to set the metadata change flag will
> make things work.  Unfortunately, the change flag seems to be guarded so
> one has to make an actual change to get it to be set.  Something like the
> following works, but is a little crufty and deserves a comment.  Perhaps
> there's a better way?
> 
> int nBands = 5;
> dataset->Create(filename, width, height, nBands, ...);
> 
> for (int i = 1; i <= nBands; ++i)
> {
> GDALRasterBand *band = dataset->GetRasterBand(i);
> band->SetDescription(someString);
> 
> // Force metadata change flag so that description gets written.
> Current bug
> // (ticket 6592) requires something like this.
> band->SetOffset(band->GetOffset(NULL) + .1);
> band->SetOffset(band->GetOffset(NULL) - .1);

Or  just
band->SetOffset(1)
band->SetOffset(0)

since 0 is the default offset (and if you don't customize it).

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Writing descriptions to GeoTiff bands

[Andrew Bell]

On Sun, Sep 18, 2016 at 5:49 AM, Even Rouault 
wrote:

> Le dimanche 18 septembre 2016 11:02:20, Sean Gillies a écrit :
> > Hi Andrew, Even,
> >
> > I am so grateful you asked this question, Andrew.
>
> UPDATE: Andrew, after digging, I believe you have hit
> https://trac.osgeo.org/gdal/ticket/6592 whose fix hasn't yet reached any
> released version.
>

Thanks Even,

It seems like doing something else to set the metadata change flag will
make things work.  Unfortunately, the change flag seems to be guarded so
one has to make an actual change to get it to be set.  Something like the
following works, but is a little crufty and deserves a comment.  Perhaps
there's a better way?

int nBands = 5;
dataset->Create(filename, width, height, nBands, ...);

for (int i = 1; i <= nBands; ++i)
{
GDALRasterBand *band = dataset->GetRasterBand(i);
band->SetDescription(someString);

// Force metadata change flag so that description gets written.
Current bug
// (ticket 6592) requires something like this.
band->SetOffset(band->GetOffset(NULL) + .1);
band->SetOffset(band->GetOffset(NULL) - .1);

band->WriteBlock(someData);
}

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Writing descriptions to GeoTiff bands

[Even Rouault]

Le dimanche 18 septembre 2016 11:02:20, Sean Gillies a écrit :
> Hi Andrew, Even,
> 
> On Sat, Sep 17, 2016 at 9:52 PM, Even Rouault 
> 
> wrote:
> > Le vendredi 16 septembre 2016 23:11:07, Even Rouault a écrit :
> > > Le vendredi 16 septembre 2016 22:57:13, Andrew Bell a écrit :
> > > > Hi,
> > > > 
> > > > My code for creating a Tiff raster looks something like this:
> > > > 
> > > > int nBands = 5;
> > > > dataset->Create(filename, width, height, nBands, ...);
> > > > 
> > > > for (int i = 1; i <= nBands; ++i)
> > > > {
> > > > 
> > > > GDALRasterBand *band = dataset->GetRasterBand(i);
> > > > band->SetDescription(someString);
> > > > band->WriteBlock(someData);
> > > > 
> > > > }
> > > > 
> > > > It appears that only the description to band 1 is written (it's the
> > 
> > only
> > 
> > > > one reported by gdalinfo).  A little debugging leads me to believe
> > > > that what's happening is that WriteBlock() invokes Crystalize() ->
> > > > WriteMetadata(), which takes care of setting the band description. 
> > > > But once Crystalize() is called, it sets a flag so as to be a NOOP
> > > > in
> > 
> > future
> > 
> > > > calls.  I'm not using streaming.
> > > > 
> > > > I'm trying to understand if this behavior is by design, a limitation
> > 
> > that
> > 
> > > > I can't find in the documentation or a bug.
> > > 
> > > It's a limitation due to how libtiff works mostly and/or how we use it
> > 
> > (but
> > 
> > > mostly how libtiff works, and a bit how the TIFF format itself makes it
> > > hard). Basically for GTiff, you need to do all operations that affect
> > > metadata, in a broad meaning, ie georeferencing, description, offsets,
> > > color table, TIFF & GDAL metadata, etc... before writing any imagery. 
> > > If we allowed to change metadata after crzystalization, this would
> > > require rewriting the whole set of TIFF tags at the end of file each
> > > time their serialized form increase.
> > > 
> > > So rewrite your loop into 2: one to set all descriptions, and another
> > > one to write blocks.
> > > 
> > > Other formats may have similar limitations, so it is generally safe to
> > > proceed this way in general.
> > 
> > Actually the above is partly true & wrong. It is indeed discouraged to
> > change
> > metadata after having started writting imagery, but in the case of the
> > band description, you can still do it. As I said this will cause the
> > TIFF directory
> > to be rewritten, so a bit of storage loss, but nevertheless the
> > descriptions
> > are then correctly retrieved.
> > 
> > I used the following Python test script
> > 
> > {{{
> > from osgeo import gdal
> > gdal.SetCacheMax(0) # to force Fill() to commit to file immediatly
> > ds = gdal.GetDriverByName('GTiff').Create('test.tif', 1000, 1000, 5)
> > 
> > for i in range(5):
> > ds.GetRasterBand(i+1).SetDescription('foo%d' % i)
> > ds.GetRasterBand(i+1).Fill(100)
> > 
> > }}}
> > 
> > Works on latest state of trunk , 2.1 and 2.0 branches
> 
> I am so grateful you asked this question, Andrew.

UPDATE: Andrew, after digging, I believe you have hit 
https://trac.osgeo.org/gdal/ticket/6592 whose fix hasn't yet reached any 
released version.

> 
> Even, two follow up questions, one concrete, one more abstract. Is
> "crystalized" a state of all raster datasets, no matter the driver,

No. First it is only relevant to drivers that have Create capabilities (I mean 
contrary to the ones that support CreateCopy() only). And  some of them do not 
have restrictions. For example, in the case of drivers that use a dedicated 
file for imagery and a (text) header for metadata, you can use the API without 
any particular constraints since all the metadata updates are stored in the 
state variables of the dataset and flushed to to the header file at dataset 
closing.

In the case of TIFF, you can (bugs aside) update metadata after imagery, with 
the inconvenient of the TIFF tags being rewritten at the end of the file. In 
fact TIFF cryztalisation could (probably?) be avoided if we were OK for the 
TIFF tags to be always written at the end of file, which would make the file 
not 
compatible of streaming for readers (the seek to end of file could be 
particularly costly if you do a gdalinfo on a huge TIFF in a zip). This 
concept of cryztalisation was added long time ago ( 
https://trac.osgeo.org/gdal/changeset/1977 ) and apparently, it was to avoid 
an early crystalization which occured at dataset creation, so you ended up 
always with duplicated tags.

With Erdas Imagine from what I can see you can update at any time and the 
metadata is written only once at the end of file at dataset closing.

In other drivers like MBTiles/GPKG, you cannot modify the geotransform once 
you have set it (would require to shift imagery within tiles and update tile 
coordinates. and for that reason, you do have to call it before being allowed 
to write any imagery), but you can still set user metadata at any time, so it 
is 

Re: [gdal-dev] Writing descriptions to GeoTiff bands

[Sean Gillies]

Hi Andrew, Even,

On Sat, Sep 17, 2016 at 9:52 PM, Even Rouault 
wrote:

> Le vendredi 16 septembre 2016 23:11:07, Even Rouault a écrit :
> > Le vendredi 16 septembre 2016 22:57:13, Andrew Bell a écrit :
> > > Hi,
> > >
> > > My code for creating a Tiff raster looks something like this:
> > >
> > > int nBands = 5;
> > > dataset->Create(filename, width, height, nBands, ...);
> > >
> > > for (int i = 1; i <= nBands; ++i)
> > > {
> > >
> > > GDALRasterBand *band = dataset->GetRasterBand(i);
> > > band->SetDescription(someString);
> > > band->WriteBlock(someData);
> > >
> > > }
> > >
> > > It appears that only the description to band 1 is written (it's the
> only
> > > one reported by gdalinfo).  A little debugging leads me to believe that
> > > what's happening is that WriteBlock() invokes Crystalize() ->
> > > WriteMetadata(), which takes care of setting the band description.  But
> > > once Crystalize() is called, it sets a flag so as to be a NOOP in
> future
> > > calls.  I'm not using streaming.
> > >
> > > I'm trying to understand if this behavior is by design, a limitation
> that
> > > I can't find in the documentation or a bug.
> >
> > It's a limitation due to how libtiff works mostly and/or how we use it
> (but
> > mostly how libtiff works, and a bit how the TIFF format itself makes it
> > hard). Basically for GTiff, you need to do all operations that affect
> > metadata, in a broad meaning, ie georeferencing, description, offsets,
> > color table, TIFF & GDAL metadata, etc... before writing any imagery.  If
> > we allowed to change metadata after crzystalization, this would require
> > rewriting the whole set of TIFF tags at the end of file each time their
> > serialized form increase.
> >
> > So rewrite your loop into 2: one to set all descriptions, and another one
> > to write blocks.
> >
> > Other formats may have similar limitations, so it is generally safe to
> > proceed this way in general.
>
> Actually the above is partly true & wrong. It is indeed discouraged to
> change
> metadata after having started writting imagery, but in the case of the band
> description, you can still do it. As I said this will cause the TIFF
> directory
> to be rewritten, so a bit of storage loss, but nevertheless the
> descriptions
> are then correctly retrieved.
>
> I used the following Python test script
>
> {{{
> from osgeo import gdal
> gdal.SetCacheMax(0) # to force Fill() to commit to file immediatly
> ds = gdal.GetDriverByName('GTiff').Create('test.tif', 1000, 1000, 5)
> for i in range(5):
> ds.GetRasterBand(i+1).SetDescription('foo%d' % i)
> ds.GetRasterBand(i+1).Fill(100)
> }}}
>
> Works on latest state of trunk , 2.1 and 2.0 branches
>

I am so grateful you asked this question, Andrew.

Even, two follow up questions, one concrete, one more abstract. Is
"crystalized" a state of all raster datasets, no matter the driver, and is
there a method of determining whether the dataset is crystalized? Could
this situation be made less complicated or be made more safe for developers
by splitting the existing update access mode into update-metadata and
update-imagery access modes?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Writing descriptions to GeoTiff bands

[Even Rouault]

Le vendredi 16 septembre 2016 22:57:13, Andrew Bell a écrit :
> Hi,
> 
> My code for creating a Tiff raster looks something like this:
> 
> int nBands = 5;
> dataset->Create(filename, width, height, nBands, ...);
> 
> for (int i = 1; i <= nBands; ++i)
> {
> GDALRasterBand *band = dataset->GetRasterBand(i);
> band->SetDescription(someString);
> band->WriteBlock(someData);
> }
> 
> It appears that only the description to band 1 is written (it's the only
> one reported by gdalinfo).  A little debugging leads me to believe that
> what's happening is that WriteBlock() invokes Crystalize() ->
> WriteMetadata(), which takes care of setting the band description.  But
> once Crystalize() is called, it sets a flag so as to be a NOOP in future
> calls.  I'm not using streaming.
> 
> I'm trying to understand if this behavior is by design, a limitation that I
> can't find in the documentation or a bug.

It's a limitation due to how libtiff works mostly and/or how we use it (but 
mostly how libtiff works, and a bit how the TIFF format itself makes it hard). 
Basically for GTiff, you need to do all operations that affect metadata, in a 
broad meaning, ie georeferencing, description, offsets, color table, TIFF & 
GDAL metadata, etc... before writing any imagery.  If we allowed to change 
metadata after crzystalization, this would require rewriting the whole set of 
TIFF tags at the end of file each time their serialized form increase.

So rewrite your loop into 2: one to set all descriptions, and another one to 
write blocks.

Other formats may have similar limitations, so it is generally safe to proceed 
this way in general.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Writing descriptions to GeoTiff bands

[Andrew Bell]

Hi,

My code for creating a Tiff raster looks something like this:

int nBands = 5;
dataset->Create(filename, width, height, nBands, ...);

for (int i = 1; i <= nBands; ++i)
{
GDALRasterBand *band = dataset->GetRasterBand(i);
band->SetDescription(someString);
band->WriteBlock(someData);
}

It appears that only the description to band 1 is written (it's the only
one reported by gdalinfo).  A little debugging leads me to believe that
what's happening is that WriteBlock() invokes Crystalize() ->
WriteMetadata(), which takes care of setting the band description.  But
once Crystalize() is called, it sets a flag so as to be a NOOP in future
calls.  I'm not using streaming.

I'm trying to understand if this behavior is by design, a limitation that I
can't find in the documentation or a bug.

Any pointers appreciated.

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev