Re: [gdal-dev] Feasibility of expanding VRT schema to allow users to specify X/Y dimension for HDF data?
Hi, Even!
I'm making some progress in utilizing OpenOptions to modify HDF drivers.
For automated testing of the new HDF drivers, should I use
gdal.OpenEx() like below in gor2vrt.py?
src_ds = gdal.OpenEx( infile, gdal.OF_VECTOR, open_options = openoptions )
Or does gdal.Open() also support open_options argument?
--
HDF:Antifragile Solution for Elastic Bigdata Analytics
On Fri, Aug 5, 2016 at 2:59 AM, Even Rouault wrote:
> Frank,
>
> From what I understand from Joe's needs, it looks like simple transposing of a
> 2D raster wouldn't be enough. Here we would need to "transpose" pixels
> scattered through different subdatasets due to the Nd > 2 dimensionality of
> the
> original dataset, which would be impractical to express at the VRT level, and
> likely inefficient.
>
> Something I've thought about would be to make Nd > 2 rasters native objects at
> the GDAL level, but this would have likely deep implications on the code base
> and should be considered carefully, and would be of interest for a limited set
> of drivers (netCDF, HDF4, HDF5, Rasdaman).
>
> Even
>
>> Brian / Even,
>>
>> Certainly it is desirable for the HDF (and perhaps other super
>> flexible formats like netcdf) to support an open option to select
>> alternative axes. But the ability to transpose a dataset could also
>> be quite valuable in the VRT driver to "fix" any input transposed
>> dataset.
>>
>> I'm also not entirely certain why one couldn't supply an appopriately
>> transposed geotransform to accomplish something similar. This could
>> be done without any code changes in the existing VRT format.
>>
>> Best regards,
>> Frank
>>
>> On Thu, Aug 4, 2016 at 3:32 PM, Even Rouault
> wrote:
>> > On Thursday 04 August 2016 16:31:25 H. Joe Lee wrote:
>> >> Hi,
>> >>
>> >> My name is Joe Lee and I'm very interested in improving GDAL's
>> >>
>> >> capability to access NASA HDF4/HDF5 data so that users can work with
>> >> HDF easily through GDAL. For example, my goal is to allow users to
>> >> translate any HDF data into GeoTIFF via gdal_translate.
>> >>
>> >> I've worked with diverse NASA HDF products and provided solution for
>> >>
>> >> visualizing data correctly through Python/MATLAB/IDL/NCL [1] and I
>> >> know that many products, other than HDF-EOS, may not work well with
>> >> GDAL because HDF is flexible and NASA data producers do not
>> >> necessarily follow the conventions that GDAL uses.
>> >>
>> >> By default, GDAL HDF4/HDF5 driver uses the following convention for
>> >>
>> >> unknown products.
>> >>
>> >> For HDF4 (frmts/hdf4/hdf4imagedataset.cpp):
>> >>
>> >> // Search for the starting "X" and "Y" in the names or take
>> >> // the last two dimensions as X and Y sizes
>> >> iXDim = nDimCount - 1;
>> >> iYDim = nDimCount - 2;
>> >>
>> >> For HDF5 (frmts/hdf5/hdf5imagedataset.cpp):
>> >> int GetYIndex() const { return IsComplexCSKL1A() ? 0 : ndims - 2;
>> >> }
>> >> int GetXIndex() const { return IsComplexCSKL1A() ? 1 : ndims - 1;
>> >> }
>> >>
>> >> The above code works well as long as Unknown HDF product follows the
>> >>
>> >> above convention. However, in reality, HDF data can have an arbitrary
>> >>
>> >> order in terms of Band, X and Y dimension like this:
>> >> dset4D[XDim=360][YDim=180][Band1=2][Band2=3]
>> >> dimindex:0 12 3
>> >>
>> >> Since ndims=4, ndims-2 becomes 2 and ndims-1=3. In such case, GDAL
>> >>
>> >> generates 360x180 bands of 2x3 images, instead of the desired 2x3
>> >> bands of 360x180 images.
>> >>
>> >> Thus, I'm wondering if GDAL can expand VRT schema so that VRT allows
>> >>
>> >> users to specify the correct dimension index because specifying
>> >> dimension order for each different NASA product in [1] is
>> >> impractical. For example, I'd like suggest a new tag like
>> >>
>> >> "SourceDimension" like below:
>> >>
>> >>
>> >>
>> >> > >>
>> >> relativeToVRT="0">HDF4_SDS:UNKNOWN:"DATA_WITH_4D_DATASET.hdf":7> >> len ame>
>> >>
>> >> 1
>> >> > >>
>> >> DataType="UInt16" BlockXSize="360" BlockYSize="180" />
>> >>
>> >>...
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Once user specifies correct dimensions by editing VRT, it can be
>> >>
>> >> used by GDAL HDF4/HDF5 drivers and the HDF drivers read the data
>> >> correctly for GDAL's image buffer.
>> >>
>> >> Do you think it's right and feasible approach to solve wrong X/Y
>> >>
>> >> dimension order problem? Or do you have any other existing solution
>> >> that can mitigate this problem in GDAL? The GEE project team has
>> >> experimented the idea by creating another separate XML file [2] but I
>> >> think it's time to sync with GDAL development team and come up with
>> >> the most elegant solution. In my opinion, VRT looks best and I wish
>> >> GDAL development team can give me some feedback on this idea.
>> >
>> > Joe,
>> >
>> > I would rather suggest to add open options to the drivers and pass them
>> > with the exiti
Re: [gdal-dev] ERROR 6: No translation for Lambert_Conformal_Conic to PROJ.4 format is known.
Hi Even Even Rouault-2 wrote > Helmut, > >> when I do: testepsg of the ogr2ogr'ed shapefile prj file: >> >> testepsg Gesamtgewaessernetz_v11_Tirol_epsg31287.prj >> >> ERROR 6: No translation for Lambert_Conformal_Conic to PROJ.4 format is >> known. >> PROJ.4 rendering of [Gesamtgewaessernetz_v11_Tirol_epsg31287.prj] = >> >> is this normal? > > Yes. testepsg foo.prj expects a .prj file conforming to OGC WKT, whereas a > .prj > file generated by the Shapefile driver will follow ESRI WKT. > > testepsg (or gdalsrsinfo) > ESRI::Gesamtgewaessernetz_v11_Tirol_epsg31287.prj > will report > > PROJ.4 : '+proj=lcc +lat_1=49 +lat_2=46 +lat_0=47.5 > +lon_0=13.33 > +x_0=40 +y_0=40 +ellps=bessel +units=m +no_defs ' > > OGC WKT : > PROJCS["MGI_Austria_Lambert", > GEOGCS["GCS_MGI", > DATUM["Militar_Geographische_Institute", > SPHEROID["Bessel_1841",6377397.155,299.1528128]], > PRIMEM["Greenwich",0], > UNIT["Degree",0.017453292519943295]], > PROJECTION["Lambert_Conformal_Conic_2SP"], > PARAMETER["standard_parallel_1",49], > PARAMETER["standard_parallel_2",46], > PARAMETER["latitude_of_origin",47.5], > PARAMETER["central_meridian",13.33], > PARAMETER["false_easting",40], > PARAMETER["false_northing",40], > UNIT["Meter",1]] > > Even > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > ___ > gdal-dev mailing list > gdal-dev@.osgeo > http://lists.osgeo.org/mailman/listinfo/gdal-dev Thanks for the testepsg (or gdalsrsinfo) ESRI::... hint! - best regards Helmut -- View this message in context: http://osgeo-org.1560.x6.nabble.com/ERROR-6-No-translation-for-Lambert-Conformal-Conic-to-PROJ-4-format-is-known-tp5286399p5286405.html Sent from the GDAL - Dev mailing list archive at Nabble.com. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Writing descriptions to GeoTiff bands
Le vendredi 16 septembre 2016 22:57:13, Andrew Bell a écrit :
> Hi,
>
> My code for creating a Tiff raster looks something like this:
>
> int nBands = 5;
> dataset->Create(filename, width, height, nBands, ...);
>
> for (int i = 1; i <= nBands; ++i)
> {
> GDALRasterBand *band = dataset->GetRasterBand(i);
> band->SetDescription(someString);
> band->WriteBlock(someData);
> }
>
> It appears that only the description to band 1 is written (it's the only
> one reported by gdalinfo). A little debugging leads me to believe that
> what's happening is that WriteBlock() invokes Crystalize() ->
> WriteMetadata(), which takes care of setting the band description. But
> once Crystalize() is called, it sets a flag so as to be a NOOP in future
> calls. I'm not using streaming.
>
> I'm trying to understand if this behavior is by design, a limitation that I
> can't find in the documentation or a bug.
It's a limitation due to how libtiff works mostly and/or how we use it (but
mostly how libtiff works, and a bit how the TIFF format itself makes it hard).
Basically for GTiff, you need to do all operations that affect metadata, in a
broad meaning, ie georeferencing, description, offsets, color table, TIFF &
GDAL metadata, etc... before writing any imagery. If we allowed to change
metadata after crzystalization, this would require rewriting the whole set of
TIFF tags at the end of file each time their serialized form increase.
So rewrite your loop into 2: one to set all descriptions, and another one to
write blocks.
Other formats may have similar limitations, so it is generally safe to proceed
this way in general.
Even
--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] ERROR 6: No translation for Lambert_Conformal_Conic to PROJ.4 format is known.
Helmut, > when I do: testepsg of the ogr2ogr'ed shapefile prj file: > > testepsg Gesamtgewaessernetz_v11_Tirol_epsg31287.prj > > ERROR 6: No translation for Lambert_Conformal_Conic to PROJ.4 format is > known. > PROJ.4 rendering of [Gesamtgewaessernetz_v11_Tirol_epsg31287.prj] = > > is this normal? Yes. testepsg foo.prj expects a .prj file conforming to OGC WKT, whereas a .prj file generated by the Shapefile driver will follow ESRI WKT. testepsg (or gdalsrsinfo) ESRI::Gesamtgewaessernetz_v11_Tirol_epsg31287.prj will report PROJ.4 : '+proj=lcc +lat_1=49 +lat_2=46 +lat_0=47.5 +lon_0=13.33 +x_0=40 +y_0=40 +ellps=bessel +units=m +no_defs ' OGC WKT : PROJCS["MGI_Austria_Lambert", GEOGCS["GCS_MGI", DATUM["Militar_Geographische_Institute", SPHEROID["Bessel_1841",6377397.155,299.1528128]], PRIMEM["Greenwich",0], UNIT["Degree",0.017453292519943295]], PROJECTION["Lambert_Conformal_Conic_2SP"], PARAMETER["standard_parallel_1",49], PARAMETER["standard_parallel_2",46], PARAMETER["latitude_of_origin",47.5], PARAMETER["central_meridian",13.33], PARAMETER["false_easting",40], PARAMETER["false_northing",40], UNIT["Meter",1]] Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Writing descriptions to GeoTiff bands
Hi,
My code for creating a Tiff raster looks something like this:
int nBands = 5;
dataset->Create(filename, width, height, nBands, ...);
for (int i = 1; i <= nBands; ++i)
{
GDALRasterBand *band = dataset->GetRasterBand(i);
band->SetDescription(someString);
band->WriteBlock(someData);
}
It appears that only the description to band 1 is written (it's the only
one reported by gdalinfo). A little debugging leads me to believe that
what's happening is that WriteBlock() invokes Crystalize() ->
WriteMetadata(), which takes care of setting the band description. But
once Crystalize() is called, it sets a flag so as to be a NOOP in future
calls. I'm not using streaming.
I'm trying to understand if this behavior is by design, a limitation that I
can't find in the documentation or a bug.
Any pointers appreciated.
--
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] ERROR 6: No translation for Lambert_Conformal_Conic to PROJ.4 format is known.
hi, I have vector dataset [1] in EPSG:31254. then I do ogr2ogr -s_srs EPSG:31254 -t_srs EPSG:31287 if I do testepsg EPSG:31287, I get following +proj4 string: PROJ.4 rendering of [EPSG:31287] = +proj=lcc +lat_1=49 +lat_2=46 +lat_0=47.5 +lon_0=13.33 +x_0=40 +y_0=40 +ellps=bessel +towgs84=577.326,90.129,463.919,5.137,1.474,5.297,2.4232 +units=m +no_defs when I do: testepsg of the ogr2ogr'ed shapefile prj file: testepsg Gesamtgewaessernetz_v11_Tirol_epsg31287.prj ERROR 6: No translation for Lambert_Conformal_Conic to PROJ.4 format is known. PROJ.4 rendering of [Gesamtgewaessernetz_v11_Tirol_epsg31287.prj] = is this normal? any idea? [1] https://www.tirol.gv.at/data/datenkatalog/umwelt/gewaessernetz/ - best regards Helmut -- View this message in context: http://osgeo-org.1560.x6.nabble.com/ERROR-6-No-translation-for-Lambert-Conformal-Conic-to-PROJ-4-format-is-known-tp5286399.html Sent from the GDAL - Dev mailing list archive at Nabble.com. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Feasibility of expanding VRT schema to allow users to specify X/Y dimension for HDF data?
Joe, > I'm making some progress in utilizing OpenOptions to modify HDF drivers. > > For automated testing of the new HDF drivers, should I use > gdal.OpenEx() like below in gor2vrt.py? > > src_ds = gdal.OpenEx( infile, gdal.OF_VECTOR, open_options = openoptions ) That's fine. Except you'd rather specify gdal.OF_RASTER for HDF. > > Or does gdal.Open() also support open_options argument? gdal.Open() has been kept to its historical API. Use gdal.OpenEx() for specifying open options. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] SQLite driver?
Yes, it had a small image attached. But never mind, I ended up finding that the crash resulted from the fact that some other gdal.dll without sqlite driver managed to get in the way. The bad new was that, once this solved, I got another crash latter but there is nothing in it that I can relate to gdal, so issue is closed here. Thanks to people that tried to help. Joaquim On 2016-09-14 6:55 PM, Joaquim Luis wrote: FWIW, my reply with an attached image is waiting for approval. Joaquim, I checked in the mailman backend and I don't see a message being held. But for sure always always when using mailing lists never try attaching things, for example if you need to include an image just use an external service like http://pasteboard.co/ and include the url to that image, in your response. -jeff ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] SQLite driver?
On 2016-09-14 6:55 PM, Joaquim Luis wrote: FWIW, my reply with an attached image is waiting for approval. Joaquim, I checked in the mailman backend and I don't see a message being held. But for sure always always when using mailing lists never try attaching things, for example if you need to include an image just use an external service like http://pasteboard.co/ and include the url to that image, in your response. -jeff -- Jeff McKenna MapServer Consulting and Training Services http://www.gatewaygeomatics.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
