Re: [gdal-dev] GDAL 3.9.0beta1 available for testing

[Sean Gillies via gdal-dev]

Thanks for the fix and suggestions! I'm looking forward to 3.9.0.

On Mon, Apr 22, 2024 at 11:27 AM Even Rouault 
wrote:

> Sean,
>
> Rasterio's test suite has 4 errors with GDAL 3.9.0beta1.
>
> Metadata output of gdalinfo has changed.
>
> ok, I've run locally rasterio tests and I see gdalinfo now outputs:
> Metadata:
>   a=1
>   b=2
>   AREA_OR_POINT=Area
>
> whereas the test expects
>
> Metadata:
>   a=1
>   AREA_OR_POINT=Area
>   b=2
>
> Order of metadata items shouldn't be relied upon. I believe we have
> changed something internally (likely changing from char** to CPLStringList
> that does sorting) that might indeed have affected it. You could also
> potentially use "gdalinfo -json" output to get something easier to parse.
> But don't you have a rasterio API to get GDAL metadata that would avoid
> using gdalinfo at all?
>
> As soon as there are docker images I'll look more closely at this.
>
> ghcr.io/osgeo/gdal:ubuntu-full-3.9.0beta1 is already available
>
>
> Writing a mask to a Rasterio RGB.byte.tif dataset no longer creates a 
> RGB.byte.tif.msk
> file:
> https://github.com/rasterio/rasterio/actions/runs/8786444142/job/24109342188#step:8:2231
> .
>
> Yes, this is intended. Due to this change:
> GTiff driver:
>  * change default value of GDAL_TIFF_INTERNAL_MASK config option to YES
>
> You might run the test under GDAL_TIFF_INTERNAL_MASK=NO if you want to
> test external mask.
>
>
> The text of an error message which was "tests/data/corrupt.tif, band 1:
> IReadBlock failed at X offset 1, Y offset 1: TIFFReadEncodedTile() failed."
> in 3.8 has changed to "corrupt.tif, band 1: IReadBlock failed at X offset
> 1, Y offset 1: TIFFReadEncodedTile() failed." This is not a big deal, we
> can adjust Rasterio's test.
>
> Yes, related to
> https://github.com/OSGeo/gdal/commit/cb5161d907e8b4fa5cb68aaf891c1a258d6475b0
>
>
> Behavior of SRC_METHOD/DST_METHOD=NO_GEOTRANSFORM for warp options seems
> to have changed:
> https://github.com/rasterio/rasterio/actions/runs/8786444142/job/24109342188#step:8:2287.
> I'm not entirely sure what the intent is in Rasterio's test, will dig into
> that. I just wanted to say that I see a change in GDAL.
>
> ok, this is related to https://github.com/OSGeo/gdal/pull/9336 , which
> indeed wasn't honoring the SRC_METHOD=NO_GEOTRANSFORM and was using
> directly the geotransform from the source dataset.
>
> Fix queued in https://github.com/OSGeo/gdal/pull/9724
>
> (that said, tests reprojecting from 4326 to 3857 while ignoring the
> geotransform are a bit weird)
>
> Even
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.9.0beta1 available for testing

[Sean Gillies via gdal-dev]

Thanks, Jukka!

On Mon, Apr 22, 2024 at 10:48 AM Rahkonen Jukka <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi,
>
>
>
> The mask thing may happen due to https://github.com/OSGeo/gdal/pull/9604.
>
>
>
> -Jukka Rahkonen-
>
>
>
> *Lähettäjä:* gdal-dev  *Puolesta *Sean
> Gillies via gdal-dev
> *Lähetetty:* maanantai 22. huhtikuuta 2024 19.15
> *Vastaanottaja:* Even Rouault 
> *Kopio:* gdal-dev@lists.osgeo.org
> *Aihe:* Re: [gdal-dev] GDAL 3.9.0beta1 available for testing
>
>
>
> Hi Even,
>
>
>
> Rasterio's test suite has 4 errors with GDAL 3.9.0beta1.
>
>
>
> Metadata output of gdalinfo has changed. As soon as there are docker
> images I'll look more closely at this.
>
>
>
> Writing a mask to a Rasterio RGB.byte.tif dataset no longer creates a 
> RGB.byte.tif.msk
> file:
> https://github.com/rasterio/rasterio/actions/runs/8786444142/job/24109342188#step:8:2231
> .
>
>
>
> The text of an error message which was "tests/data/corrupt.tif, band 1:
> IReadBlock failed at X offset 1, Y offset 1: TIFFReadEncodedTile() failed."
> in 3.8 has changed to "corrupt.tif, band 1: IReadBlock failed at X offset
> 1, Y offset 1: TIFFReadEncodedTile() failed." This is not a big deal, we
> can adjust Rasterio's test.
>
>
>
> Behavior of SRC_METHOD/DST_METHOD=NO_GEOTRANSFORM for warp options seems
> to have changed:
> https://github.com/rasterio/rasterio/actions/runs/8786444142/job/24109342188#step:8:2287.
> I'm not entirely sure what the intent is in Rasterio's test, will dig into
> that. I just wanted to say that I see a change in GDAL.
>
>
>
>
>
> On Mon, Apr 22, 2024 at 6:12 AM Even Rouault via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
> Hi,
>
> I've prepared a beta1 of GDAL 3.9.0 to get feedback from early testers.
>
> The NEWS file is here:
>
>https://github.com/OSGeo/gdal/blob/v3.9.0beta1/NEWS.md
>
> For packagers,
> https://gdal.org/development/rfc/rfc96_deferred_plugin_loading.html may
> make it more attractive to build some drivers that typically have heavy
> dependencies as plugins, installable in separate packages, due to the
> load time penalty having being improved. It is let to the appreciation
> of packagers to decide which drivers are worth building as plugins
> installable in a separate package.
> You may also pass CMake options
> ([GDAL/OGR]_DRIVER__PLUGIN_INSTALLATION_MESSAGE=xxx) so
> user get a hint of which package they need to install when GDAL detects
> that a file may be opened by a plugin which is not available on the file
> system but known to have been built as a plugin. Cf
>
> https://gdal.org/development/building_from_source.html#deferred-loaded-plugins
> for more details.
>
> For end-users, the following utilities have been updated to use a new
> argument parsing framework: gdaladdo, gdalinfo, gdal_translate,
> gdalwarp, gdal_grid, gdal_viewshed, gdalbuildvrt, nearblack, ogrinfo,
> ogr2ogr, sozip.
> This helps detecting errors such as specifying twice an argument that
> should be specified once (where generally the last instance was the one
> used). While we have tried to retain backwards compatibility for nominal
> use cases, obviously if you have scripts that accidentally specify an
> argument several times whereas it should be specified at most once, they
> will have to be corrected.
>
> Docker images with 3.9.0beta1 are currently cooking.
>
> master is now marked as 3.10.0dev, and the release/3.9 branch has been
> created. All bugfixes for 3.9 should be backported into it.
>
> Source snapshots at:
>
> - https://download.osgeo.org/gdal/3.9.0/gdal-3.9.0beta1.tar.gz
> - https://download.osgeo.org/gdal/3.9.0/gdal-3.9.0beta1.tar.xz
> - https://download.osgeo.org/gdal/3.9.0/gdal380beta1.zip
>
> Autotest snapshots:
>
> - https://download.osgeo.org/gdal/3.9.0/gdalautotest-3.9.0beta1.tar.gz
> - https://download.osgeo.org/gdal/3.9.0/gdalautotest-3.9.0beta1.zip
>
> Even
>
>
>
> --
>
> Sean Gillies
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.9.0beta1 available for testing

[Sean Gillies via gdal-dev]

Hi Even,

Rasterio's test suite has 4 errors with GDAL 3.9.0beta1.

Metadata output of gdalinfo has changed. As soon as there are docker images
I'll look more closely at this.

Writing a mask to a Rasterio RGB.byte.tif dataset no longer creates a
RGB.byte.tif.msk
file:
https://github.com/rasterio/rasterio/actions/runs/8786444142/job/24109342188#step:8:2231
.

The text of an error message which was "tests/data/corrupt.tif, band 1:
IReadBlock failed at X offset 1, Y offset 1: TIFFReadEncodedTile() failed."
in 3.8 has changed to "corrupt.tif, band 1: IReadBlock failed at X offset
1, Y offset 1: TIFFReadEncodedTile() failed." This is not a big deal, we
can adjust Rasterio's test.

Behavior of SRC_METHOD/DST_METHOD=NO_GEOTRANSFORM for warp options seems to
have changed:
https://github.com/rasterio/rasterio/actions/runs/8786444142/job/24109342188#step:8:2287.
I'm not entirely sure what the intent is in Rasterio's test, will dig into
that. I just wanted to say that I see a change in GDAL.


On Mon, Apr 22, 2024 at 6:12 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> I've prepared a beta1 of GDAL 3.9.0 to get feedback from early testers.
>
> The NEWS file is here:
>
>https://github.com/OSGeo/gdal/blob/v3.9.0beta1/NEWS.md
>
> For packagers,
> https://gdal.org/development/rfc/rfc96_deferred_plugin_loading.html may
> make it more attractive to build some drivers that typically have heavy
> dependencies as plugins, installable in separate packages, due to the
> load time penalty having being improved. It is let to the appreciation
> of packagers to decide which drivers are worth building as plugins
> installable in a separate package.
> You may also pass CMake options
> ([GDAL/OGR]_DRIVER__PLUGIN_INSTALLATION_MESSAGE=xxx) so
> user get a hint of which package they need to install when GDAL detects
> that a file may be opened by a plugin which is not available on the file
> system but known to have been built as a plugin. Cf
>
> https://gdal.org/development/building_from_source.html#deferred-loaded-plugins
> for more details.
>
> For end-users, the following utilities have been updated to use a new
> argument parsing framework: gdaladdo, gdalinfo, gdal_translate,
> gdalwarp, gdal_grid, gdal_viewshed, gdalbuildvrt, nearblack, ogrinfo,
> ogr2ogr, sozip.
> This helps detecting errors such as specifying twice an argument that
> should be specified once (where generally the last instance was the one
> used). While we have tried to retain backwards compatibility for nominal
> use cases, obviously if you have scripts that accidentally specify an
> argument several times whereas it should be specified at most once, they
> will have to be corrected.
>
> Docker images with 3.9.0beta1 are currently cooking.
>
> master is now marked as 3.10.0dev, and the release/3.9 branch has been
> created. All bugfixes for 3.9 should be backported into it.
>
> Source snapshots at:
>
> - https://download.osgeo.org/gdal/3.9.0/gdal-3.9.0beta1.tar.gz
> - https://download.osgeo.org/gdal/3.9.0/gdal-3.9.0beta1.tar.xz
> - https://download.osgeo.org/gdal/3.9.0/gdal380beta1.zip
>
> Autotest snapshots:
>
> - https://download.osgeo.org/gdal/3.9.0/gdalautotest-3.9.0beta1.tar.gz
> - https://download.osgeo.org/gdal/3.9.0/gdalautotest-3.9.0beta1.zip
>
> Even
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Block cache and VRT sources

[Sean Gillies via gdal-dev]

Thanks, Even. I think I've got my head around it. Since I've never written
a driver, I've never fully learned the difference between IReadBlock and
IRasterIO or where one is used instead of the other. But I'm learning that
now.

On Fri, Apr 19, 2024 at 9:35 AM Even Rouault 
wrote:

>
> Le 19/04/2024 à 17:28, Sean Gillies a écrit :
>
> Even,
>
> Does the shared attribute of a VRT sourceFilename element not affect
> caching at all?
>
> If shared is set to 0, then one GDALDataset per VRTSource will be opened.
> This has little benefit.
>
> The scope of sharing was initially greater since it extented to
> potentially several GDALDataset instance of VRT, but this wasn't safe in
> multithreaded scenarios where you would read one VRT in a thread, another
> one in another thread, but both would point to let's same the same TIFF
> GDALDataset. Hence the scope of sharing was reduced to a single GDALDataset
> VRT instance to be safe by default.
>
> Is the cache avoided so that potentially stale data isn't propagated, or
> for other reasons?
>
> Are you reacting to "if it is triggered. VRTSource::IRasterIO() calls
> IRasterIO() on the source band, which doesn't necessarily always trigger
> block-based reading" ?  I just meant that calling IRasterIO() on a
> GDALDataset/GDALRasterBand doesn't necessarily cause the block cache to
> trigger. This is dependent of the driver and the parameters of the request.
> And this isn't specific to use it from a VRT.
>
>
> On Fri, Apr 19, 2024 at 9:09 AM Even Rouault 
> wrote:
>
>> Sean,
>>
>> Within a given GDALDataset opened on a VRT, if the VRT references several
>> times the same source, only one GDALDataset will be opened for it, so you
>> may benefit from the block cache mechanism (if it is triggered.
>> VRTSource::IRasterIO() calls IRasterIO() on the source band, which doesn't
>> necessarily always trigger block-based reading).  But if you open another
>> VRT (or the same one), it will not share the same GDALDataset for sources
>> that may be common with the first one, so no re-use of existing block
>> cache. For network sources, the I/O cache at the /vsicurl/ level works
>> however on filenames, not VSIFILE* instances, so you will save network reads
>>
>> Even
>> Le 19/04/2024 à 16:48, Sean Gillies via gdal-dev a écrit :
>>
>> Happy Friday, folks!
>>
>> Are the source rasters of a VRT added to the block cache such that
>> different VRTs using the same source can avoid reads from disk or the
>> network? Or is it intended that the VSI cache covers this need instead?
>>
>> Thanks,
>>
>>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Block cache and VRT sources

[Sean Gillies via gdal-dev]

Even,

Does the shared attribute of a VRT sourceFilename element not affect
caching at all? Is the cache avoided so that potentially stale data isn't
propagated, or for other reasons?

On Fri, Apr 19, 2024 at 9:09 AM Even Rouault 
wrote:

> Sean,
>
> Within a given GDALDataset opened on a VRT, if the VRT references several
> times the same source, only one GDALDataset will be opened for it, so you
> may benefit from the block cache mechanism (if it is triggered.
> VRTSource::IRasterIO() calls IRasterIO() on the source band, which doesn't
> necessarily always trigger block-based reading).  But if you open another
> VRT (or the same one), it will not share the same GDALDataset for sources
> that may be common with the first one, so no re-use of existing block
> cache. For network sources, the I/O cache at the /vsicurl/ level works
> however on filenames, not VSIFILE* instances, so you will save network reads
>
> Even
> Le 19/04/2024 à 16:48, Sean Gillies via gdal-dev a écrit :
>
> Happy Friday, folks!
>
> Are the source rasters of a VRT added to the block cache such that
> different VRTs using the same source can avoid reads from disk or the
> network? Or is it intended that the VSI cache covers this need instead?
>
> Thanks,
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Block cache and VRT sources

[Sean Gillies via gdal-dev]

Happy Friday, folks!

Are the source rasters of a VRT added to the block cache such that
different VRTs using the same source can avoid reads from disk or the
network? Or is it intended that the VSI cache covers this need instead?

Thanks,

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] CVE-2024-3094 (aka "xz hackdoor") and GDAL

[Sean Gillies via gdal-dev]

Thank you for the analysis, Even!

I've made similar announcements for the fiona and rasterio projects and
plagiarized your subject line. Fiona and rasterio wheels on PyPI include
versions of liblzma no more recent than 5.4.4 and do not include libarchive.

https://github.com/Toblerity/Fiona/discussions/1367
https://github.com/rasterio/rasterio/discussions/3047

On Sat, Mar 30, 2024 at 1:16 PM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> TLDR: no specific reason to worry.
>
> My longer analysis:
>
> Those following the recent security news have certainly come across
> https://lwn.net/Articles/967180 or similar articles, and if you don't
> have and have been running a cutting edge Linux distribution recently, you
> *should* follow related security alerts emitted by your vendor.
>
> liblzma (the library part of the xz software distribution) is an optional
> libtiff dependency for the TIFF LZMA codec, and an optional direct GDAL
> dependency since GDAL 1.8.0. Initially only used when building GDAL with
> its internal libtiff, and since GDAL 3.4.0, directly too as a Zarr codec.
>
> Beyond activity in liblzma / xz being potentially suspicious with at least
> 2 years of "contributions" by malicious actor https://github.com/JiaT75,
> which makes the current headlines, it has also been found that they
> contributed to libarchive too since 2021 (
> https://github.com/libarchive/libarchive/commits?author=JiaT75), and
> their latest commit https://github.com/libarchive/libarchive/pull/1609 is
> in particular pointed as suspicious. As far as I can see, this commit is
> present in libarchive >= 3.6. I mention libarchive because it is an
> optional GDAL dependency since GDAL 3.7.0 for the /vsi7z/ and /vsirar/
> virtual file systems.
>
> We don't have vendored copies of liblzma or libarchive in the GDAL source
> code repository.
>
> Regarding GDAL Docker images, here's the status of versions they ship:
>
> - ghcr.io/osgeo/gdal:alpine-small-3.8.4 and
> ghcr.io/osgeo/gdal:alpine-normal-3.8.4:
> xz-libs-5.4.3-r0
> libarchive-3.7.2-r0
>
> - ghcr.io/osgeo/gdal:ubuntu-full-3.8.4:
> liblzma5:amd645.2.5-2ubuntu1
> libarchive13:amd643.6.0-1ubuntu1
>
> - ghcr.io/osgeo/gdal:ubuntu-small-3.8.4:
> liblzma5:amd64  5.2.5-2ubuntu1
>
> So those liblzma versions are not currently considered as affected.
> Regarding libarchive status and the impact of JiaT75's activity there, it
> seems to me we are yet in undecided territory and it wouldn't be surprising
> to see update of those libraries in the coming weeks, as exploration of
> past activity from that compromising entity is performed.
>
> It is a good time to recall usual general best-practices:
>
> - avoid opening datasets coming from unknown or untrusted sources
>
> - if you do so, prefer doing that on dedicated workstations that don't
> have access to sensitive information
>
> - you may reduce your potential exposure by disabling at build time
> optional dependencies you don't need.
>
> Even
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSI sidecar and sibling file lookup

[Sean Gillies via gdal-dev]

Update... It looks like there are at least two different kinds of logic
around sidecar/sibling files.

My Rasterio VSI plugin currently implements only open, tell, seek, read,
write, and close. The mandatory methods. I can see GDAL probing for .aux
and .AUX files. But not .aux.xml files for some reason that I don't
understand. The dataset I'm exposing through this VSI plugin has a .ovr
file, but there's no attempt to find it. It looks like .aux.xml and .ovr
require VSI sibling files and/or readdir support, but that .aux file
support does not? I'm deducing this from the behavior of the system. It
would be hard to figure it out just by reading the code.

On Fri, Feb 16, 2024 at 8:29 AM Sean Gillies  wrote:

> Thanks for the tip, Thomas!
>
> On Thu, Feb 15, 2024 at 9:45 AM thomas bonfort 
> wrote:
>
>> Hi Sean,
>> I believe/recall this is very driver dependent. Some drivers will look
>> for their sidecars in the provided sibling files list returned by
>> VSISiblingFiles, whereas others will unconditionally try to open well-known
>> sidecar names they will have computed from their own name. IIRC I added
>> VSISiblingFiles so that some vsi backends (namely object storage based
>> ones) could explicitly return an empty list of sibling files in order to
>> prevent drivers to emit a subsequent Readdir() (which might break drivers
>> that require sidecar files to work, but speeds up those where sidecar files
>> are optional and not used in a cloud-storage environment (namely cogs) )
>>
>> On Tue, Feb 13, 2024 at 6:12 PM Sean Gillies via gdal-dev <
>> gdal-dev@lists.osgeo.org> wrote:
>>
>>> Hi all,
>>>
>>> It's not clear to me from reading
>>> https://gdal.org/user/virtual_file_systems.html if VSI sidecar and
>>> sibling file lookup works in general, by design, or whether it's an
>>> implementation detail of the standard VSI filesystems ("vsizip", "vsicurl",
>>> etc.).
>>>
>>> More specifically: if I have a "/vsipyopener/foo.tif" object, do drivers
>>> always look for other files at paths precisely relative to that one? Does
>>> this vary among drivers?
>>>
>>
>
> --
> Sean Gillies
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSI sidecar and sibling file lookup

[Sean Gillies via gdal-dev]

Thanks for the tip, Thomas!

On Thu, Feb 15, 2024 at 9:45 AM thomas bonfort 
wrote:

> Hi Sean,
> I believe/recall this is very driver dependent. Some drivers will look for
> their sidecars in the provided sibling files list returned by
> VSISiblingFiles, whereas others will unconditionally try to open well-known
> sidecar names they will have computed from their own name. IIRC I added
> VSISiblingFiles so that some vsi backends (namely object storage based
> ones) could explicitly return an empty list of sibling files in order to
> prevent drivers to emit a subsequent Readdir() (which might break drivers
> that require sidecar files to work, but speeds up those where sidecar files
> are optional and not used in a cloud-storage environment (namely cogs) )
>
> On Tue, Feb 13, 2024 at 6:12 PM Sean Gillies via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
>> Hi all,
>>
>> It's not clear to me from reading
>> https://gdal.org/user/virtual_file_systems.html if VSI sidecar and
>> sibling file lookup works in general, by design, or whether it's an
>> implementation detail of the standard VSI filesystems ("vsizip", "vsicurl",
>> etc.).
>>
>> More specifically: if I have a "/vsipyopener/foo.tif" object, do drivers
>> always look for other files at paths precisely relative to that one? Does
>> this vary among drivers?
>>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] VSI sidecar and sibling file lookup

[Sean Gillies via gdal-dev]

Hi all,

It's not clear to me from reading
https://gdal.org/user/virtual_file_systems.html if VSI sidecar and sibling
file lookup works in general, by design, or whether it's an implementation
detail of the standard VSI filesystems ("vsizip", "vsicurl", etc.).

More specifically: if I have a "/vsipyopener/foo.tif" object, do drivers
always look for other files at paths precisely relative to that one? Does
this vary among drivers?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] What is the empty pixel value for datasets?

[Sean Gillies via gdal-dev]

Hi all,

I haven't found a specification for the empty pixel value of datasets
created by, say, gdal_create. Is it format specific?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Where to fins my own files

[Sean Gillies via gdal-dev]

Happy New Year to you too, Abel.

Other drivers, such as PCI and NITF,  include some files in the data
directory.  See  https://github.com/OSGeo/gdal/tree/master/data.
Distributions generally copy these to $prefix/share/gdal on unix-like
systems. I'm not aware of a practice of copying data out of a driver's
directory to that same location, though it's probably possible with Cmake.
Your driver's own directory won't necessarily exist on a system where you
install GDAL.

For datum files, have you checked whether they already exist in the PROJ
project?

On Sat, Jan 6, 2024, 2:32 PM Abel Pau via gdal-dev 
wrote:

> Hi,
>
> Happy New 2024 to everyone who celebrates it!
>
>
>
> I am programming a driver and in some point I need to read a file that
> contains some information (and it’s better to keep this in a file than in c
> code) about the Horizontal Reference System.
>
> I have this file here: “GDAL\ogr\ogrsf_frmts\miramon”, but when coding I
> don’t know how to get to this folder.
>
>
>
> There is any method to open a file (using VSIFOpenL(), for instance) in
> THIS specific directory? Can I use “GDAL\ogr\ogrsd_frmts\miramon” like that?
>
> If not, where can I put this file to be found when executing the code? And
> how to write this path in the code?
>
> Thanks in advance!!
>
>
>
>
>
>
>
> *Abel Pau Garcia*
>
> *GIS developer*
>
> [image: https://www.creaf.cat/sites/default/files/creaf-signature.png]
>
> *a@creaf.uab.cat* 
>
> *Let's chat on Teams!*
> 
>
> *Tel. +34 934814277*
>
> [image: https://www.creaf.cat/sites/default/files/so-en-signature.png]
>
> [image:
> https://www.creaf.cat/sites/default/files/twitter-icon-signature.png]
> [image:
> https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png]
> [image:
> https://www.creaf.cat/sites/default/files/youtube-icon-signature.png]
> [image:
> https://www.creaf.cat/sites/default/files/instagram-icon-signature.png]
> 
>
> *www.creaf.cat* * | **http://blog.creaf.cat*
> 
>
> [image: https://www.creaf.cat/sites/default/files/uab_logo_signatura.png]
>
> CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona)
>
>
> Before printing this electronic message, think about the environment.
>
> [image: http://www.creaf.uab.cat/_signatura/line.gif]
>
>
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Virtual Raster Tile Index (VRTTI) driver, and associated gdaltindex improvements

[Sean Gillies via gdal-dev]

Hi Even,

I'm wondering how this relates to STAC. Do you imagine that GDAL users
should and will use this to publish large collections of data? When should
they use this and when should they use STAC collections instead?

Back in the day our tile indexes were shapefiles. I don't think this would
be a good practice now. Would all vector formats be supported? How about
column-oriented formats? How backwards compatible will this be with
existing tile index files? Can I just do a gdalinfo ":example.shp" on a
MapServer tile index?

This feature has been latent and available for a while, yes? It's obviously
useful. My main misgivings are about the overlap with existing GDAL
features and protocols (VRT, WMS, WMTS, STAC, MBTiles). I think too many
choices can create a not great experience for users.

On Wed, Dec 20, 2023, 11:33 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> For those not actively following github tickets & PR, I just want to
> point to a new pending major functionality to improve management of
> virtual mosaics with a very large number of tiles/sources (> tens of
> thousands of tiles), by referencing them as features of a vector layer
> (typically created by gdaltindex), instead of a XML file as in
> traditional VRT, augmented with additional metadata.
>
> More details in https://github.com/OSGeo/gdal/pull/8983 (and in initial
> ticket in https://github.com/OSGeo/gdal/issues/8861)
>
> Even
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Requiring numpy for the Python bindings

[Sean Gillies via gdal-dev]

For what it's worth, Rasterio has required Numpy for a few versions, and
there's never been a single complaint about it. The footprint and extra
complexity of numpy is just a drop in the bucket.

I suppose that someone using only the OGR parts of the Python bindings
might be inconvenienced, but not by much. And I suspect that the number of
these people who don't already have Numpy in their environment via
GeoPandas or Shapely is pretty small.

On Mon, Dec 4, 2023 at 3:52 PM Daniel Evans via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Agree with Kurt - it's always struck me as a little surprising that it was
> optional. There's also not very much of a Python stack you can build that
> involves GDAL but doesn't pull in Numpy through another dependency, as Even
> said.
>
> I would've thought that in any constrained environment where GDAL is
> required, but Numpy is too much of a overhead, that using Python would be a
> somewhat unusual choice to go with over the many other bindings that GDAL
> offers.
>
> Cheers,
> Daniel
>
> On Mon, 4 Dec 2023, 18:50 Kurt Schwehr via gdal-dev, <
> gdal-dev@lists.osgeo.org> wrote:
>
>> I lean towards just requiring numpy. It's super common and once a system
>> brings in gdal python, it can't be a super constrained env where keeping
>> things really small is critical. Just requiring numpy simplifies a number
>> of aspects.
>>
>> I think the setup.py topic as a whole is somewhat separate.l.  So
>> whichever way the numpy dep goes (required or optional), switching to
>> pyproject.toml seems important. I'm fairly new to pyproject.toml, so I
>> might be missing something.
>>
>> -Kurt
>>
>> On Mon, Dec 4, 2023 at 10:38 AM Even Rouault via gdal-dev <
>> gdal-dev@lists.osgeo.org> wrote:
>>
>>> Hi,
>>>
>>> The current situation where numpy is an optional dependency of the GDAL
>>> Python bindings is quite cumbersome to deal with our setup.py's
>>> setuptools . All details (a bit tricky) in
>>> https://github.com/OSGeo/gdal/issues/8069 . It seems it would be
>>> simpler
>>> if the bindings just required numpy, which is the confugration most
>>> people using the bindings likely actually end up using anyway.
>>>
>>> Any opposition to that?
>>>
>>> Even
>>>
>>>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.1RC3 is available & motion to approve it

[Sean Gillies via gdal-dev]

I finally have configured a GitHub action that tests rasterio against the
latest GDAL tags and don't see any problems with 3.8.1RC3:
https://github.com/rasterio/rasterio/actions/runs/7022736307.

+1 for me

On Tue, Nov 28, 2023 at 8:32 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> well, another annoying 3.8.0 regression related to ogr2ogr GPKG ->
> Shapefile when field names are longer than 10 characters (ah
> Shapefile!!!) popped up today, so here's a RC3 (crossing fingers it's
> the good one):
>
>https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc3.tar.xz
>https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc3.tar.gz
>https://download.osgeo.org/gdal/3.8.1/gdal381rc3.zip
>
> A snapshot of the gdalautotest suite is also available:
>
> https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc3.tar.gz
>https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc3.zip
>
> The changes since RC2 are:
>
> * CMake: make GDAL_USE_LIBKML and GDAL_USE_OPENJPEG honor
> GDAL_USE_EXTERNAL_LIBS
> * Detect failure in installation of the Python bindings
> * Fix installation issue with Python 3.12 on Debian
> * GDALOverviewDataset::IRasterIO(): use parent dataset when possible for
> more
>efficiency
> * gdal_footprint: fix wrong taking into account of alpha band (#8834)
> * gdal_footprint: fix taking into account of individual bands that have
> nodata
> * COG: for JPEG compression, convert single band+alpha as single band
> JPEG + 1-bit mask band
> * GTIFF SRS reader: include VertCRS name from EPSG in CompoundCRS name
> if there's no citation geokey
> * ogr2ogr: fix GPKG -> Shapefile when field names are truncated (#8849,
> 3.8.0 regression)
> * CSV writer: do not quote integer fields by default (only if
> STRING_QUOTING=ALWAYS is specified)
> * GPX: make detection of extensions element more robust (#8827)
> * Shapefile: recognize '  0' as a null date, fix writing an invalid
> "/00/00" date
> * Python bindings: add a combineBands option to gdal.Footprint()
>
>
> Withdrawing previous motion and:
>
> ==> Motion: adopt 3.8.1 RC3 as final 3.8.1 release
>
> Starting with my +1,
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

[Sean Gillies via gdal-dev]

Sounds good to me, Even. Rasterio's wheels can remain at the forefront of
terrible for now.

On Thu, Nov 16, 2023 at 5:10 AM Even Rouault 
wrote:

> Hi Sean,
> >
> > I think this makes great sense for the project. I don't yet understand
> > what it means for an enterprise like Rasterio's PyPI wheels.
>
> I'd say it probably changes nothing. The RFC just postpones the time
> where the plugins are loaded, but the fact that they are dlopen()'ed
> (early or late) probably makes them non discoverable by delocate, since
> libgdal doesn't link to them in a way that is advertised in its shared
> library metadata. If your plan is to still have a rasterio wheel with a
> monolithic GDAL, then you don't need to build GDAL drivers as plugins
> and this RFC doesn't change anything to the status quo.
>
> I'm not familiar at all with the wheel Python packaging tools, but if
> you'd want to have GDAL plugins in separate package(s) then you'd need
> to have a way of having the gdal_XXX.so / ogr_XXX.so be put in some
> known location that can be advertized to libgdal core with
> GDAL_DRIVER_PATH.
>
> Even
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

[Sean Gillies via gdal-dev]

Hi Even,

I think this makes great sense for the project. I don't yet understand what
it means for an enterprise like Rasterio's PyPI wheels.

Here's a refresher for people who aren't familiar with Python packaging
tools like delocate, auditwheel, and delvewheel. Today, these tools detect
when Rasterio extension modules link libgdal, and then find the libraries
that libgdal links via metadata in the shared libraries. This allows
libgdal and its dependencies to be relocated so that they can be
distributed alongside Rasterio's Python and extension modules.

I don't know that delocate et al will be able to find libraries that are
dlopened by the new GDAL proxy drivers. Does anyone else know? I'm starting
a new job this week and may not have time to experiment until next week.


On Wed, Nov 15, 2023 at 2:52 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> Motion: adopt RFC 96: Deferred C++ plugin loading
>
> https://github.com/OSGeo/gdal/pull/8648
>
> Pre-rendered view:
>
>
> https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst
>
>
> Starting with my +1,
>
> Even
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release

[Sean Gillies via gdal-dev]

I'm changing Rasterio's CI to test against both the head of the master
branch and the head of the current release branch, so in the future I'll
likely only speak up if something is broken right before the release.
3.7.3RC1 looks good to me. I can't think of anything missing.

On Wed, Nov 1, 2023 at 5:14 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> Motion:
>
> Adopt GDAL 3.7.3RC1 as 3.7.3 release
>
> Starting with my +1
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

[Sean Gillies via gdal-dev]

Hi Even,

Rasterio's CI picked up a change to the AAIGrid driver in 3.8. The 3.7
version driver used to have whitespace before a row and no whitespace
after. It looks like this has flipped in 3.8. Is it intentional? I only
noticed because one of my tests is parsing the file as text. It's certainly
not a big deal. Hopefully this format has mostly expired on the internet :)

On Tue, Oct 31, 2023 at 8:34 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers.
>
> Sorry no updated NEWS.md file yet, but I'd in particular be interesting
> by testing of ogr2ogr workflows, since they have underwent significant
> changes in the underlying implementation:
>
> - when the source layer is a layer implementing the ArrowStream API
> (that is  GeoPackage, FlatGeoBuf, Arrow or Parquet), and when no ogr2ogr
> options than -of, -where, -spat, -lco, -dsco, -gt,
> -append/-overwrite/-update are used (and -sql as well for GeoPackage).
> When enabling CPL_DEBUG, you'll see a "OGR2OGR: Using WriteArrowBatch()"
> trace when that new code path is taken. If specifying other options, the
> feature-iteration-based traditional implementation is used
>
> - and/or when the output layer is GeoPackage (new layer), due to the
> revamped much faster spatial index creation.  This enhanced spatial
> index creation is not ogr2ogr specific and is actually available more
> generally for CreateLayer() + CreateFeature() or CreateLayer() +
> WriteArrowBatch() scenarios.
>
> Point of attention would be when in situations with large files and/or
> with low RAM.
>
> The ghcr.io/osgeo/gdal:ubuntu-small-latest,
> ghcr.io/osgeo/gdal:ubuntu-full-latest,
> ghcr.io/osgeo/gdal:alpine-normal-latest Docker images have been
> refreshed with 3.8.0beta1 (ghcr.io/osgeo/gdal:alpine-small-latest still
> building at time of writing).
>
> (Note: the GDAL master conda builds mentioned at
> https://gdal.org/download.html#gdal-master-conda-builds have been broken
> for a couple weeks and are thus not usable to test beta1 currently. I'm
> investigating)
>
> Source snapshots at:
>
> - https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.gz
>
> - https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.xz
>
> - https://download.osgeo.org/gdal/3.8.0/gdal380beta1.zip
>
> Autotest snapshots:
>
> - https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.tar.gz
>
> - https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.zip
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression

[Sean Gillies via gdal-dev]

Thanks for the announcement, Even! I wonder if we should track such issues
in a list? Or maybe give them a unique GitHub label?

We don't plan to release a 3.6.5, correct?

I'm going to make a Rasterio post release that patches 3.6.4 by tomorrow:
https://github.com/rasterio/rasterio/issues/2943.

On Mon, Oct 16, 2023 at 9:26 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

>
> Le 16/10/2023 à 17:14, Kurt Schwehr a écrit :
>
> Thanks for the heads up!
>
> Can you share that SHAs of the fix and cause?
>
> master: d083af1ec9c38e79bfb0532885570de6e5b8a410
>
> 3.7 branch (should apply to 3.6 as well):
> b5858ed5bc5004c97f7cd6000674015bdc70b33b
>
> cause: a worker thread of the multithreaded decoding could, in a situation
> where the block cache is full, cause a "dirty" (ie modified but not yet
> serialized to disk) block to be written, resulting in either a deadlock
> between the lock of the multithreaded decoder and the lock of the job queue
> mechanism, or other decoding threads could see their file handle being
> seek() by another thread.  In short: multithreading is hard.
>
>
> On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
>> Hi,
>>
>> For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff
>> compression+decompression, in particular within the same file, as for
>> example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow
>> can lead to deadlocks (process stalled forever) and/or data corruption
>> (sometimes without errors at generation). Cf
>> https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is
>> in https://github.com/OSGeo/gdal/pull/8561
>>
>> The issue is particularly visible on Windows, or more generally any
>> operating system (or file system where the output file is located) which
>> has no VSIVirtualHandle::PRead() implementation, but it can also be
>> occasionally reproduced on Linux (at least as a deadlock).
>>
>> Even
>>
>>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL Maintainers Meeting Minutes

[Sean Gillies via gdal-dev]

Thank you, Howard for reminding sponsors how important their support is!

On Fri, Oct 6, 2023 at 9:14 AM Howard Butler via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Howard Butler, Even Rouault, Dan Baston, and Alessandro Pasotti held the
> monthly GDAL Maintainers Meeting on 09/28/2023. The following items were
> discussed and reported upon:
>
> Fundraising update
> 
>
> Good news on fundraising: Amazon Web Services, who had previously
> announced they were not going to be able to renew their Gold level
> sponsorship for 2024, was able to reverse the decision and continue
> sponsoring GDAL at the same level for the next year.
>
> Also Maxar, who had previously announced they were not going to be able to
> renew their Gold level for 2024, was able to reverse the decision and
> continue sponsoring GDAL for the next year the Bronze level.
>
> Thanks again to the advocates within both of these organizations for
> continuing to push for resources despite being told 'no'. We still have a
> number of renewals to complete before we are done for the cycle this year.
>
> Maintenance activities update
> --
>
> * Even attended the NumFOCUS Project Summit September 11-13th in
> Amsterdam. https://www.summit.numfocus.org/ where he gave a lightning
> talk about GDAL's extensive CI system
> http://download.osgeo.org/gdal/presentations/NumFocus%20Summit%202023%20Tour%20of%20tools%20used%20by%20GDAL%20CI%2C%20testing%20and%20documentation.pdf
> and discussed Arrow topics with Joris Van den Bossche and Martin
> Fleischmann of the GeoPandas project. He provided a full report to the GDAL
> PSC about his attendance.
>
> * Even coordinated and issued the GDAL 3.7.2 release. Detailed release
> notes about what it included can be found at
> https://github.com/OSGeo/gdal/blob/v3.7.2/NEWS.md
>
> *Even coordinated and issued the libtiff 4.6.0 release. Detailed release
> notes about what it included can be found at
> https://gitlab.com/libtiff/libtiff/-/releases/v4.6.0
>
> * Even spent significant time updating the PDF driver in response to QGIS
> performance scenarios that showed it needed attention. This included
> updating the backend to use Podofo 0.10.
>
> * Even wrote RFC 95 to align GDAL's integer types with typical stdint
> approaches. https://github.com/OSGeo/gdal/pull/8399 It was decided to
> table implementation until GDAL has its next major (4.0) release. The next
> release, 3.8.0 due this fall, is not intended to be that major release.
>
> * Maintainers had a short discussion about adding something like an
> GDALOpenJSON method to allow applications to provide a single JSON data
> structure for layer and data source options, along with VSI protocol hints
> and (maybe) environment variables. Discussion about such an idea should
> continue in open forums such as the mailing list.
>
> * Alessandro worked on the subdataset API and thought about approaches to
> provide the entire set of option possibilities (paths, open options,
> dataset options, etc), which prompted the GDALOpenJSON discussion above.
>
> * Dan continued working through making all of the GDAL test suite tests
> run independently. That important milestone is almost complete.
>
> * Dan exposed GDALClose to the Python bindings in support of explicit
> destruction of resources for the test suite (rather than hoping __del__
> does it by dropping a reference).
>
> * Dan began working up a Ubuntu 22.04-based CI test container.
>
> Upcoming Events
> --
>
> * FOSS4GNA 2023 is being held in Baltimore, Maryland, USA on October
> 23-25th.
>
> * OSGeo 2023 Community Sprint will be held in Vienna, Austria November
> 6-9th, 2023. https://wiki.osgeo.org/wiki/OSGeo_Community_Sprint_2023 GDAL
> PSC members seeking travel support should contact Howard Butler privately
> to coordinate.
>
> Next Meeting
> --
>
> The next GDAL Maintainers Meeting is 10/26/2023 at 9:00 EDT. Any PSC
> members are welcome to join by reaching out to me for an invite.
>
> Howard
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Annual Contracts for Maintainers

[Sean Gillies via gdal-dev]

Hi Howard,

To be clear, this is a 50% time increase for Alessandro, yes? I think
that's great!

+1

On Wed, Oct 11, 2023 at 10:16 AM Howard Butler via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> PSC,
>
> I'm a little late but I would like to make the following motions in
> regards to GDAL maintainers:
>
> * Authorize Alessandro Pasotti for up to 30 hrs/month with a rate increase
> of 10 euros/hr until 30 NOV 2024.
> * Authorize Even Rouault for up to 90 hrs/month with a rate increase of 10
> $/hr until 30 JUL 2024
>
> Howard
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for discussion/review of RFC95: Use standard C/C++ integer types

[Sean Gillies]

I've asked before about making GDAL's error propagation system better.
That's one thing I'd like to add to a list of proposed changes for 4.0.
Could we have a separate call for 4.0 ideas and review it next month (or
later)? I'm so busy with a running project and looking for a new job that I
don't have as much time for open source project roadmapping as I did a few
weeks ago.

On Wed, Sep 20, 2023 at 11:23 AM Even Rouault 
wrote:

> Hi Even,
>
>
> I'm in favor of overhauling the types in the next major version. However,
> I'm not convinced we need to jump immediately to 4.0. The current situation
> isn't ideal, but isn't holding us back very much right?
>
> No, there's no urgency. It is just one of the many continuous rather
> boring improvements we do in the code base, except that one is noticed
> externally.  My pain concern about defering is that the candidate
> implementation will rot quickly, and the manual parts are painful to redo
> (but nothing dramatic: a few hours of effort, not weeks).
>
>
> Speaking for Fiona and Rasterio, supporting GDAL versions 3.5-3.7 and 4.0
> at the same time will be a pain and will require some conditional
> compilation changes throughout the code. These projects will need some time
> to prepare.
>
> I've experimented a bit about trying the RFC 95 branch to build mapserver,
> PDAL and QGIS against it and being compatible with older GDAL versions as
> well:
>
> - https://github.com/MapServer/MapServer/pull/6936 : minimal change is
> just to define GDAL_USE_OLD_INT_TYPES
>
> - https://github.com/PDAL/PDAL/pull/4179 : minimal change is just to
> define GDAL_USE_OLD_INT_TYPES
>
> - https://github.com/qgis/QGIS/pull/54680: set GDAL_USE_OLD_INT_TYPES, a
> few if GDAL >= 4.0 branches and a few other changes that make it work with
> all GDAL versions
>
> So nothing dramatic. Hopefully rasterio or fiona wouldn't be too
> troublesome to adapt
>
> And I'd be more enthusiastic about supporting 3.7 and 4.0 simultaneously
> if 4.0 made GDAL's C API tangibly better, like dataset unification in 2.0
> or the PROJ switch in 3.0.
>
> Do you have something in mind about a tangibly better improvement to the
> API that would make it worth a 4.0?
>
> Even
>
>
> On Tue, Sep 19, 2023 at 10:30 AM Even Rouault 
> wrote:
>
>> No other reaction ? Are people comfortable with bumping to 4.0 ? If so,
>> no opinion on what we should slip in while we are it (thinking more
>> about breaking changes than new features, which generally can be added
>> afterwards) ?
>>
>> Le 16/09/2023 à 14:53, Even Rouault a écrit :
>> >
>> >> About GDAL 4.0 vs 3.8, I'm fine with 4.0. But I do not know if
>> >> "users" will expect a bigger change in functionalities for a mayor
>> >> release update.
>> >
>> > Yes, there are a few other tickets flagged as appropriate for 4.0:
>> > https://github.com/OSGeo/gdal/milestone/33
>> >
>> > All of them could probably be implemented without making the 3.8/4.0
>> > schedule drift. The exportToWKT with WKT2 as default would involve
>> > some work in GDAL drivers, given that some of them are dependent on
>> > WKT1, but hopefully nothing that cannot be overcome. The main impact
>> > would probably be on the autotest suite (fast resolution would be
>> > similar to drivers, and replace all exportToWKT() with
>> > exportToWKT(["FORMAT=WKT1"]), and possibly switch progressively to WKT2)
>> >
>> > Other topics that could/should be split for a 4.0 ?
>> >
>> > Thinking about CRS stuff, currently gdaltransform operates with the
>> > GIS friendly axis order, which is at odds with the fact that
>> > OGRSpatialReference default and PROJ's cs2cs which use the authority
>> > compliant axis order since PROJ 6.0 / GDAL 3.0. Not sure if we'd want
>> > to make gdaltransform follow cs2cs (possibly with a
>> > --axis-order=gis_friendly/authority_compliant explicit flag)
>> >
>> > Maybe some 'Ex' (which stands for API) functions in the C API could be
>> > removed and their extra/modified arguments reincorporated with the
>> > original non-Ex function. Would totally make sense for the few ones
>> > impacted by RFC95 like GDALGetDefaultHistogramEx,
>> > GDALSetDefaultHistogramEx, GDALGetRasterHistogramEx
>> >
>> > There might be also some defaults (open, creation options) that could
>> > be changed, although nothing immediately jumps to mind
>> >
>> > Even
>> >
>> >
>>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: add Javier Jiménez Shaw to GDAL PSC

[Sean Gillies]

Welcome aboard, Javier! I'm looking forward to working with you.

On Wed, Sep 20, 2023 at 4:08 AM Javier Jimenez Shaw 
wrote:

> Thank you! It is an honour being part of the PSC. :D
>
> On Wed, 20 Sept 2023 at 10:47, Even Rouault 
> wrote:
>
>> Motion passed with +1 from PSC members MateuszL, HowardB, JukkaR, KurtS,
>> FrankW, DanielM and me.
>>
>> Congrats Javier! You can file a PR to add yourself to
>> https://gdal.org/community/index.html#psc
>>
>> Le 16/09/2023 à 13:31, Even Rouault a écrit :
>> > Hi,
>> >
>> > I would like to nominate Javier Jiménez Shaw for GDAL PSC membership.
>> > Javier has been involved with GDAL for quite a time now, as a
>> > responsive user & ticket reporter, and has occasionally contributed
>> > fixes. Javier is well anchored in our ecosystem, with deep knowledge
>> > of CRS topics and PROJ (I've also nominated him for PROJ membership),
>> > contributing to PDAL as well. As part of his job, he's involved in
>> > orthomosaic/raster generation out of images and photogrammetry
>> > pipelines. His perspective would be most welcome.
>> >
>> > Starting with my +1
>> >
>> > Even
>> >
>>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for discussion/review of RFC95: Use standard C/C++ integer types

[Sean Gillies]

Hi Even,

I'm in favor of overhauling the types in the next major version. However,
I'm not convinced we need to jump immediately to 4.0. The current situation
isn't ideal, but isn't holding us back very much right? GDAL is growing new
features rapidly and it doesn't seem that the old types have been a blocker
or huge friction for core developers.

Speaking for Fiona and Rasterio, supporting GDAL versions 3.5-3.7 and 4.0
at the same time will be a pain and will require some conditional
compilation changes throughout the code. These projects will need some time
to prepare. And I'd be more enthusiastic about supporting 3.7 and 4.0
simultaneously if 4.0 made GDAL's C API tangibly better, like dataset
unification in 2.0 or the PROJ switch in 3.0.

On Tue, Sep 19, 2023 at 10:30 AM Even Rouault 
wrote:

> No other reaction ? Are people comfortable with bumping to 4.0 ? If so,
> no opinion on what we should slip in while we are it (thinking more
> about breaking changes than new features, which generally can be added
> afterwards) ?
>
> Le 16/09/2023 à 14:53, Even Rouault a écrit :
> >
> >> About GDAL 4.0 vs 3.8, I'm fine with 4.0. But I do not know if
> >> "users" will expect a bigger change in functionalities for a mayor
> >> release update.
> >
> > Yes, there are a few other tickets flagged as appropriate for 4.0:
> > https://github.com/OSGeo/gdal/milestone/33
> >
> > All of them could probably be implemented without making the 3.8/4.0
> > schedule drift. The exportToWKT with WKT2 as default would involve
> > some work in GDAL drivers, given that some of them are dependent on
> > WKT1, but hopefully nothing that cannot be overcome. The main impact
> > would probably be on the autotest suite (fast resolution would be
> > similar to drivers, and replace all exportToWKT() with
> > exportToWKT(["FORMAT=WKT1"]), and possibly switch progressively to WKT2)
> >
> > Other topics that could/should be split for a 4.0 ?
> >
> > Thinking about CRS stuff, currently gdaltransform operates with the
> > GIS friendly axis order, which is at odds with the fact that
> > OGRSpatialReference default and PROJ's cs2cs which use the authority
> > compliant axis order since PROJ 6.0 / GDAL 3.0. Not sure if we'd want
> > to make gdaltransform follow cs2cs (possibly with a
> > --axis-order=gis_friendly/authority_compliant explicit flag)
> >
> > Maybe some 'Ex' (which stands for API) functions in the C API could be
> > removed and their extra/modified arguments reincorporated with the
> > original non-Ex function. Would totally make sense for the few ones
> > impacted by RFC95 like GDALGetDefaultHistogramEx,
> > GDALSetDefaultHistogramEx, GDALGetRasterHistogramEx
> >
> > There might be also some defaults (open, creation options) that could
> > be changed, although nothing immediately jumps to mind
> >
> > Even
> >
> >
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Formalizing GDAL "file name" syntax in an RFC?

[Sean Gillies]

Hi Alessandro,

On Fri, Sep 15, 2023 at 8:31 AM ElPaso  wrote:

> Il 14/09/23 17:14, Sean Gillies ha scritto:
> > Hi all,
> >
> > In https://github.com/OSGeo/gdal/pull/8155#issuecomment-1704923263 I
> > think I see (for the first time?) the beginning of a specification of
> > the syntax for GDAL "file names". I think it would be helpful if there
> > was an RFC for this.
> >
> > I'm sure a lot of applications construct GDAL file names without much
> > understanding of what's correct or incorrect. A formal spec could help
> > make it more likely that anyone can construct a valid filename on the
> > first try.
> >
> > A stretch goal for the RFC could be to come up with a syntax that is
> > sufficiently general that authors of new format drivers don't have to
> > create their own new idiosyncratic file names.
> >
>
> Hi Sean,
>
>
> I totally agree that it would be useful to have a formally defined
> syntax to describe data sources, we have the same problem with QGIS
> QgsDataSourceUri and we are treating the URIs as a private
> implementation detail, but in QGIS we have a GUI to set the data source
> strings that partially mitigatest the issue.
>
>
> We discussed a few times what it could be the best format to encode a
> data source and URL encoding is probably a good candidate because it's
> well known and well supported by libraries.
>
>
> Maybe something like:
>
>
> :///
>
> mssql://username:password@hostname:port/?table=table_name=arg1...
>
> gpkg:///path/to.gpkg?table=table_name=arg1...
>
> postgis://username:password@hostname:port/?table=table_name=arg1...
>
> shapefile:///path/to.shp/?arg1=arg1...
>

Yes, something like this is my dream :) I started working on such a
proposal a while back and then was derailed by a job change. I might try to
dust it off and start writing on it again.

I think the "vrt://" URI scheme was a mistake, though, and that we
shouldn't extend it to dozens of other formats. At least not without
consulting with other software communities. Ideally these schemes would be
universal across the internet.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Formalizing GDAL "file name" syntax in an RFC?

[Sean Gillies]

Even,

I agree that there is an untidy legacy. And it's probably not worth
formalizing the marginal filenames and connection strings. How about an RFC
and formal syntax only for hierarchical datasets then? It seems like this
is the direction the industry is growing.

Yes, I think a formal syntax for /vsi filenames would be useful. It's
almost done already, right? Only in prose, not ABNF (or whatever), but
that's a good start.

On Thu, Sep 14, 2023 at 10:21 AM Even Rouault 
wrote:

> Sean,
>
> It is far from obvious to me to find a universal pattern that would fit
> existing and future use cases. If we would find a universal syntax, there
> would be the problem of deciding what to do with code that doesn't match
> it: support only new way (breaking external code),  or supporting both old
> and new ways (additional complexity in our code base)
>
> How would we deal with the existing practices which are very diverse ?
> Examples:
>
> - True filenames (most of the drivers)
>
> - Database based drivers:
>
> PG:"dbname='databasename' host='addr' port='5432' user='x' password='y'"   
> (Postgis vector)PG:"[host=''] [port=''] [dbname='' [user=''] [password=''] 
> [schema=''] [table=''] [column=''] [where=''] [mode=''] 
> [outdb_resolution='']"  (Postgis raster, consistent with previous one)
> MYSQL:dbname,user="userid",password="password",host=x,port=y
> OCI:username/password@host_name:port_number/service_name:MY_SCHEMA.MY_VIEWgeoraster:{,/}{,@}[db],[schema.][table],[column],[where]
> georaster:{,/}{,@}[db],,
>
> - Web services (some of them also accept plain filenames that are .xml
> files describing the service and parameters)
>
> WMS:http://demo.opengeo.org/geoserver/gwc/service/wms?SERVICE=WMS=1.1.1;
> 
> REQUEST=GetMap=og%3Abugsites=EPSG:900913&
> 
> BBOX=-1.15841845090625E7,5479006.186718751,-1.1505912992109375E7,5557277.703671876&
> 
> FORMAT=image/png=256=25=0.0046653459640220=true"
> (note the mix of pure WMS GetMap request query parameters + GDAL specific 
> properties TILESIZE, OVERVIEWCOUNT, MINRESOLUTION)
>
> WCS:http://194.66.252.155/cgi-bin/BGS_EMODnet_bathymetry/ows?VERSION=1.1.0=BGS_EMODNET_CentralMed-MColWFS:http://www2.dmsolutions.ca/cgi-bin/mswfs_gmapOAPIF:https://www.ldproxy.nrw.de/rest/services/katasterWMTS:http://maps.wien.gv.at/wmts/1.0.0/WMTSCapabilities.xml,layer=lb
> ( layer= is GDAL specific)
>
>
> - Raster subdatasets (whether to surround filename with double quotes is
> driver specific)
>
> NITF_IM:{image_number}:{filename}
>
> NITF_TOC_ENTRY:{product_type}_{chartname}_{scale}_{somenumber}_{anothernumber}:GNCJNCN/rpf/a.toc
> ECRG_TOC_ENTRY:{product_name}:{disk_name}:{scale}:{filename}
> GTIFF:{directory_number}:{filename}
> GTIFF:off:{directory_offset}:{filename}
> GPKG:{filename}:{tablename}
> netCDF:{filename}:{variable_name}
> HDF5:{filename}:{dataset_path}
> ZARR:{filename}  (mostly when filename is /vsicurl/ and we don't have a
> ReadDirectory() API)
> ZARR:{filename}:{path_to_array}
>
> ZARR:{filename}:{path_to_array}:{third_dim_index}[:{fourth_dim_index}[:{fifth_dim_index}...]]
>
> - Other syntax
>
> vrt:// connection string (
> https://gdal.org/drivers/raster/vrt.html#vrt-connection-string) . e.g.
> vrt://my.tif?bands=3,2,1
>
> Do you have also /vsi syntax in mind ?
>
> Even
>
> Le 14/09/2023 à 17:14, Sean Gillies a écrit :
>
> Hi all,
>
> In https://github.com/OSGeo/gdal/pull/8155#issuecomment-1704923263 I
> think I see (for the first time?) the beginning of a specification of the
> syntax for GDAL "file names". I think it would be helpful if there was an
> RFC for this.
>
> I'm sure a lot of applications construct GDAL file names without much
> understanding of what's correct or incorrect. A formal spec could help make
> it more likely that anyone can construct a valid filename on the first try.
>
> A stretch goal for the RFC could be to come up with a syntax that is
> sufficiently general that authors of new format drivers don't have to
> create their own new idiosyncratic file names.
>
> --
> Sean Gillies
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Formalizing GDAL "file name" syntax in an RFC?

[Sean Gillies]

Hi all,

In https://github.com/OSGeo/gdal/pull/8155#issuecomment-1704923263 I think
I see (for the first time?) the beginning of a specification of the syntax
for GDAL "file names". I think it would be helpful if there was an RFC for
this.

I'm sure a lot of applications construct GDAL file names without much
understanding of what's correct or incorrect. A formal spec could help make
it more likely that anyone can construct a valid filename on the first try.

A stretch goal for the RFC could be to come up with a syntax that is
sufficiently general that authors of new format drivers don't have to
create their own new idiosyncratic file names.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Grant Dan Baston Merge Rights

[Sean Gillies]

Enthusiastic +1 from me

On Tue, Sep 12, 2023, 1:22 PM Howard Butler  wrote:

> Dear PSC,
>
> Dan Baston is an active GDAL maintainer who does not currently have merge
> rights. Let's fix that so he can get more work done without waiting on
> someone to merge things up :)
>
> /me starts with +1
>
> Howard
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Improving details of the project build system and/or documentation

[Sean Gillies]

On Sun, May 28, 2023 at 2:14 PM Even Rouault 
wrote:

> Hi Sean,
>
>
> Using the CMake files as a source of truth helps. But they don't describe
> everything. For example, they don't explain that GDAL needs libhdf5 to be
> built with special features. That one stumped me for a while. I was seeing
> a build message like "HDF5 not detected (found version 1.12.1)". I finally
> found the clue in the vcpkg repo.
>
> ==> https://gdal.org/development/building_from_source.html#hdf5: "The
> HDF5 CXX library is needed for the KEA
> <https://gdal.org/drivers/raster/kea.html#raster-kea> driver."
>
> That said, it isn't obvious to realize that detection of HDF5 fails
> because of that. That's a bit of a downside with CMake Find modules
> whose output isn't always very helpful.
>
> Actually, I've changed the logic in
> https://github.com/OSGeo/gdal/pull/7851 to only check for HDF5 CXX if we
> have found libkea before. That reduces the requirements to what is really
> needed.
>
Thank you!

>
> Also, the CMake files don't explain that the GDAL MBTiles driver depends
> on the OGR MVT driver.
>
> Also fixed per https://github.com/OSGeo/gdal/pull/7851.  It is easy to
> miss capturing such dependencies to a driver that is built-in by default
>
> Even
>
Is MVT really a default driver? In
https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/CMakeLists.txt#L80
it looks like it has library and driver dependencies. And anyway, its
dependencies were met in my build script. It should have been discovered if
it was a default driver?

Testing build systems is tough. I know GDAL can't test that all optional
drivers can be configured and built in isolation. At the same time,
profiles of drivers between the bare minimum and
everything-but-the-kitchen-sink are useful to some communities of users,
like the projects that would like to "pip install rasterio" in their own
builds. I resented being told on this list (not by you, Even) that I could
take it or leave it with regards to the current build situation. For my
part, I can do a better job of trying rasterio wheel builds against the
upcoming versions of GDAL, and I will do so.


>
>
> On Mon, May 22, 2023 at 8:53 AM Even Rouault 
> wrote:
>
>> Hi Sean,
>>
>> I presume you got this because you defined OGR_BUILD_OPTIONAL_DRIVERS=OFF
>> which then cause OGR_ENABLE_DRIVER_AVC to be set to OFF.
>>
>> We already a quite overwhelming amount of documentation in
>> https://gdal.org/development/building_from_source.html about all the
>> existing variables, and I'm not sure adding an exhaustive list of all the
>> inter driver dependencies will be user-digestable and could rot easily.
>>
>> That said in https://github.com/OSGeo/gdal/pull/7806 I've tried to
>> improve the current error message with a hint for the reason for the error
>> and the (likely) reason for it to happen.
>>
>> I've also added pointers in the doc page to the CMakeLists.txt files
>> where the dependencies are expressed. Hopefully people who are in the
>> business of making custom GDAL builds can make sense of that.
>>
>> Even
>> Le 20/05/2023 à 01:26, Sean Gillies a écrit :
>>
>> Hi all,
>>
>> I really appreciate the documentation at
>> https://gdal.org/development/building_from_source.html. But there are
>> gaps. For example, today I ran into
>>
>> CMake Error at cmake/helpers/GdalDriverHelper.cmake:331 (message):
>> GDAL_ENABLE_DRIVER_AIGRID cannot be enabled because condition
>> OGR_ENABLE_DRIVER_AVC is not met. To ignore this error (but the driver
>> will not be built), set the GDAL_IGNORE_FAILED_CONDITIONS variable
>>
>> This dependence isn't documented. It's a bit frustrating to work through
>> missing directives one at a time when adding new drivers to my build.
>>
>> Is it possible for a driver's dependencies to be enabled when I enable a
>> driver?
>>
>> If not, can we consider using GDAL maintenance funding and resources on
>> documenting the heck out of this system?
>>
>>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Improving details of the project build system and/or documentation

[Sean Gillies]

Hi Even,

I took better notes during my latest upgrade and wrote them down here:
https://github.com/rasterio/rasterio-wheels/pull/106#issue-1720809259.

Using the CMake files as a source of truth helps. But they don't describe
everything. For example, they don't explain that GDAL needs libhdf5 to be
built with special features. That one stumped me for a while. I was seeing
a build message like "HDF5 not detected (found version 1.12.1)". I finally
found the clue in the vcpkg repo.

Also, the CMake files don't explain that the GDAL MBTiles driver depends on
the OGR MVT driver.


On Mon, May 22, 2023 at 8:53 AM Even Rouault 
wrote:

> Hi Sean,
>
> I presume you got this because you defined OGR_BUILD_OPTIONAL_DRIVERS=OFF
> which then cause OGR_ENABLE_DRIVER_AVC to be set to OFF.
>
> We already a quite overwhelming amount of documentation in
> https://gdal.org/development/building_from_source.html about all the
> existing variables, and I'm not sure adding an exhaustive list of all the
> inter driver dependencies will be user-digestable and could rot easily.
>
> That said in https://github.com/OSGeo/gdal/pull/7806 I've tried to
> improve the current error message with a hint for the reason for the error
> and the (likely) reason for it to happen.
>
> I've also added pointers in the doc page to the CMakeLists.txt files where
> the dependencies are expressed. Hopefully people who are in the business of
> making custom GDAL builds can make sense of that.
>
> Even
> Le 20/05/2023 à 01:26, Sean Gillies a écrit :
>
> Hi all,
>
> I really appreciate the documentation at
> https://gdal.org/development/building_from_source.html. But there are
> gaps. For example, today I ran into
>
> CMake Error at cmake/helpers/GdalDriverHelper.cmake:331 (message):
> GDAL_ENABLE_DRIVER_AIGRID cannot be enabled because condition
> OGR_ENABLE_DRIVER_AVC is not met. To ignore this error (but the driver
> will not be built), set the GDAL_IGNORE_FAILED_CONDITIONS variable
>
> This dependence isn't documented. It's a bit frustrating to work through
> missing directives one at a time when adding new drivers to my build.
>
> Is it possible for a driver's dependencies to be enabled when I enable a
> driver?
>
> If not, can we consider using GDAL maintenance funding and resources on
> documenting the heck out of this system?
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Improving details of the project build system and/or documentation

[Sean Gillies]

Hi all,

I really appreciate the documentation at
https://gdal.org/development/building_from_source.html. But there are gaps.
For example, today I ran into

CMake Error at cmake/helpers/GdalDriverHelper.cmake:331 (message):
GDAL_ENABLE_DRIVER_AIGRID cannot be enabled because condition
OGR_ENABLE_DRIVER_AVC is not met. To ignore this error (but the driver
will not be built), set the GDAL_IGNORE_FAILED_CONDITIONS variable

This dependence isn't documented. It's a bit frustrating to work through
missing directives one at a time when adding new drivers to my build.

Is it possible for a driver's dependencies to be enabled when I enable a
driver?

If not, can we consider using GDAL maintenance funding and resources on
documenting the heck out of this system?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Intermittent failure with Python mocks

[Sean Gillies]

Hi Jon,

The number one gotcha with patching is documented here
https://docs.python.org/3/library/unittest.mock.html#where-to-patch. In a
nutshell, you have to patch your own module. I see you're patching gdal in
your tests. Patching gdal from your test doesn't do anything, it's too
late.

Hope this helps!

On Thu, Apr 13, 2023, 7:53 AM Jon Morris  wrote:

> I’m getting intermittent fails with our Python unit tests and I wondered
> if anyone has seen the same issue. I think it’s a race condition where a
> mock patch is either not applied at all or maybe applied too slowly so the
> patched function still has its original return value. It might be a pytest
> issue, but I’ve now seen it with two different GDAL methods so I’m just
> curious if anyone has any idea what may be going on.
>
>
>
> The first example I reported on Stack Overflow (
> https://stackoverflow.com/q/75185263/3182496) and looks something like
> this:
>
>
>
> mock_feat_defn = mocker.patch.object(ogr.FeatureDefn,
> 'GetFieldCount')
>
> mock_feat_defn.side_effect = RuntimeError("Some other error
> message")
>
>
>
> with pytest.raises(RuntimeError, match="Some other error message"):
>
> get_from_layer_schema(test_lyr, get_names=True,
> get_types=False)
>
>
>
> The second example is this:
>
>
>
> with mock.patch('osgeo.gdal.Driver.Delete', side_effect=None):
>
> with self.assertRaisesRegex(DatasetError, "Could not delete"):
>
> delete_path(shp_path)
>
>
>
> In both cases we’re mocking a GDAL method call to check one of our
> functions correctly raises an error, but sometimes it works and sometimes
> it doesn’t. If it’s not a GDAL issue, no problem, I will keep looking
> elsewhere for a solution.
>
>
>
> Jon
> e: ​ *jon.mor...@jbarisk.com* 
> d: +44 (0)1756 587229
> t:  *+44 (0)1756 799919* <+44%20(0)1756%20799919>
> *www.jbarisk.com* 
> [image: Facebook] 
> [image: LinkedIn] 
> [image: Twitter] 
> [image: YouTube]
> 
>
>
> All JBA Risk Management's email messages contain confidential information
> and are intended only for the individual(s) named. If you are not the named
> addressee you should not disseminate, distribute or copy this e-mail.
> Please notify the sender immediately by email if you have received this
> email by mistake and delete this email from your system.
> JBA Risk Management Limited is registered in England, company number
> 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North
> Yorkshire, BD23 3FD, England.
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Report on GEOS maintenance grant

[Sean Gillies]

Thank you for the write up, Dan! I appreciate what you've done for the
project and community. The outcome of the grant is beyond my expectations,
for sure.

On Mon, Feb 27, 2023 at 7:35 AM Daniel Baston  wrote:

> Hi All,
>
> As February comes to a close, I’m winding down the GEOS maintenance work
> that the GDAL PSC funded in 2022 [1]. I appreciate the opportunity to do
> this work and am especially thankful to those who reported issues, reviewed
> pull requests, or participated in discussions about this work.
>
> For anyone who’s interested, I wanted to share a few of the highlights
> from my perspective.
>
> New functionality: GEOS now supports reading, writing, and overlay of
> geometries with an M dimension. I added a number of C API utility functions
> requested by client library developers: GEOSLineSubstring,
> GEOSEqualsIdentical, and thread/context-specific interrupt handlers. I also
> brought in clustering algorithms such as DBSCAN that were previously only
> available in PostGIS.
>
> Performance enhancements: I found some nice performance wins in key GEOS
> algorithms such as geometry intersection testing (GEOSIntersects and
> GEOSPreparedIntersects), distance (GEOSPreparedDistance and
> GEOSPreparedDistanceWithin) and buffering complex geometries. There are
> also some gains in core internal classes such as CoordinateSequence and
> STRtree that deliver across-the-board gains.
>
> Project maintenance. I was able to simplify our CI setup by removing the
> use of Azure and AppVeyor services and bringing the associated environments
> into GitHub Actions, write a harness for performance testing and
> visualization, and perform code cleanup such as migrating from raw to
> managed pointers in key classes. I resolved a number of long-standing bug
> reports with WKT input/output, geometry simplification, and
> crashes/incorrect results from multiple algorithms in edge cases such as
> empty geometry inputs.
>
> Not everything panned out, or at least not yet. Something we’ve wanted in
> GEOS for a long time is the ability to create a GEOS geometry directly from
> an external memory buffer, such as a PostGIS LWGEOM or a GDAL OGRGeometry
> (“zero-copy”). I added experimental support for this, but so far I
> haven’t found a case where it offers much performance benefit.
>
> Dan
>
> [1] https://lists.osgeo.org/pipermail/gdal-dev/2022-February/055433.html
> _______
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] More consistent use of pytest's parameterization in autotests

[Sean Gillies]

I thought about writing something down, too, but didn't see anything about
writing tests at https://gdal.org/development/testing.html and I decided I
wasn't qualified to start a new section about testing standards.

On Mon, Feb 20, 2023, 10:52 AM Even Rouault 
wrote:

> Hi Sean,
>
> I fully agree that pytest.mark.parametrize() is cleaner and the way to go,
> and I use it extensively in new tests. For what you were referring too,
> this was a change in an existing test that used the old for looping habits,
> so I felt it was a bit too much to ask the contributor to refactor it.
>
> I've tried to enhance the existing documentation page related to tests
> with recommendations from this email thread and other things from related
> RFCs: https://github.com/OSGeo/gdal/pull/7277
>
> Even
> Le 20/02/2023 à 16:42, Sean Gillies a écrit :
>
> Hi all,
>
> I just saw a maintainer recommend to a contributor that the contributor
> loop over test cases from within a test function and it prompted me to
> speak up about a better practice: using pytest parameterization
> https://docs.pytest.org/en/6.2.x/parametrize.html#pytest-mark-parametrize-parametrizing-test-functions
> .
>
> Using a loop, like at
> https://github.com/OSGeo/gdal/blob/master/autotest/gcore/gdal_stats.py#L660,
> can hide bugs and add friction to development. In that particular case,
> there are 5 different fill values to be checked. If there are different
> code paths for each of these values (a worst case scenario) and each has a
> small bug, you will have to run the whole test suite 5 times to expose the
> 5 bugs one at a time. Additionally, these loops fix the order that the
> checks are made, and bugs can hide in test order dependency. Ideally,
> GDAL's tests run in random order.
>
> In a nutshell, you hoist the loop and list out into a test function
> decorator and then pytest discovers and runs all the separate cases. Pytest
> doesn't stop on the first failure in the list unless you explicitly tell it
> to do so.
>
> # Original GDAL test code
> def test_fill_value():
> for fill_val in [0, 1, 32767, 32768, 65535]:
> func(fill_val)
> assert something
>
> # With pytest parameterization
> @pytest.mark.parametrize("fill_val", [0, 1, 32767, 32768, 65535])
> def test_fill_value(fill_val):
> func(fill_val)
> assert something
>
> I'm not proposing rewrites of old tests, but I think it is worth adopting
> a better practice for tests now and in the future.
>
> --
> Sean Gillies
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] More consistent use of pytest's parameterization in autotests

[Sean Gillies]

Hi all,

I just saw a maintainer recommend to a contributor that the contributor
loop over test cases from within a test function and it prompted me to
speak up about a better practice: using pytest parameterization
https://docs.pytest.org/en/6.2.x/parametrize.html#pytest-mark-parametrize-parametrizing-test-functions
.

Using a loop, like at
https://github.com/OSGeo/gdal/blob/master/autotest/gcore/gdal_stats.py#L660,
can hide bugs and add friction to development. In that particular case,
there are 5 different fill values to be checked. If there are different
code paths for each of these values (a worst case scenario) and each has a
small bug, you will have to run the whole test suite 5 times to expose the
5 bugs one at a time. Additionally, these loops fix the order that the
checks are made, and bugs can hide in test order dependency. Ideally,
GDAL's tests run in random order.

In a nutshell, you hoist the loop and list out into a test function
decorator and then pytest discovers and runs all the separate cases. Pytest
doesn't stop on the first failure in the list unless you explicitly tell it
to do so.

# Original GDAL test code
def test_fill_value():
for fill_val in [0, 1, 32767, 32768, 65535]:
func(fill_val)
assert something

# With pytest parameterization
@pytest.mark.parametrize("fill_val", [0, 1, 32767, 32768, 65535])
def test_fill_value(fill_val):
func(fill_val)
assert something

I'm not proposing rewrites of old tests, but I think it is worth adopting a
better practice for tests now and in the future.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for discussion on RFC 92 text: WKB Only geometries

[Sean Gillies]

Hi Even,

I'm glad you mentioned a shift of concepts. The older Feature API and new
Arrow API are really quite different. I actually don't know how often I
will ever combine them in one program, for example to turn shapefiles into
a huge Parquet file or the reverse. If anybody is doing this kind of thing
I'd be interested to read about it.

This RFC seems to have some things in common with the direct access to
compressed raster blocks. Both by-pass the older raster and vector data
models, yes? And they're quite specific to certain formats. Do you
anticipate that GDAL and OGR will continue to develop more specialty APIs
in addition to the general ones?

On Sat, Feb 4, 2023, 11:55 AM Even Rouault 
wrote:

> Hi Sean,
>
> but wouldn't it be possible for all OGRFeatures to carry WKB data by
> default and add a method to provide it to callers?
>
> My understanding of what you propose would involve massive code rewrites
> in all drivers and wouldn't be desirable from a performance point of view,
> because most drivers can't generate WKB easily (PostGIS and GPKG are the
> exceptions rather the norm). So either all other drivers should be modified
> to compose WKB at hand (massive coding effort. Probably several weeks of
> effort and significant risk of regressions). Or get it from the
> ExportToWkb() method of the OGRGeometry instance they currently build, but
> then you pay the price in memory and CPU time to generate WKB that might
> not be consumed by users.
>
> | And only construct an OGRGeometry when it's asked for? Such as when
> GetGeometryRef is called?
>
> Good point, we could both make GetGeometryRef() and GetGeomFieldRef()
> virtual methods whose default implementation would be the same as
> currently, ie. return the value of the corresponding member variable in the
> base OGRFeature class stored with
> SetGeometry[Directly]()/SetGeomField[Directly]()
>
> And add a new virtual method:
>
> virtual GByte* OGRFeature::GetWKBGeometry(int iGeomField, size_t*
> pnOutSize) const
>
> whose default implementation would just use
> GetGeomFieldRef(iGeomField)->ExportToWkb().
>
> The few drivers that can provide a more efficient implementation (GPKG
> typically) would create a derived class OGRFeatureGPKG with a specific
> implementation of those new virtual methods to avoid systematic OGRGeometry
> instantiation. The only drawback I see is that making GetGeometryRef() and
> GetGeomFieldRef() virtual would have a slight performance impact, but
> probably small enough.
>
>
> But fundamentally I'm wondering if RFC 92 hasn't been made mostly out
> fashioned now that we have RFC 86. RFC 86 generally leads to 2x speed-up or
> more on real-world datasets compared to OGRFeature iteration (as measured
> by the bench_ogr_c_api vs bench_ogr_batch utilities) on drivers that have
> implemented it (currently Arrow, Parquet, FlatGeoBuf, GPKG), whereas RFC 92
> only applies to GPKG & PostGIS and in the best - artificial - case only
> lead to 30% speed-up.
>
> Of course, adopting RFC 86 requires significant effort from GDAL users,
> but the benefit is really measurable whereas with RFC 92 it would be
> marginal in most scenarios. As far as I can tell, the performance boost of
> RFC 86 comes mostly from saving creation & destruction of millions of
> OGRFeature instances, its array members, string attributes, geometries
> objects, more than the columnar organization of the ArrowArray data
> structures. In the GeoPackage driver, I've also shown that it makes it
> possible for efficient multi-threading pre-fetching, totally transparent
> for the user.
>
> But to avoid selling false hopes, the benefit of RFC 86 in end-to-end
> scenarios would probably drop significantly (at least if looking at
> performance gain in percentage. The absolute performance savings on the
> GDAL side would remain) if you need to recreate individual features (QGIS'
> QgsFeature or MapServer' msShape objects) from the content of ArrowArray.
> So this is likely a complete shift of concepts that would be required.
>
> Even
>
>
>
> On Tue, Jan 31, 2023 at 4:27 AM Even Rouault 
> wrote:
>
>> Hi,
>>
>> Please find for review "RFC 92 text: WKB Only geometries" at
>> https://github.com/OSGeo/gdal/pull/7149
>>
>> This RFC provides shortcuts to avoid instantiation of full OGRGeometry
>> instances
>> in scenarios where only the WKB representation of geometries is needed.
>> The
>> hope is to save CPU time.
>>
>> This is something I wanted to at least experiment. I've mixed feelings
>> if it's something we actually want to adopt.
>>
>> Even
>>
>> --
>> http://www.spatialys.com
>> My software is free, but my time ge

Re: [gdal-dev] Call for discussion on RFC 92 text: WKB Only geometries

[Sean Gillies]

Hi Even,

I'm not much of a C++ programmer, but wouldn't it be possible for all
OGRFeatures to carry WKB data by default and add a method to provide it to
callers? And only construct an OGRGeometry when it's asked for? Such as
when GetGeometryRef is called?

On Tue, Jan 31, 2023 at 4:27 AM Even Rouault 
wrote:

> Hi,
>
> Please find for review "RFC 92 text: WKB Only geometries" at
> https://github.com/OSGeo/gdal/pull/7149
>
> This RFC provides shortcuts to avoid instantiation of full OGRGeometry
> instances
> in scenarios where only the WKB representation of geometries is needed. The
> hope is to save CPU time.
>
> This is something I wanted to at least experiment. I've mixed feelings
> if it's something we actually want to adopt.
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review on RFC 91: GDALDataset::Close() method

[Sean Gillies]

I'm excited!

On Fri, Jan 20, 2023 at 4:23 PM Kurt Schwehr  wrote:

> While it might not be exciting, I think this is a solid API improvement.
>
> On Fri, Jan 20, 2023 at 11:11 AM Even Rouault 
> wrote:
>
>> Hi,
>>
>> Just to notify you that "RFC 91: GDALDataset::Close() method" is
>> available for review in https://github.com/OSGeo/gdal/pull/7108
>>
>> Admittedly, nothing particularly exciting, but this should provide GDAL
>> users with better error detection.
>>
>> Even
>>
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Approve RFC90: Direct access to compressed raster data

[Sean Gillies]

I'm +1. Norman Barker wrote something like this when we worked together, so
I get the point of it clearly.

https://www.rfc-editor.org/rfc/rfc6838#page-13 has some things to say about
the parameters returned by GetCompressionFormat after, e.g., "image/jpeg".

* Parameters are case insensitive
* Order has no meaning
* More than one instance of a parameter is an error

It's possible that some image media types are defined so that no such
parameters are allowed. I don't see that this is the case for image/jpeg,
so we're probably fine.

If we don't want to think about official media types at all, which might be
a good idea, couldn't we use driver names instead? "JPEG; foo=bar" instead
of "image/jpeg; foo=bar" for example.


On Mon, Jan 16, 2023 at 10:03 AM Even Rouault 
wrote:

> Hi,
>
> Motion: Approve RFC90: Direct access to compressed raster data:
>
> https://github.com/OSGeo/gdal/pull/7020
>
> Starting with my +1
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?

[Sean Gillies]

No objection from me!


On Tue, Dec 13, 2022, 8:19 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> https://github.com/qgis/QGIS/issues/51188 has been brought to my
> attention. The issue is that the new background building of the RTree of
> GeoPackage files introduced in 3.6.0 didn't work well with committing
> transactions in between, which is easily triggered by ogr2ogr. All
> features were inserted but with default settings ~ 10% of them lacked an
> entry in the spatial index, which can be enough to break interactive
> display and workflows relying on spatial filtering. When using ogr2ogr,
> I believe the issue can only be seen when creating layers with more than
> 100 000 features, since that's the default value for the interval at
> which transactions are committed.
>
> I've a fix ready in https://github.com/OSGeo/gdal/pull/6911. The fix
> itself is a simple two-liner:
>
> https://github.com/OSGeo/gdal/pull/6911/commits/3f5f6225fe82e0c2e0241e4f66bfb861cdf4fe9d
>
> Given the status of GeoPackage being the default format for QGIS, I
> believe this is a severe enough issue to warrant an advanced 3.6.1
> release, and an official retraction of 3.6.0.
>
> Thoughts ?
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [OSGeo/gdal] Add GDT_Int8 support (RFC 87) (PR #6633)

[Sean Gillies]

Even,

Argh, I typed "unsigned char" when I meant "signed". Changing GDT_Byte to
*signed char* is too big of a change, I guess?

I like the plan for PIXELTYPE.

Sorry about the noise, everybody!

On Tue, Nov 8, 2022 at 5:04 PM Even Rouault 
wrote:

> Sean,
>
>
> Changing GDT_Byte to unsigned char is too big of a change, I guess? I can
> work with that.
>
> GDT_Byte semantic is already unsigned char / uint8. What did you mean?
>
>
> Is there any advantage to a GDT_UInt8 type that can't be changed by a
> PIXELTYPE option?
>
> That would be super confusing if we had both GDT_Byte and GDT_UInt8 and
> they are not just simple aliases.
>
> The PIXELTYPE option should die. The RFC proposes to make it die on the
> reading side (as a metadata item).
>
> We could also make it die on the writing side as a creation option, but I
> didn't dare to do it right now and just propose this is considered a legacy
> deprecated way. Would certainly be something worth doing for a GDAL 4.0 if
> such thing ever happened.
>
>
> On Tue, Nov 8, 2022 at 12:39 PM Even Rouault 
> wrote:
>
>> Ubyte (same as uint8) vs byte?
>>
>> what do you suggest exactly: keep GDT_Byte in the enumeration and add #define
>> GDT_UByte GDT_Byte to create the alias ?
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/OSGeo/gdal/pull/6633#issuecomment-1307736762>, or
>> unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AAAIHIKG7X65SX4OULJMUFTWHKT5TANCNFSM6AAR2E6MJY>
>> .
>> You are receiving this because you commented.Message ID:
>> 
>>
>
>
> --
> Sean Gillies
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [OSGeo/gdal] Add GDT_Int8 support (RFC 87) (PR #6633)

[Sean Gillies]

Sorry, please disregard my previous email (from my phone at lunch). I had a
poor recollection of the GDALDataType enum.

Changing GDT_Byte to unsigned char is too big of a change, I guess? I can
work with that.

Is there any advantage to a GDT_UInt8 type that can't be changed by a
PIXELTYPE option?

On Tue, Nov 8, 2022 at 12:39 PM Even Rouault 
wrote:

> Ubyte (same as uint8) vs byte?
>
> what do you suggest exactly: keep GDT_Byte in the enumeration and add #define
> GDT_UByte GDT_Byte to create the alias ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/OSGeo/gdal/pull/6633#issuecomment-1307736762>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAIHIKG7X65SX4OULJMUFTWHKT5TANCNFSM6AAR2E6MJY>
> .
> You are receiving this because you commented.Message ID:
> 
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Computing statistics from raster dataset without writing aux.xml

[Sean Gillies]

Hi,

Another approach, also involving a copy, is to read the raster data as
numpy arrays and compute the statistics of those arrays. The GDAL method
also has to read the raster data to compute statistics, of course, but
doesn't store the entirety of it in memory.


On Mon, Oct 10, 2022, 6:51 AM Laurențiu Nicola via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi Mats,
>
> This won't help you, but I wanted to suggest you make a temporary
> directory, copy the .aux.xml file there, then set GDAL_PAM_PROXY_DIR.
> Unfortunately, at least in my test, GDAL still seems to save the file next
> to the input, even when calling SetConfigOption before doing anything
> else.
>
> Regards,
> Laurentiu
>
> On Mon, Oct 10, 2022, at 13:11, Budalen, Mats Bruun via gdal-dev wrote:
>
> Hello!
>
>
>
> I’m writing a Python program which reads raster data. The program uses the
> *ComputeStatistics()* function on the dataset, which automatically
> attempts to write the result to an .aux.xml next to the raster file. I do
> not under any circumstance want to try to write to the source directory, so
> I have set *gdal.SetConfigOption('GDAL_PAM_ENABLED','NO')* before I open
> the dataset.
>
>
>
> My program will sometimes come across datasets where the SRS is only
> defined in an aux.xml. Unfortunately, the above ConfigOption prevents the
> SRS definition from being read from the aux.xml.
>
>
>
> Is there any way to compute statistics without creating/changing the
> aux.xml, while at the same time reading the SRS?
>
>
>
>
>
> Regards,
>
>
>
> Mats Budalen
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Approve Even Rouault as a contracted GDAL maintainer for 2022-2023

[Sean Gillies]

+1

On Sat, Aug 27, 2022 at 6:43 AM Howard Butler  wrote:

> Dear PSC,
>
> It has come to my attention that Even's term as a GDAL Maintainer
> officially ended 31 JUL 2022. I propose that we extend him for another year
> from 01 AUG 2022 to 31 JUL 2023 under the previous terms and conditions.
>
> Although a small formality, we should make sure to keep a small bit of
> process around these things.
>
> /me starts with +1
>
> Howard
>
> PS, expect a GDAL Sponsorship Annual Report next week.
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL internal error handling and implications for other language bindings

[Sean Gillies]

d fail.
> Making such an assumption currently given the whole code base would be
> overly cautious.  Worth mentioning: the CPLTurnFailureIntoWarning()
> function (introduced in
> https://github.com/OSGeo/gdal/commit/f151e9f652b9d036cacdebf67ca88f59d5680cb3
> and from what I can see , it is pretty much only used at the 2 initial only
> places of gdal_misc.cpp of this commit), that as its name suggests can
> cause any CE_Failure to be turned into a CE_Warning. It could be handy to
> implement a "any CE_Failure that goes to the default or user-set error
> handling method means that the function failed" policy, but I'm not sure
> how we could implement this consistently in the code base. That would
> require analyzing all potential code paths where this could happen.
> Impractical (this also applies to the (*) at the beginning of this
> paragraph)
>
> (I guess we all agree that a successful function may emit one or several
> CPLError(CE_Warning, ...) during its execution)
>
I would prefer that no errors were emitted during a successful call, but
the next best thing is to have a better understanding of how the code
behaves now, and I feel like I have that. As long as no functions return a
successful error code while failing we're in pretty good shape!

> It's possible that some functions return a failed error code while not
> setting any error.
>
> Yes, sometimes error messages aren't really helpful when it is about
> corrupted data. e.g if you parse WKB geometries there are tons of potential
> error situations, and emitting a specific error for each one  would require
> a lot of coding effort, would bloat the code base & the size of the GDAL
> binary.
>
That makes sense.

Given all this, I think I'll modify my error recording approach to also use
the return values of functions. The combination of errors and return values
will be enough information to determine whether rasterio should raise a
Python exception.

Thanks!

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL internal error handling and implications for other language bindings

[Sean Gillies]

if Rasterio 1.4dev Interactive Inspector (Python
3.8.10)Type "src.meta", "src.read(1)", or "help(src)" for more
information.>>> src.read()rasterio._err.CPLE_AppDefinedError:
TIFFFillTile:Read error at row 512, col 0, tile 3; got 38232 bytes,
expected 47086
The above exception was the direct cause of the following exception:
rasterio._err.CPLE_AppDefinedError: TIFFReadEncodedTile() failed.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):  File "rasterio/_io.pyx", line 934,
in rasterio._io.DatasetReaderBase._readio_multi_band(self._hds, 0,
xoff, yoff, width, height, out, indexes_arr, resampling=resampling)
File "rasterio/_io.pyx", line 166, in rasterio._io.io_multi_band
with stack_errors():  File "/usr/lib/python3.8/contextlib.py", line
120, in __exit__next(self.gen)  File "rasterio/_err.pyx", line
245, in stack_errorsraise lastrasterio._err.CPLE_AppDefinedError:
/home/seangillies/projects/rasterio/tests/data/corrupt.tif, band 1:
IReadBlock failed at X offset 1, Y offset 1: TIFFReadEncodedTile()
failed.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):  File "", line 1, in
  File "rasterio/_io.pyx", line 610, in
rasterio._io.DatasetReaderBase.readout = self._read(indexes, out,
window, dtype, resampling=resampling)  File "rasterio/_io.pyx", line
937, in rasterio._io.DatasetReaderBase._readraise
RasterioIOError("Read or write failed. {}".format(cplerr)) from
cplerrrasterio.errors.RasterioIOError: Read or write failed.
/home/seangillies/projects/rasterio/tests/data/corrupt.tif, band 1:
IReadBlock failed at X offset 1, Y offset 1: TIFFReadEncodedTile()
failed.

I think this could make communication in the Rasterio issue tracker much
more productive. More information about the causes of an error is right
there in the traceback instead of being split between traceback and stderr
(or other log stream). It could at least eliminate one round of asking for
more error detail in a bug report. And there's the ability to catch an
exception and go up the chain in code, potentially very powerful when you
need it.

The effectiveness of this error recorder and chainer could depend on how
many different styles of error handling exist in GDAL. I've pointed out two
kinds above. In OGRXLSDataSource::Open, we have error handling that
actively prevents error events from being emitted until the function gives
up on trying to handle errors. In IReadBlock, there doesn't seem to be any
such error handling involving the GDAL error context. I believe we've seen
cases of GDAL functions that set errors while returning a success error
code. It's possible that some functions return a failed error code while
not setting any error. Lots of different cases could make the error
recording and chaining approach fruitless.

Are there other styles or paradigms in use? Are there GDAL modules that
will challenge the assumptions that I'm making as I write my error
recorder? If you know of any, I'd love to hear about them.

Here are a few links for reference:

* Exception handling in Python's C API:
https://docs.python.org/3/c-api/exceptions.html#exception-handling (I feel
like GDAL could use some documentation like this).
* Python exception chaining:
https://docs.python.org/3/tutorial/errors.html#exception-chaining
* On the difference between an exception raised while handling and "raise
from":
https://blog.ram.rachum.com/post/621791438475296768/improving-python-exception-chaining-with

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Fwd: Motion: adopt GDAL 3.5.0RC4 as final 3.5.0 release

[Sean Gillies]

+1

-- Forwarded message -
From: Sean Gillies 
Date: Thu, May 12, 2022 at 10:52 AM
Subject: Re: [gdal-dev] Motion: adopt GDAL 3.5.0RC4 as final 3.5.0 release
To: Even Rouault 


+1

Great work, everybody! And thank you for the build support.

On Wed, May 11, 2022 at 3:36 AM Even Rouault 
wrote:

> Hi,
>
> Things should be good enough with RC4.
>
> Motion:
>
> Adopt GDAL 3.5.0RC4 as final 3.5.0 release
>
> Starting with my +1
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.5.0rc4 available (Re: GDAL 3.5.0 release candidate available)

[Sean Gillies]

Explicitly setting rpath in LDFLAGS solved my PROJ 9 problem. This feels
like a regression. It isn't necessary for Linux and wasn't necessary for
autotools builds of PROJ 8.2.1.

Back on topic: GDAL 3.5.0rc4 seems fine.


On Tue, May 10, 2022, 4:18 PM Sean Gillies  wrote:

> Thanks, Alan!
>
> I feel like we're in the middle of going backwards to go forwards when it
> comes to FOSS4G software builds.
>
> On Tue, May 10, 2022 at 4:05 PM Alan Snow  wrote:
>
>> This might be a helpful reference:
>> https://github.com/pyproj4/pyproj-wheels/issues/45
>>
>> On Tue, May 10, 2022, 4:33 PM Even Rouault 
>> wrote:
>>
>>> Sean,
>>>
>>> So your issue is with a GDAL autoconf build. I can't see any significant
>>> changes in configure.ac between GDAL 3.5.0 and 3.4.0 related to rpath.
>>> This has probably not changed in years. I do see in your logs that the
>>> linking line of GDAL has a
>>>
>>>  -rpath /usr/local/lib
>>>
>>> and that there's a /usr/local/lib/libproj.25.dylib file
>>>
>>> But perhaps the issue is more with the PROJ CMake build (hypothesis that
>>> you could check by testing with a PROJ 8.2.1 autoconf build) ? I don't
>>> know... Mac + rpath is hostile territory for me.
>>>
>>> Does running a GDAL binary, like gdalinfo, works before using that
>>> delocate module ?
>>>
>>> Yes 5413 is CMake specific.
>>>
>>> Even
>>> Le 10/05/2022 à 23:02, Sean Gillies a écrit :.
>>>
>>> Hi Even,
>>>
>>> I'm running into a problem with GDAL 3.5.0rc4 and PROJ 9.0.0 on macos.
>>> It occurs when I use https://github.com/matthew-brett/delocate to
>>> relocate GDAL and its dependencies into rasterio wheels.
>>>
>>> The error:
>>>
>>> ERROR:delocate.libsana:
>>> @rpath/libproj.25.dylib not found:
>>> Needed by: /usr/local/lib/libgdal.31.dylib
>>> Search path:
>>>
>>> ERROR:delocate.libsana:@rpath/libproj.25.dylib not found, requested by
>>> /usr/local/lib/libgdal.31.dylib
>>>
>>> Here is its context:
>>> https://github.com/rasterio/rasterio-wheels/runs/6375882173?check_suite_focus=true#step:4:4025
>>> .
>>>
>>> I searched the GDAL tracker for related issues and only found
>>> https://github.com/OSGeo/gdal/issues/5413. It looks like it could be
>>> related, but it also looks like the fix is cmake specific and isn't
>>> available for autotools builds. Is that correct?
>>>
>>>
>>>
>>> On Tue, May 10, 2022 at 8:13 AM Even Rouault 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I've issued a rc4 with the following fixes w.r.t rc3:
>>>> * fix build error with CMake >= 3.10.2 and < 3.11
>>>> * fix build errors with Fedora mingw32
>>>>
>>>> Updated archives:
>>>>
>>>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.xz
>>>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.gz
>>>>https://download.osgeo.org/gdal/3.5.0/gdal350rc4.zip
>>>>
>>>> Even
>>>>
>>>> Le 06/05/2022 à 15:06, Even Rouault a écrit :
>>>> > Hi,
>>>> >
>>>> > I have prepared a GDAL/OGR 3.5.0 release candidate.
>>>> >
>>>> > NEWS at:
>>>> >
>>>> >   https://github.com/OSGeo/gdal/blob/v3.5.0RC1/NEWS.md
>>>> >
>>>> > Note the first item about the new CMake build system, and the
>>>> > deprecation of the autoconf & nmake build systems that will be
>>>> removed
>>>> > in GDAL 3.6.0
>>>> >
>>>> > Pick up an archive among the following ones (by ascending size):
>>>> >
>>>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.xz
>>>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.gz
>>>> >   https://download.osgeo.org/gdal/3.5.0/gdal350rc1.zip
>>>> >
>>>> > A snapshot of the gdalautotest suite is also available :
>>>> >
>>>> > https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.tar.gz
>>>> >   https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.zip
>>>> >
>>>> > A snapshot of the documentation is at:
>>>> >
>>>> >   https://download.osgeo.org/gdal/3.5.0/gdal350doc.zip
>>>> >
>>>> >

Re: [gdal-dev] GDAL 3.5.0rc4 available (Re: GDAL 3.5.0 release candidate available)

[Sean Gillies]

Thanks, Alan!

I feel like we're in the middle of going backwards to go forwards when it
comes to FOSS4G software builds.

On Tue, May 10, 2022 at 4:05 PM Alan Snow  wrote:

> This might be a helpful reference:
> https://github.com/pyproj4/pyproj-wheels/issues/45
>
> On Tue, May 10, 2022, 4:33 PM Even Rouault 
> wrote:
>
>> Sean,
>>
>> So your issue is with a GDAL autoconf build. I can't see any significant
>> changes in configure.ac between GDAL 3.5.0 and 3.4.0 related to rpath.
>> This has probably not changed in years. I do see in your logs that the
>> linking line of GDAL has a
>>
>>  -rpath /usr/local/lib
>>
>> and that there's a /usr/local/lib/libproj.25.dylib file
>>
>> But perhaps the issue is more with the PROJ CMake build (hypothesis that
>> you could check by testing with a PROJ 8.2.1 autoconf build) ? I don't
>> know... Mac + rpath is hostile territory for me.
>>
>> Does running a GDAL binary, like gdalinfo, works before using that
>> delocate module ?
>>
>> Yes 5413 is CMake specific.
>>
>> Even
>> Le 10/05/2022 à 23:02, Sean Gillies a écrit :.
>>
>> Hi Even,
>>
>> I'm running into a problem with GDAL 3.5.0rc4 and PROJ 9.0.0 on macos. It
>> occurs when I use https://github.com/matthew-brett/delocate to relocate
>> GDAL and its dependencies into rasterio wheels.
>>
>> The error:
>>
>> ERROR:delocate.libsana:
>> @rpath/libproj.25.dylib not found:
>> Needed by: /usr/local/lib/libgdal.31.dylib
>> Search path:
>>
>> ERROR:delocate.libsana:@rpath/libproj.25.dylib not found, requested by
>> /usr/local/lib/libgdal.31.dylib
>>
>> Here is its context:
>> https://github.com/rasterio/rasterio-wheels/runs/6375882173?check_suite_focus=true#step:4:4025
>> .
>>
>> I searched the GDAL tracker for related issues and only found
>> https://github.com/OSGeo/gdal/issues/5413. It looks like it could be
>> related, but it also looks like the fix is cmake specific and isn't
>> available for autotools builds. Is that correct?
>>
>>
>>
>> On Tue, May 10, 2022 at 8:13 AM Even Rouault 
>> wrote:
>>
>>> Hi,
>>>
>>> I've issued a rc4 with the following fixes w.r.t rc3:
>>> * fix build error with CMake >= 3.10.2 and < 3.11
>>> * fix build errors with Fedora mingw32
>>>
>>> Updated archives:
>>>
>>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.xz
>>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.gz
>>>https://download.osgeo.org/gdal/3.5.0/gdal350rc4.zip
>>>
>>> Even
>>>
>>> Le 06/05/2022 à 15:06, Even Rouault a écrit :
>>> > Hi,
>>> >
>>> > I have prepared a GDAL/OGR 3.5.0 release candidate.
>>> >
>>> > NEWS at:
>>> >
>>> >   https://github.com/OSGeo/gdal/blob/v3.5.0RC1/NEWS.md
>>> >
>>> > Note the first item about the new CMake build system, and the
>>> > deprecation of the autoconf & nmake build systems that will be removed
>>> > in GDAL 3.6.0
>>> >
>>> > Pick up an archive among the following ones (by ascending size):
>>> >
>>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.xz
>>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.gz
>>> >   https://download.osgeo.org/gdal/3.5.0/gdal350rc1.zip
>>> >
>>> > A snapshot of the gdalautotest suite is also available :
>>> >
>>> > https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.tar.gz
>>> >   https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.zip
>>> >
>>> > A snapshot of the documentation is at:
>>> >
>>> >   https://download.osgeo.org/gdal/3.5.0/gdal350doc.zip
>>> >
>>> > The GDAL-GRASS plugin sources and release process has been moved to
>>> > https://github.com/OSGeo/gdal-grass
>>> >
>>> > I'll call for a vote promoting it to next week if no serious problems
>>> > are reported before.
>>> >
>>> > The "release/3.5" branch has been created for all bug fixes related to
>>> > 3.5.x. master is now targetting 3.6.0dev.
>>> >
>>> > Best regards,
>>> >
>>> > Even
>>> >
>>> --
>>> http://www.spatialys.com
>>> My software is free, but my time generally not.
>>>
>>> ___
>>> gdal-dev mailing list
>>> gdal-dev@lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>
>>
>> --
>> Sean Gillies
>>
>> ___
>> gdal-dev mailing 
>> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>> -- http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.5.0rc4 available (Re: GDAL 3.5.0 release candidate available)

[Sean Gillies]

I agree that this seems more likely to be an issue with PROJ. I found
suggestions from Mike Taves in the PROJ project and am following them.

I'm trying GDAL 3.5.0rc4 with PROJ 8.2.1 and will follow up in an hour.

On Tue, May 10, 2022 at 3:33 PM Even Rouault 
wrote:

> Sean,
>
> So your issue is with a GDAL autoconf build. I can't see any significant
> changes in configure.ac between GDAL 3.5.0 and 3.4.0 related to rpath.
> This has probably not changed in years. I do see in your logs that the
> linking line of GDAL has a
>
>  -rpath /usr/local/lib
>
> and that there's a /usr/local/lib/libproj.25.dylib file
>
> But perhaps the issue is more with the PROJ CMake build (hypothesis that
> you could check by testing with a PROJ 8.2.1 autoconf build) ? I don't
> know... Mac + rpath is hostile territory for me.
>
> Does running a GDAL binary, like gdalinfo, works before using that
> delocate module ?
>
> Yes 5413 is CMake specific.
>
> Even
> Le 10/05/2022 à 23:02, Sean Gillies a écrit :.
>
> Hi Even,
>
> I'm running into a problem with GDAL 3.5.0rc4 and PROJ 9.0.0 on macos. It
> occurs when I use https://github.com/matthew-brett/delocate to relocate
> GDAL and its dependencies into rasterio wheels.
>
> The error:
>
> ERROR:delocate.libsana:
> @rpath/libproj.25.dylib not found:
> Needed by: /usr/local/lib/libgdal.31.dylib
> Search path:
>
> ERROR:delocate.libsana:@rpath/libproj.25.dylib not found, requested by
> /usr/local/lib/libgdal.31.dylib
>
> Here is its context:
> https://github.com/rasterio/rasterio-wheels/runs/6375882173?check_suite_focus=true#step:4:4025
> .
>
> I searched the GDAL tracker for related issues and only found
> https://github.com/OSGeo/gdal/issues/5413. It looks like it could be
> related, but it also looks like the fix is cmake specific and isn't
> available for autotools builds. Is that correct?
>
>
>
> On Tue, May 10, 2022 at 8:13 AM Even Rouault 
> wrote:
>
>> Hi,
>>
>> I've issued a rc4 with the following fixes w.r.t rc3:
>> * fix build error with CMake >= 3.10.2 and < 3.11
>> * fix build errors with Fedora mingw32
>>
>> Updated archives:
>>
>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.xz
>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.gz
>>https://download.osgeo.org/gdal/3.5.0/gdal350rc4.zip
>>
>> Even
>>
>> Le 06/05/2022 à 15:06, Even Rouault a écrit :
>> > Hi,
>> >
>> > I have prepared a GDAL/OGR 3.5.0 release candidate.
>> >
>> > NEWS at:
>> >
>> >   https://github.com/OSGeo/gdal/blob/v3.5.0RC1/NEWS.md
>> >
>> > Note the first item about the new CMake build system, and the
>> > deprecation of the autoconf & nmake build systems that will be removed
>> > in GDAL 3.6.0
>> >
>> > Pick up an archive among the following ones (by ascending size):
>> >
>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.xz
>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.gz
>> >   https://download.osgeo.org/gdal/3.5.0/gdal350rc1.zip
>> >
>> > A snapshot of the gdalautotest suite is also available :
>> >
>> > https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.tar.gz
>> >   https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.zip
>> >
>> > A snapshot of the documentation is at:
>> >
>> >   https://download.osgeo.org/gdal/3.5.0/gdal350doc.zip
>> >
>> > The GDAL-GRASS plugin sources and release process has been moved to
>> > https://github.com/OSGeo/gdal-grass
>> >
>> > I'll call for a vote promoting it to next week if no serious problems
>> > are reported before.
>> >
>> > The "release/3.5" branch has been created for all bug fixes related to
>> > 3.5.x. master is now targetting 3.6.0dev.
>> >
>> > Best regards,
>> >
>> > Even
>> >
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
> --
> Sean Gillies
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.5.0rc4 available (Re: GDAL 3.5.0 release candidate available)

[Sean Gillies]

Hi Even,

I'm running into a problem with GDAL 3.5.0rc4 and PROJ 9.0.0 on macos. It
occurs when I use https://github.com/matthew-brett/delocate to relocate
GDAL and its dependencies into rasterio wheels.

The error:

ERROR:delocate.libsana:
@rpath/libproj.25.dylib not found:
Needed by: /usr/local/lib/libgdal.31.dylib
Search path:

ERROR:delocate.libsana:@rpath/libproj.25.dylib not found, requested by
/usr/local/lib/libgdal.31.dylib

Here is its context:
https://github.com/rasterio/rasterio-wheels/runs/6375882173?check_suite_focus=true#step:4:4025
.

I searched the GDAL tracker for related issues and only found
https://github.com/OSGeo/gdal/issues/5413. It looks like it could be
related, but it also looks like the fix is cmake specific and isn't
available for autotools builds. Is that correct?



On Tue, May 10, 2022 at 8:13 AM Even Rouault 
wrote:

> Hi,
>
> I've issued a rc4 with the following fixes w.r.t rc3:
> * fix build error with CMake >= 3.10.2 and < 3.11
> * fix build errors with Fedora mingw32
>
> Updated archives:
>
>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.xz
>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc4.tar.gz
>https://download.osgeo.org/gdal/3.5.0/gdal350rc4.zip
>
> Even
>
> Le 06/05/2022 à 15:06, Even Rouault a écrit :
> > Hi,
> >
> > I have prepared a GDAL/OGR 3.5.0 release candidate.
> >
> > NEWS at:
> >
> >   https://github.com/OSGeo/gdal/blob/v3.5.0RC1/NEWS.md
> >
> > Note the first item about the new CMake build system, and the
> > deprecation of the autoconf & nmake build systems that will be removed
> > in GDAL 3.6.0
> >
> > Pick up an archive among the following ones (by ascending size):
> >
> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.xz
> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.gz
> >   https://download.osgeo.org/gdal/3.5.0/gdal350rc1.zip
> >
> > A snapshot of the gdalautotest suite is also available :
> >
> > https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.tar.gz
> >   https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.zip
> >
> > A snapshot of the documentation is at:
> >
> >   https://download.osgeo.org/gdal/3.5.0/gdal350doc.zip
> >
> > The GDAL-GRASS plugin sources and release process has been moved to
> > https://github.com/OSGeo/gdal-grass
> >
> > I'll call for a vote promoting it to next week if no serious problems
> > are reported before.
> >
> > The "release/3.5" branch has been created for all bug fixes related to
> > 3.5.x. master is now targetting 3.6.0dev.
> >
> > Best regards,
> >
> > Even
> >
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.5.0rc3 available (Re: GDAL 3.5.0 release candidate available)

[Sean Gillies]

Thanks, Even. RC3 compiles and I don't see any regressions from rasterio's
perspective. Looking forward to the new release!

On Mon, May 9, 2022 at 1:30 PM Even Rouault 
wrote:

> Sean,
>
> I've issue a rc3 with a fix for that issue:
>
>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc3.tar.xz
>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc3.tar.gz
>https://download.osgeo.org/gdal/3.5.0/gdal350rc3.zip
>
> Even
> Le 09/05/2022 à 20:47, Sean Gillies a écrit :
>
> Hi Even,
>
> I'm still using the legacy configure script and have found that
> ogrmvtdataset.cpp doesn't compile on my system due to
>
> In file included from ogrmvtdataset.cpp:43:
> .../ogr_geos.h:40:12: fatal error: geos_c.h: No such file or directory
>
> This is GDAL 3.5.0rc2. I haven't tried the earlier pre-releases. GDAL
> 3.4.3 compiled and installed on the same computer using the same script,
> which is:
>
> LD_LIBRARY_PATH=/home/seangillies/local/lib ./configure
> --prefix=/home/seangillies/local --with-curl
> --with-geos=/home/seangillies/local/bin/geos-config
> --with-proj=/home/seangillies/local
>
>
> On Mon, May 9, 2022 at 3:48 AM Even Rouault 
> wrote:
>
>> Hi,
>>
>> I've issued a RC2 with the following changes:
>>
>> - CMake: fix build of NITF driver if rebuilding after disabling JPEG
>> driver
>>
>> - port/cpl_recode_iconv.cpp: fix invalid cast error with uclibc (#5684)
>>
>> - PNG: report cause when unable to create file
>>
>> - PCIDSK: fix build on 32-bit architectures on Debian (fixes #5694)
>>
>> - TileDB: compiler warning fix on 32bit
>>
>> - ogr2ogr: avoid potential nullptr deref (CID 1488679)
>>
>> - netCDF: add support for writing/reading geolocation array without a
>> grid_mapping variable
>>
>> Updated archives:
>>
>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc2.tar.xz
>>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc2.tar.gz
>>https://download.osgeo.org/gdal/3.5.0/gdal350rc2.zip
>> https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc2.tar.gz
>>https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc2.zip
>>
>> Even
>>
>> Le 06/05/2022 à 15:06, Even Rouault a écrit :
>> > Hi,
>> >
>> > I have prepared a GDAL/OGR 3.5.0 release candidate.
>> >
>> > NEWS at:
>> >
>> >   https://github.com/OSGeo/gdal/blob/v3.5.0RC1/NEWS.md
>> >
>> > Note the first item about the new CMake build system, and the
>> > deprecation of the autoconf & nmake build systems that will be removed
>> > in GDAL 3.6.0
>> >
>> > Pick up an archive among the following ones (by ascending size):
>> >
>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.xz
>> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.gz
>> >   https://download.osgeo.org/gdal/3.5.0/gdal350rc1.zip
>> >
>> > A snapshot of the gdalautotest suite is also available :
>> >
>> > https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.tar.gz
>> >   https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.zip
>> >
>> > A snapshot of the documentation is at:
>> >
>> >   https://download.osgeo.org/gdal/3.5.0/gdal350doc.zip
>> >
>> > The GDAL-GRASS plugin sources and release process has been moved to
>> > https://github.com/OSGeo/gdal-grass
>> >
>> > I'll call for a vote promoting it to next week if no serious problems
>> > are reported before.
>> >
>> > The "release/3.5" branch has been created for all bug fixes related to
>> > 3.5.x. master is now targetting 3.6.0dev.
>> >
>> > Best regards,
>> >
>> > Even
>> >
>>
>>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.5.0rc2 available (Re: GDAL 3.5.0 release candidate available)

[Sean Gillies]

Hi Even,

I'm still using the legacy configure script and have found that
ogrmvtdataset.cpp doesn't compile on my system due to

In file included from ogrmvtdataset.cpp:43:
.../ogr_geos.h:40:12: fatal error: geos_c.h: No such file or directory

This is GDAL 3.5.0rc2. I haven't tried the earlier pre-releases. GDAL 3.4.3
compiled and installed on the same computer using the same script, which is:

LD_LIBRARY_PATH=/home/seangillies/local/lib ./configure
--prefix=/home/seangillies/local --with-curl
--with-geos=/home/seangillies/local/bin/geos-config
--with-proj=/home/seangillies/local


On Mon, May 9, 2022 at 3:48 AM Even Rouault 
wrote:

> Hi,
>
> I've issued a RC2 with the following changes:
>
> - CMake: fix build of NITF driver if rebuilding after disabling JPEG driver
>
> - port/cpl_recode_iconv.cpp: fix invalid cast error with uclibc (#5684)
>
> - PNG: report cause when unable to create file
>
> - PCIDSK: fix build on 32-bit architectures on Debian (fixes #5694)
>
> - TileDB: compiler warning fix on 32bit
>
> - ogr2ogr: avoid potential nullptr deref (CID 1488679)
>
> - netCDF: add support for writing/reading geolocation array without a
> grid_mapping variable
>
> Updated archives:
>
>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc2.tar.xz
>https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc2.tar.gz
>https://download.osgeo.org/gdal/3.5.0/gdal350rc2.zip
> https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc2.tar.gz
>https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc2.zip
>
> Even
>
> Le 06/05/2022 à 15:06, Even Rouault a écrit :
> > Hi,
> >
> > I have prepared a GDAL/OGR 3.5.0 release candidate.
> >
> > NEWS at:
> >
> >   https://github.com/OSGeo/gdal/blob/v3.5.0RC1/NEWS.md
> >
> > Note the first item about the new CMake build system, and the
> > deprecation of the autoconf & nmake build systems that will be removed
> > in GDAL 3.6.0
> >
> > Pick up an archive among the following ones (by ascending size):
> >
> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.xz
> >   https://download.osgeo.org/gdal/3.5.0/gdal-3.5.0rc1.tar.gz
> >   https://download.osgeo.org/gdal/3.5.0/gdal350rc1.zip
> >
> > A snapshot of the gdalautotest suite is also available :
> >
> > https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.tar.gz
> >   https://download.osgeo.org/gdal/3.5.0/gdalautotest-3.5.0rc1.zip
> >
> > A snapshot of the documentation is at:
> >
> >   https://download.osgeo.org/gdal/3.5.0/gdal350doc.zip
> >
> > The GDAL-GRASS plugin sources and release process has been moved to
> > https://github.com/OSGeo/gdal-grass
> >
> > I'll call for a vote promoting it to next week if no serious problems
> > are reported before.
> >
> > The "release/3.5" branch has been created for all bug fixes related to
> > 3.5.x. master is now targetting 3.6.0dev.
> >
> > Best regards,
> >
> > Even
> >
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDALWarpAppOptionsNew and GDALWarp usage?

[Sean Gillies]

Thanks, Even!

When I checked the return of GDALWarpAppOptionsNew() and the error stack, I
found that the problem was that I'd been adding an unexpected "-wt
Unknown". After screening that out, I'm getting the resampling I expect.

On Wed, Apr 13, 2022 at 2:10 PM Even Rouault 
wrote:

> Sean,
>
> This looks fine, except for the argv[0] value, "lolwut". What is that
> supposed to be ? I assume GDALWarpAppOptionsNew() fails on that and returns
> NULL.
>
> The obvious demo for using GDALWarp() is the gdalwarp binary itself:
> https://github.com/OSGeo/gdal/blob/master/apps/gdalwarp_bin.cpp
>
> The COG driver is also a user of it, perhaps more illustrative than the
> above :
> https://github.com/OSGeo/gdal/blob/master/frmts/gtiff/cogdriver.cpp#L552
>
> Even
> Le 13/04/2022 à 21:53, Sean Gillies a écrit :
>
> Hi all,
>
> I'm trying to figure out how to use the GDALWarp function from
> gdalwarp_lib.cpp. It looks like it takes a pointer to a GDALWarpAppOptions
> structure that is parsed from a string list, basically gdalwarp's argv,
> right?
>
> I'm able to get it to warp data, but the options, specifically the
> resampling option, don't seem to be passed into the warper. The output is
> the same for all resampling values.
>
> Here's my C usage (Cython, actually, but translates to C).
>
> cdef GDALWarpAppOptions *warp_options = NULL
> cdef char **argv = NULL
>
> try:
>
>
> argv = CSLAddString(argv, "lolwut")
> resampling_opt_value = Resampling(resampling).name  # like
> "bilinear"
> argv = CSLAddString(argv, "-r")
> argv = CSLAddString(argv, resampling_opt_value)
> ...
> CSLPrint(argv, NULL)
> warp_options = GDALWarpAppOptionsNew(argv, NULL)
> output_ds = GDALWarp(
> NULL,
> dst_dataset,
> 1,
> src_datasets,
> warp_options,
> NULL
> )
> finally:
> ...
>
> The printed output, my argv, is
>
> lolwut
> -r
> bilinear
>
> Does GDALWarpAppOptionsNew expect more values at the head of the command
> line? Is that why it is missing "-r"? Or is my string list made up of the
> wrong kind of strings? Documentation examples of building the arguments for
> GDALWarp are scarce. I'd be super grateful for help from anyone who has got
> code this to work.
>
> Thanks,
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDALWarpAppOptionsNew and GDALWarp usage?

[Sean Gillies]

Hi all,

I'm trying to figure out how to use the GDALWarp function from
gdalwarp_lib.cpp. It looks like it takes a pointer to a GDALWarpAppOptions
structure that is parsed from a string list, basically gdalwarp's argv,
right?

I'm able to get it to warp data, but the options, specifically the
resampling option, don't seem to be passed into the warper. The output is
the same for all resampling values.

Here's my C usage (Cython, actually, but translates to C).

cdef GDALWarpAppOptions *warp_options = NULL
cdef char **argv = NULL

try:

argv = CSLAddString(argv, "lolwut")
resampling_opt_value = Resampling(resampling).name  # like
"bilinear"
argv = CSLAddString(argv, "-r")
argv = CSLAddString(argv, resampling_opt_value)
...
CSLPrint(argv, NULL)
warp_options = GDALWarpAppOptionsNew(argv, NULL)
output_ds = GDALWarp(
NULL,
dst_dataset,
1,
src_datasets,
warp_options,
NULL
)
finally:
...

The printed output, my argv, is

lolwut
-r
bilinear

Does GDALWarpAppOptionsNew expect more values at the head of the command
line? Is that why it is missing "-r"? Or is my string list made up of the
wrong kind of strings? Documentation examples of building the arguments for
GDALWarp are scarce. I'd be super grateful for help from anyone who has got
code this to work.

Thanks,

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC for the new cloud credentials framework in PR 5463 and 5390?

[Sean Gillies]

Even,

At the very least, the new file duplicates storage of credentials that may
already be stored in cloud-specific credentials files, and creates a new
way for users to expose their secrets. Also, cloud providers and
organizations have moved or are moving to focusing on short-lived
credentials, SSO, etc. How useful is a cross-cloud credentials file if it
supports only static credentials? Why not support named profiles already
defined in cloud-specific files? Python and C++ programmers haven't needed
this framework because they can maintain their own maps of credentials or
roles to resources, so I guess this feature is mainly for command line
users? Do command line users do this kind of thing enough to warrant a new
framework?

On Thu, Mar 24, 2022 at 8:45 AM Even Rouault 
wrote:

> Sean,
>
> I saw them as business-as-usual enhancements not impacting the software in
> fundamental ways. I'm not sure what I would put in a RFC that is not in
> their commit message. Maybe I don't understand what your concern is.
>
> Even
> Le 24/03/2022 à 15:28, Sean Gillies a écrit :
>
> Hi all,
>
> The intent and scope of the features developed in
> https://github.com/OSGeo/gdal/pull/5463 and
> https://github.com/OSGeo/gdal/pull/5390 seem rather big and unclear to
> me. This seems to me to warrant an RFC. Yes? No?
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] RFC for the new cloud credentials framework in PR 5463 and 5390?

[Sean Gillies]

Hi all,

The intent and scope of the features developed in
https://github.com/OSGeo/gdal/pull/5463 and
https://github.com/OSGeo/gdal/pull/5390 seem rather big and unclear to me.
This seems to me to warrant an RFC. Yes? No?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [geos-devel] GEOS Maintenance Grant

[Sean Gillies]

Howard,

I'm in favor, but first I'd like to see the GEOS project adopt a code of
conduct. Sponsors look for one these days, right? It's probably a good move
for GEOS in the long run.

On Tue, Feb 15, 2022 at 8:37 AM Howard Butler  wrote:

> GDAL PSC,
>
> When we wrote the GDAL RFCs on sponsorship, we provided an escape clause
> to allow us to direct resources to other projects upon which GDAL depends.
> Our sponsorship numbers are still increasing, which provides us an
> opportunity to directly support some of those projects, and one of them is
> obviously GEOS. GEOS provides all of the geometry algebra support for
> GDAL/OGR and many other open source geospatial softwares including Shapely,
> PostGIS, GeoPandas, MapServer, and more.
>
> Dan Baston of the GEOS PSC has been identified as the developer with
> capacity and interest in the next year to take on GEOS development on APIs
> and performance, which he has a long history of doing for the project. This
> support should allow him to work longer, multi-release upgrades that will
> provide strong performance and convenience benefits for the project.
>
> I motion to provide the GEOS PSC with a $50,000 USD grant to address
> performance, API, and other work that does not attract directed funding in
> GEOS. The GEOS PSC will be responsible for coordinating work tasks, rates,
> and development timelines. Howard Butler or Even Rouault of the GDAL
> NumFocus liaison team will coordinate dispersement as directed by the GEOS
> PSC and NumFocus rules.
>
> Thank you again to the GDAL Sponsors https://gdal.org/sponsors/index.html
> who have made this kind of grant possible. A better GEOS makes for a better
> GDAL.
>
> Howard
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 85: Policy regarding substantial code additions

[Sean Gillies]

+1 from me.

Howard, I really appreciate how you're reminding us that what proprietary
vendors want is for GDAL to help distribute their software. Writing code
that works is the easy part. Getting it onto computers and getting people
to use it is the hard part.

On Fri, Jan 21, 2022 at 3:18 AM Even Rouault 
wrote:

> Hi,
>
> As the discussion seems to have calmed down, I move to adopt RFC 85:
> Policy regarding substantial code additions:
> https://github.com/OSGeo/gdal/pull/5128
>
> Starting with my +1
>
> Even
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Faster alternative to GetFeatureCount?

[Sean Gillies]

Hi Jon,

The performance of GetFeatureCount, filters, and GetNextFeature depends a
lot on the data format. If your data is in Postgres, for example, these
operations are super fast because the database keeps the number of rows
(features) in memory and has highly optimized views of selected rows. If
your data is in a CSV file, GDAL has to read every line to count the number
of features and when getting filtered features has to read all the lines in
between the ones you want. See
https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/csv/ogrcsvlayer.cpp#L1667
.

Can you switch to a more optimized format?

On Thu, Jan 20, 2022 at 2:02 AM Jon Morris  wrote:

> I'm writing applications using the GDAL Python bindings and when I profile
> for performance, GetFeatureCount frequently comes out near the top. I'm
> often using it to check whether a spatial or attribute filter has returned
> any features and don't need the full count. When the layer contains
> millions of features, there would be a big performance improvement if we
> could exit the count early.
>
>
>
> Is there a better way of doing this? I've tried using GetNextFeature
> instead, but there must be quite a lot of overhead in that function as it
> is much slower. All I need to know is if the layer has has 0, 1 or >1
> features, I don't need the actual count. Can anyone suggest the fastest way
> of doing this in Python? I'm using GDAL 3.3.1 at the moment but could
> upgrade if there is new functionality that would help.
>
>
>
> Thanks,
>
>
>
> Jon
>
>
>
> *Jon Morris*
>
> *Software Developer*
>
>
>
>
> e:  *jon.mor...@jbarisk.com* 
> t:  *+44 (0)1756 799919* <+44%20(0)1756%20799919>
> *www.jbarisk.com* <http://www.jbarisk.com/>
> [image: Facebook] <https://www.facebook.com/TheFloodPeople>
> [image: LinkedIn] <https://www.linkedin.com/company/jba-risk-management/>
> [image: Twitter] <https://twitter.com/JBARisk>
> [image: YouTube]
> <https://www.youtube.com/channel/UC0iatom2jYbW96voW0rlpCw>
>
>
> All JBA Risk Management's email messages contain confidential information
> and are intended only for the individual(s) named. If you are not the named
> addressee you should not disseminate, distribute or copy this e-mail.
> Please notify the sender immediately by email if you have received this
> email by mistake and delete this email from your system.
> JBA Risk Management Limited is registered in England, company number
> 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North
> Yorkshire, BD23 3FD, England.
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] HTTP range request retrying?

[Sean Gillies via gdal-dev]

Hi Even,

Yes, that's a good point. What I've done at
https://github.com/mapbox/rasterio/blob/maint-1.2/tests/test_warp.py#L2022
makes the tests sensitive to the text of CPLError messages, which could
change. It's not ideal. On the other hand, I do have confirmation that
range requests can be retried. It can be a hard thing to demonstrate
reliably with production servers, which might return a 503 only one time in
a million.

On Tue, Oct 5, 2021 at 10:09 AM Even Rouault 
wrote:

> Hi Sean,
>
> The code for retry of range requests of regions of data is at
> https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_curl.cpp#L1602
> . It indeed looks like there's no test for that. You might file an issue
> about adding one.
>
> Side node: it might become a maintenance burden on rasterio if you test
> such low level implementation detail of GDAL, especially if you test like
> we do which exact requests are done and in which order, if we change at
> some later point how requests are done
>
> Even
> Le 05/10/2021 à 17:44, Sean Gillies via gdal-dev a écrit :
>
> Hi all,
>
> In writing some rasterio tests I am able to get GDAL to retry head
> requests for a vsicurl dataset, but am unable to trigger retries for range
> requests to regions of the data. I don't recognize any tests for range
> request retrying in or near
> https://github.com/OSGeo/gdal/blob/master/autotest/gcore/vsicurl.py#L473.
> Do we intend to support this or is it a missing feature?
>
> --
> Sean Gillies
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] HTTP range request retrying?

[Sean Gillies via gdal-dev]

Hi all,

In writing some rasterio tests I am able to get GDAL to retry head requests
for a vsicurl dataset, but am unable to trigger retries for range requests
to regions of the data. I don't recognize any tests for range request
retrying in or near
https://github.com/OSGeo/gdal/blob/master/autotest/gcore/vsicurl.py#L473.
Do we intend to support this or is it a missing feature?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Approve Nyall Dawson as a contracted GDAL maintainer

[Sean Gillies via gdal-dev]

On Tue, Sep 14, 2021 at 8:59 AM Howard Butler  wrote:

> Dear PSC,
>
> As a result of our fundraising activity and development of NumFOCUS as a
> financial conduit, it is my pleasure to put forward a motion to approve
> Nyall Dawson as a contracted GDAL maintainer for the year 2021-2022
> beginning on October 1st, 2021 and ending September 30th, 2022.
>
> Details of the maintenance tasking and duties can be found at the
> previously approved RFC 83
> https://gdal.org/development/rfc/rfc83_use_of_project_sponsorship.html
>
> The terms of the contract are for a maximum of 500 hours at 90 €/hr.
>
> Howard
>
> PS I will coordinating the contracting activity in my role as the GDAL
> NumFOCUS contracting liaison, with Frank Warmerdam acting as the secondary.
> Please contact us directly with any comments or concerns.
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>

Excellent!

+1

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Get error handler user data when removing CPL Error Handler

[Sean Gillies via gdal-dev]

Hi,

For what it's worth, In the Python package named rasterio we're using the
push/pop API:
https://github.com/mapbox/rasterio/blob/master/rasterio/_env.pyx#L336.
While Rust's needs may differ, a single handler without any support for
user data works well for Python: everything goes to the logging
infrastructure, one of the kind of globals that Howard refers to in
https://github.com/OSGeo/gdal/blob/master/gdal/doc/source/development/rfc/rfc37_cplerror_userdata.rst#rationale,
and Python developers extend the logger if they want behavior that is
different from the basic defaults.

On Sun, Sep 12, 2021 at 8:12 AM Even Rouault 
wrote:

> Hi,
>
> no there's no thread safe API to do what you want. You'd need a new
> function
>
> CPLErrorHandler CPLSetErrorHandlerEx2( CPLErrorHandler pfnErrorHandlerNew,
> void* pUserData, void** ppOldUserData )
>
> to do that.
>
> But as you mention threads that might compete to set an error handler,
> using CPLSetErrorHandlerEx() is probably not the best strategy. You'd be
> better with CPLPushErrorHandler() / CPLPopErrorHandler() that only affects
> the current thread.
>
> Even
> Le 12/09/2021 à 16:03, Rajsekar Manokaran a écrit :
>
> Hi,
>
> In the gdal rust bindings (https://github.com/georust/gdal), we're trying
> to facilitate the use of CPLSetErrorHandlerEx and related APIs.  While
> setting a handler, we may pass a heap allocated data pointer to the second
> argument, which is then read via the CPLGetErrorHandlerUserData in the
> handler and passed to the user.
>
> However, while removing or setting another handler, we're unable to find a
> race-free method to get the associated user data of the previous handler.
> This is needed to properly deallocate the memory.
>
> Is there an atomic way to get both the previous handler (as returned by
> CPLSetErrorHandler), along with the associated user data?  The issue with
> making two calls, is that another thread might make changes in between the
> two calls.
>
> We could synchronize in our API, but it still has the same issue if the
> user parallely used the C APIs directly or via a different interface.
>
> Relevant PR in rust gdal bindings:
> https://github.com/georust/gdal/pull/215
>
> -
> Regards
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: promote GDAL 3.3.2 RC3

[Sean Gillies]

Yes, rasterio will be ready today. Please go ahead. Thanks for waiting!

On Mon, Sep 6, 2021 at 11:30 AM Even Rouault 
wrote:

> Hi Sean,
>
> did you manage to adapt rasterio / can we promote the release ?
>
> Even
> Le 02/09/2021 à 17:21, Even Rouault a écrit :
>
> Hi Sean,
>
> sure we can delay a bit the release of the final tarballs. Anyway as we
> have the minimum 2 business day rule to pass a motion, this won't be before
> Monday. Those issues with axis order seem to be a never ending source of
> annoyance. Were existing before I got into the geo-business and will
> probably be still there when I retire...
>
> I didn't anticipate that change to break rasterio, although those type of
> changes tend to have tricky consequences, but at the same time having
> AutoIdentifyEPSG() putting EPSG:4326 on a CRS with "wrong" axis order
> caused other issues.
>
> Even
>
>
> Le 02/09/2021 à 17:19, Sean Gillies a écrit :
>
> Oops, forgot to reply to the list. Also, to be more specific, could we
> hold 3.3.2 until Monday, September 6?
>
> On Thu, Sep 2, 2021 at 9:10 AM Sean Gillies  wrote:
>
>> I wish I'd noticed the change around EPSG:4326 sooner. I didn't see a
>> particular announcement, but the breakage was obvious in rasterio's CI
>> builds with GDAL master and I should have been watching more closely.
>>
>> If people update GDAL to 3.3.2 they may break their applications that use
>> rasterio. We'll probably be able to get a new version out by the end of the
>> week or beginning of next week. Could we hold the release a few more days?
>>
>> On Thu, Sep 2, 2021 at 12:54 AM Even Rouault 
>> wrote:
>>
>>> Hi,
>>>
>>> Having heard no issues being reported regarding RC3
>>>
>>> Motion:
>>>
>>> Adopt GDAL 3.3.2 RC3 as final 3.3.2 release
>>>
>>> Starting with my +1
>>>
>>> Even
>>>
>> --
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: promote GDAL 3.3.2 RC3

[Sean Gillies via gdal-dev]

Oops, forgot to reply to the list. Also, to be more specific, could we hold
3.3.2 until Monday, September 6?

On Thu, Sep 2, 2021 at 9:10 AM Sean Gillies  wrote:

> I wish I'd noticed the change around EPSG:4326 sooner. I didn't see a
> particular announcement, but the breakage was obvious in rasterio's CI
> builds with GDAL master and I should have been watching more closely.
>
> If people update GDAL to 3.3.2 they may break their applications that use
> rasterio. We'll probably be able to get a new version out by the end of the
> week or beginning of next week. Could we hold the release a few more days?
>
> On Thu, Sep 2, 2021 at 12:54 AM Even Rouault 
> wrote:
>
>> Hi,
>>
>> Having heard no issues being reported regarding RC3
>>
>> Motion:
>>
>> Adopt GDAL 3.3.2 RC3 as final 3.3.2 release
>>
>> Starting with my +1
>>
>> Even
>>
>
> --
> Sean Gillies
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] autotest questions/issues

[Sean Gillies via gdal-dev]

Hi Robert,

On Wed, Sep 1, 2021 at 1:30 AM Robert Coup 
wrote:

> ...
>
> GDAL has some odd test layouts with particular inter-test dependencies
> from when the test suite was bulk-ported via automation to work under
> pytest — this made it a lot saner, but some of the "test 18 depends on test
> 17 passing" issues remain.
>
> Rob :)
>

The test runner upgrades that you mentioned were a huge help. Now that GDAL
has a real maintenance budget we should be able to pay someone to tackle
the remaining issues in a methodical way. Run the tests in a random order,
say using https://pypi.org/project/pytest-random-order/, and fix the
setup/teardown so that tests are properly isolated.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.3.2 RC2 is available [was Re: GDAL 3.3.2 RC1 available]

[Sean Gillies via gdal-dev]

Hi Even,

Would it be difficult to change the RC URLs and release directory to
include the "rc*" string? If the download was at
https://download.osgeo.org/gdal/3.3.2rc2/gdal-3.3.2rc2.tar.gz and extracted
to gdal-3.3.2rc2 I would only need to make a one line change to my build
system to test it.

I did find some failures in the tests we run when building and testing
rasterio's binary distributions (the wheels on PyPI). GDAL 3.3.2rc2 and
PROJ 7.2.1. The errors seem to come from a failure to auto identify the
EPSG codes in spatial reference systems created from, for example,
'+init=epsg:4326' or '+proj=longlat +datum=WGS84 +no_defs'.

https://github.com/mapbox/rasterio/blob/master/rasterio/_crs.pyx#L241-L243

These failures did not occur with 3.3.0. I'm retroactively checking 3.3.1
now.

Here's the pytest output for one of the failures:

__ test_to_epsg[+proj=longlat +datum=WGS84 +no_defs]
___
proj_string = '+proj=longlat +datum=WGS84 +no_defs'
@pytest.mark.parametrize('proj_string', ['+init=epsg:4326',
'+proj=longlat +datum=WGS84 +no_defs'])
def test_to_epsg(proj_string):
"""CRS has EPSG code"""
>   assert _CRS.from_proj4(proj_string).to_epsg() == 4326
E   assert None == 4326
E -None
E +4326


On Tue, Aug 31, 2021 at 3:05 AM Even Rouault 
wrote:

> Hi,
>
> Here's a RC2 with 2 additional changes w.r.t RC1:
>
> - NEWS file to correctly mention 3.3.2
>
> - gdal_viewshed: Fix incorrect progress reporting (fixes #4390)
>
> Updated download links:
>
>https://download.osgeo.org/gdal/3.3.2/gdal-3.3.2rc2.tar.xz
>https://download.osgeo.org/gdal/3.3.2/gdal-3.3.2rc2.tar.gz
>https://download.osgeo.org/gdal/3.3.2/gdal332rc2.zip
>
> https://download.osgeo.org/gdal/3.3.2/gdalautotest-3.3.2rc2.tar.gz
>https://download.osgeo.org/gdal/3.3.2/gdalautotest-3.3.2rc2.zip
>
> Even
>
> Le 30/08/2021 à 14:27, Even Rouault a écrit :
> > Hi,
> >
> > I have prepared a GDAL/OGR 3.3.2 release candidate.
> >
> > Pick up an archive among the following ones (by ascending size):
> >
> >   https://download.osgeo.org/gdal/3.3.2/gdal-3.3.2rc1.tar.xz
> >   https://download.osgeo.org/gdal/3.3.2/gdal-3.3.2rc1.tar.gz
> >   https://download.osgeo.org/gdal/3.3.2/gdal332rc1.zip
> >
> > A snapshot of the gdalautotest suite is also available :
> >
> > https://download.osgeo.org/gdal/3.3.2/gdalautotest-3.3.2rc1.tar.gz
> >   https://download.osgeo.org/gdal/3.3.2/gdalautotest-3.3.2rc1.zip
> >
> > GDAL-GRASS plugin:
> >
> >   https://download.osgeo.org/gdal/3.3.2/gdal-grass-3.3.2.tar.gz
> >
> > The NEWS file is here :
> >
> >   https://github.com/OSGeo/gdal/blob/v3.3.2RC1/gdal/NEWS
> >
> > I'll call for a vote promoting it to final later this week if no
> > serious problems are reported before.
> >
> > Best regards,
> >
> > Even
> >
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Approve Even Rouault as a contracted GDAL maintainer

[Sean Gillies]

+1 from me. What a great milestone for the project!

On Tue, Aug 17, 2021 at 9:34 AM Howard Butler  wrote:

> Dear PSC,
>
> As a result of our fundraising activity and development of NumFOCUS as a
> financial conduit, it is my pleasure to put forward a motion to approve
> Even Rouault as a contracted GDAL maintainer for the year 2021-2022
> beginning on August 1st, 2021 and ending July 31st, 2022.
>
> Details of the maintenance tasking and duties can be found at the
> previously approved RFC 83
> https://gdal.org/development/rfc/rfc83_use_of_project_sponsorship.html
>
> The terms of the contract are for 833 hours at $120/hr USD.
>
> Howard
>
> PS I will coordinating the contracting activity in my role as the GDAL
> NumFOCUS contracting liaison, with Frank Warmerdam acting as the secondary.
> Please contact us directly with any comments or concerns.
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 83: guidelines for the use of GDAL project sponsorship

[Sean Gillies]

+1

On Tue, Jun 8, 2021 at 4:50 AM Even Rouault 
wrote:

> Hi,
>
> Motion:
>
> adopt RFC 83: guidelines for the use of GDAL project sponsorship (
> https://github.com/OSGeo/gdal/pull/3855 )
>
> Starting with my +1
>
> Even
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Introducing the cogger and godal projects

[Sean Gillies via gdal-dev]

Hi Thomas,

Congratulations! These look like great projects.

On Fri, Jun 4, 2021 at 4:07 AM thomas bonfort 
wrote:

> Hello gdal,
>
> We're releasing two projects on github under an Apache-2.0 licence
> which may be of interest to the GDAL community.
>
> The first one, https://github.com/airbusgeo/cogger is a lightweight
> geotiff to COG converter that reshuffles the bytes of a tiled geotiff
> to make it cloud compatible. Whereas it does not replace the standard
> gdal tools for initially creating a geotiff, it is much faster than
> the last "-co COPY_SRC_OVERVIEWS=YES" classical gdal_translate method
> for outputting a cog, and also allows more flexibility when varying
> compression methods or tile sizes need to applied on overviews. Cogger
> can be used as a standalone command line tool, or as a go library
> function for interacting with remote storage without the need for
> local files.
>
> The second one, https://github.com/airbusgeo/godal is a golang wrapper
> for GDAL's raster api. While there is an already established golang
> wrapper for gdal that exists (https://github.com/lukeroth/gdal), godal
> differs by offering:
> - robust error handling and reporting
> - a stable compatible api promise (to the extent of what is possible
> to achieve given gdal's own api)
> - a vsi abstraction that hands off i/o to native golang code
> - a more user-friendly API that is not an exact mirror of the gdal
> api, making lesser used functionality optional. (For example, instead
> of the GDALOpen, GDALOpenEx and GDALOpenShared methods, godal exposes
> a single Open() method with optional arguments validated at compile
> time to pass in open options, shared mode, sibling files, etc...)
> Although the raster handling side of godal is mostly feature complete
> with gdal, the vector and spatial-referencing part is not.
>
> Outside contributions to both these projects are welcome through github.
>
> Best regards,
> Thomas
>

I'm very interested in "a vsi abstraction that hands off i/o to native
golang code".

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 81: support for coordinate epochs in geospatial formats

[Sean Gillies via gdal-dev]

Even,

Sounds good. Until there is consensus on what coordinate epoch means for
OGC:CRS84 GeoJSON, the official and most widely used kind, I think it would
be better if GDAL didn't extend the format. For now, applications that need
more precision can and should use another format.

On Thu, May 27, 2021 at 12:19 PM Even Rouault 
wrote:

> Regarding KML and GeoJSON and OGC:CRS84 (or EPSG:4326 since they are the
> same thing, except axis order), that's a good and hard question. Actually
> that extends to *any* CRS built on top of them, like all the
> EPSG:32[6|7][01-60] UTM CRS, and that's probably for those later than
> things are the most problematic since there's no EPSG code for a UTM WGS 84
> (G1762) CRS. I guess people would potentially want to attach a coordinate
> epoch to them, and have a (non-encoded-in-the-format) convention that they
> actually mean WGS 84 (G1762) (or possibly an earlier realization. The datum
> ensemble reality of WGS 84 is really due to Transit which differs
> significantly from later realizations, which are pretty consistent between
> them within a few centimers) . So I wouldn't forbid such uses (I actually
> got private feedback that some people would even want to store coordinate
> epoch for CRS that aren't considered by EPSG as dynamic CRS, but which, for
> advanced uses, can still have things like deformation models, like
> NAD83(2011) that is used for most people as a static CRS, but is more a
> snapshot of a dynamic CRS with a conventional reference epoch of 2010.0).
>
> That said, I'll probably drop the commit for KML as it is clearly a hack,
> but I think support for GeoJSON, which is cleaner, could still be useful at
> some point as it is widely used as an exchange format. But I'll defer for
> GeoJSON until we see if the *OGC Features* and *Geometries JSON* SWG comes
> with something regarding this.
>
> Even
> Le 27/05/2021 à 19:24, Sean Gillies via gdal-dev a écrit :
>
> Hi all,
>
> I've got a suggestion about limiting the number of formats.
>
> GeoJSON and KML don't need support for a coordinate epoch. Both of these
> are pretty cleared intended for low accuracy data (1-2 meters). KML and
> GeoJSON don't support any CRS other than OGC:CRS84, which uses (it has been
> pointed out many times) a datum ensemble that RFC 81 does not intend to
> address. Right, Even? So let's leave these formats the way they are.
>
> GML, GeoPackage, PostGIS, and the actively used raster formats like
> GeoTIFF seem like the important ones to me.
>
> On Thu, May 27, 2021 at 7:40 AM Even Rouault 
> wrote:
>
>> Hi,
>>
>> - merging the underlying API without any format support is I believe of
>> little interest. So I'll wait for at least one format (likely
>> GeoPackage) to have merged in the master of their specification an
>> official way of storing the coordinate epoch. I've also prepared an
>> enhancement of the GeoTIFF spec regarding this
>> (https://github.com/opengeospatial/geotiff/pull/99) that will likely be
>> discussed at the next OGC GeoTIFF SWG meeting.
>>
>> - I've just chatted with Howard and a good compromise could be that for
>> formats that will have an official way of storing the coordinate epoch,
>> we store it by default (when it is set), and for formats that we
>> unilaterally extend (GML, KML, GeoJSON, etc.), we require an explicit
>> GDAL_ENABLE_COORD_EPOCH_STORAGE=YES configuration option to be set
>> (default would be NO). We might revisit in the future the default value.
>>
>> - On the vector side of things, things will probably get more
>> complicated for some drivers, as it is likely that Spatialite (see
>> https://groups.google.com/g/spatialite-users/c/NtbuBpRzYBE) or PostGIS
>> might have per-geometry coordinate epoch, not at the layer level. But I
>> don't think that would invalidate having support for it at the layer
>> level (those database already support a per-geometry CRS, but GDAL
>> support for that is probably lacking a bit in those drivers), as a
>> OGRGeometry can potentially be associated with its own
>> OGRSpatialReference object.
>>
>> Even
>>
>> Le 27/05/2021 à 15:01, Howard Butler a écrit :
>> >
>> >> On May 26, 2021, at 8:33 PM, Nyall Dawson 
>> wrote:
>> >>
>> >> Can I make the suggestion that a subset of
>> >> https://github.com/OSGeo/gdal/pull/3827 could be created and be merged
>> >> on its own? Specifically the commits which add the underlying API for
>> >> GDAL to handle epochs should be controversy-free and suitable for
>> >> merge outside of the larger/trickier question of patching in support
>> >> to the data formats.
>>

Re: [gdal-dev] Call for discussion on RFC 83: guidelines for the use of GDAL project sponsorship

[Sean Gillies via gdal-dev]

Hi Even,

I've got two questions. I don't think they need answers before we vote, but
I'm curious if asking them leads to any useful discussion.

Will we strictly require project proposals to be submitted and approved
before work starts? Will we allow works in progress to apply for funding?

Should we address, in the text of the RFC, the possibility of conflicting
proposals? Should we limit the number of proposals per person/organization
to minimize conflicts?

> Applicants will provide the amount to be funded. Proposals may be put
together by one or several individuals (in the later case, to be determined
if we can let the team have a "invoicing point of contact" and let them
arrange how to dispatch it amongst members, or if each team member should
ask for its part of funding). An applicant may submit proposals for several
subjects.

On Wed, Jun 2, 2021 at 6:02 AM Even Rouault 
wrote:

> Hi,
>
> are there any remaining comments before we put that to vote ?
>
> Even
>
> Le 19/05/2021 à 14:46, Even Rouault a écrit :
> > Hi,
> >
> > in parallel to finalizing the last steps to get the relationship with
> > NumFOCUS fully operational, here's a new RFC for your consideration to
> > give guidelines on how we can use the sponsorship funds, who can
> > apply, etc. Input welcome.
> >
> > https://github.com/OSGeo/gdal/pull/3855
> >
> > Even
> >
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 81: support for coordinate epochs in geospatial formats

[Sean Gillies via gdal-dev]

Hi all,

I've got a suggestion about limiting the number of formats.

GeoJSON and KML don't need support for a coordinate epoch. Both of these
are pretty cleared intended for low accuracy data (1-2 meters). KML and
GeoJSON don't support any CRS other than OGC:CRS84, which uses (it has been
pointed out many times) a datum ensemble that RFC 81 does not intend to
address. Right, Even? So let's leave these formats the way they are.

GML, GeoPackage, PostGIS, and the actively used raster formats like GeoTIFF
seem like the important ones to me.

On Thu, May 27, 2021 at 7:40 AM Even Rouault 
wrote:

> Hi,
>
> - merging the underlying API without any format support is I believe of
> little interest. So I'll wait for at least one format (likely
> GeoPackage) to have merged in the master of their specification an
> official way of storing the coordinate epoch. I've also prepared an
> enhancement of the GeoTIFF spec regarding this
> (https://github.com/opengeospatial/geotiff/pull/99) that will likely be
> discussed at the next OGC GeoTIFF SWG meeting.
>
> - I've just chatted with Howard and a good compromise could be that for
> formats that will have an official way of storing the coordinate epoch,
> we store it by default (when it is set), and for formats that we
> unilaterally extend (GML, KML, GeoJSON, etc.), we require an explicit
> GDAL_ENABLE_COORD_EPOCH_STORAGE=YES configuration option to be set
> (default would be NO). We might revisit in the future the default value.
>
> - On the vector side of things, things will probably get more
> complicated for some drivers, as it is likely that Spatialite (see
> https://groups.google.com/g/spatialite-users/c/NtbuBpRzYBE) or PostGIS
> might have per-geometry coordinate epoch, not at the layer level. But I
> don't think that would invalidate having support for it at the layer
> level (those database already support a per-geometry CRS, but GDAL
> support for that is probably lacking a bit in those drivers), as a
> OGRGeometry can potentially be associated with its own
> OGRSpatialReference object.
>
> Even
>
> Le 27/05/2021 à 15:01, Howard Butler a écrit :
> >
> >> On May 26, 2021, at 8:33 PM, Nyall Dawson 
> wrote:
> >>
> >> Can I make the suggestion that a subset of
> >> https://github.com/OSGeo/gdal/pull/3827 could be created and be merged
> >> on its own? Specifically the commits which add the underlying API for
> >> GDAL to handle epochs should be controversy-free and suitable for
> >> merge outside of the larger/trickier question of patching in support
> >> to the data formats.
> > :thumbsup:
> >
> > As for the patching of data formats with GDAL application-specific
> metadata, as I said, I don't have a better option, but I'm satisfied if the
> process of writing epochs into metadata is opt-in (some kind of global CRS
> option? we already have magic switches like that).
> >
> > Howard
> >
> >
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 81: support for coordinate epochs in geospatial formats

[Sean Gillies via gdal-dev]

Hi Even, Howard:

I'm inclined to approve, but I feel like there should be more discussion,
not just among PROJ developers and developers of cutting edge formats. We
should work to draw a wider group in on this.

On Thu, May 13, 2021 at 10:01 AM Even Rouault 
wrote:

> Howard,
> > It is magical
> > ---
> >
> > If you have GDAL-extended versions of a few select data formats and you
> have the correct chain of PROJ and GDAL, the behavior of your coordinates
> is going to change for various transformations. This could be confusing and
> challenging to track down in debugging scenarios. The discrepancies between
> doing the same transformations in different softwares will be especially
> noticeable.
> Having different results in coordinate transformations due to have will
> not be much different than using different PROJ versions using different
> versions of the EPSG database, possibly with different set of grids
> installed.
>

Howard, are you suggesting that there should be a configuration option to
opt in to this new feature?

A surprise 1-2 meter shift is going to break builds and applications, I
agree. I think a lot of us are tolerating inaccuracy due to using
time-varying CRS but are going to be caught out by changes in the actual
numbers.


> >
> > It extends existing formats in GDAL's own way
> > ---
> >
> > Are there many other cases where GDAL augments and extends behavior of
> formats by bolting on metadata bits? I can think of some GeoTIFF tags where
> GDAL has done this in the past. Some of them have been adopted
> industry-wide, but most have not. We definitely haven't done that to a long
> list of formats like this RFC proposes to do.
> We are not going to invent a new raster or vector format just to add a
> coordinate epoch into it, right ? So this proposal does minor and
> backward compatible enhancements to popular existing formats.
> >
>

For GeoJSON, at least, I think proposing a "coordinate_epoch" property is
pragmatic. This doesn't do anything for GeoJSON feature sequences, though.
And it doesn't do anything about data that's already out there that will
never be re-processed.

Now that I think about it, I think the RFC should say more about what it
will do for the ensemble OGC:CRS84.


> > No corresponding socializing activity
> > -
> >
> > Is GDAL going to go to the GeoJSON, GeoPackage, GeoTIFF, Flatgeobuf,
> GML, JPEG2000, KML, and Shapefile communities and advocate for these
> improvements? It would be a lot of time and effort to go back after the
> fact and officially augment all of these formats with epoch metadata.  Many
> of these are never going to have new versions either, so there isn't much
> hope of a new format version coming along with support for coordinate
> epochs.
> Well, this is a public RFC for everybody awareness, and there will be a
> page on GDAL doc. And I've created tickets on the GeoTIFF and GeoPackage
> OGC github repos pointing to it. I don't necessarily expect much follow
> up from them though, but at least we'll have tried to make advance the
> topic a bit.
> >
> > Is the format of epoch standardized?
> > -
> >
> > The proposed epoch format, such as 2021.3, *looks* like a floating point
> number, but it isn't really, is it? Do you ever need more precision than a
> year and a day? *shrug* It seems like it's own special time format. Is
> there a standard time format that could instead be used here?
>
> This format is a floating point number and is the accepted way of
> encoding coordinate epochs. The after decimal point part represents the
> fraction of the year. It is referenced in "OGC Abstract Specification
> Topic 2: Referencing by coordinates"
> (https://docs.opengeospatial.org/as/18-005r4/18-005r4.html). In table 4:
> coordinateEpoch: "date at which coordinates are referenced to a dynamic
> coordinate reference system, expressed as a decimal year in the
> Gregorian calendar. Example: 2017-03-25 in the Gregorian calendar is
> epoch 2017.23."
>
> (31+28+25) / 365. = 0.23...
>
> Two digits after the decimal point should be enough in practice. For
> 'slow' and continuous phenomenons like plate drift motion, an error of a
> couple days will not change the result by more than a fraction of
> millimetre. If you use a deformation model that takes into account
> earthquakes however, you could perhaps need to add a 3rd digit to
> identify precisely the day.
>
> Even
>

To me, it seems that the parameterization or description of coordinate
epochs in OGC 19-005r4 is a bit sketchy.

Even, are you saying that coordinate shifts in PROJ are entirely functions
of the datetime delta since the coordinate epoch?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Sean Gillies via gdal-dev]

Hi  Aaron, and everyone,

It seems like interoperability could be harmed if we release GDAL with a
JP2 driver that writes JPEG 2000 files that the main open source JP2 driver
can't read. Would it make sense to add compatibility to OpenJPEG before the
PR gets merged? Or are we already in a state of inoperability between
JP2ECW, JP2MRSID, and JP2OpenJPEG?

On Mon, May 3, 2021 at 1:52 PM Aaron Boxer  wrote:

> Hi Jean-Roc,
> Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
> it only supports JPEG 2000 Part 1. I am sure that this will eventually be
> integrated,
> but it isn't there at the moment.
> Cheers,
> Aaron
>
> On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml) <
> jrmorreale...@enoreth.net> wrote:
>
>> Hi Aaron,
>>
>> Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
>>
>> Regards,
>> Jean-Roc Morreale
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.3.0 RC1 available

[Sean Gillies via gdal-dev]

Thanks for explaining, Even. Makes sense to me.

On Wed, Apr 28, 2021 at 2:59 AM Even Rouault 
wrote:

> Sean,
>
> This was the trend of the previous cycles. I see I also produced a 3.1.4
> at about the same time of 3.2.0. This offers 6 month of support for a given
> feature release. Otherwise that cuts it to 4 months.
>
> Even
> Le 28/04/2021 à 03:59, Sean Gillies a écrit :
>
> Hi Even,
>
> I see that there's also a 3.2.3 candidate. Usually we discontinue the
> previous minor version, right? Why in this case are we have a 3.3.0rc1 and
> a 3.2.3rc1?
>
> On Mon, Apr 26, 2021 at 6:57 AM Even Rouault 
> wrote:
>
>> Hi,
>>
>> I have prepared a GDAL/OGR 3.3.0 release candidate. The feedback on beta1
>> was useful to spot and fix a few bugs.
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.3.0 RC1 available

[Sean Gillies]

Hi Even,

I see that there's also a 3.2.3 candidate. Usually we discontinue the
previous minor version, right? Why in this case are we have a 3.3.0rc1 and
a 3.2.3rc1?

On Mon, Apr 26, 2021 at 6:57 AM Even Rouault 
wrote:

> Hi,
>
> I have prepared a GDAL/OGR 3.3.0 release candidate. The feedback on beta1
> was useful to spot and fix a few bugs.
>
> The changes between beta1 and RC1 are identified in:
>
> https://github.com/OSGeo/gdal/commit/a882173d688e6042c9dc7b1b57d8d169b04abe2b
> Note a potential impact on packaging recipees regarding a few esoteric
> untested/undocumented python scripts,
> now moved as "samples" scripts of the gdal-utils package
>
> Pick up an archive among the following ones (by ascending size):
>
>https://download.osgeo.org/gdal/3.3.0/gdal-3.3.0rc1.tar.xz
>https://download.osgeo.org/gdal/3.3.0/gdal-3.3.0rc1.tar.gz
>https://download.osgeo.org/gdal/3.3.0/gdal330rc1.zip
>
> A snapshot of the gdalautotest suite is also available :
>
>https://download.osgeo.org/gdal/3.3.0/gdalautotest-3.3.0rc1.tar.gz
>https://download.osgeo.org/gdal/3.3.0/gdalautotest-3.3.0rc1.zip
>
> A snapshot of the documentation is at:
>
>https://download.osgeo.org/gdal/3.3.0/gdal330doc.zip
>
> GDAL-GRASS plugin:
>
>https://download.osgeo.org/gdal/3.3.0/gdal-grass-3.3.0.tar.gz
>
> NEWS at:
>
>https://github.com/OSGeo/gdal/blob/v3.3.0RC1/gdal/NEWS
>
> I'll call for a vote promoting it to final later this week if no
> serious problems are reported before.
>
> release/3.3 branch has been created for bug fixes and master is now
> 3.4.0dev
>
> Best regards,
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> _______
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] how to use tiledb datasets on gcs?

[Sean Gillies via gdal-dev]

Hi Vincent, Even.

Why would we do this when /vsigcs/... should work? Letting individual
drivers set their own conventions for dataset names seems, to me, to hurt
long-term maintenance and interoperability.

On Fri, Apr 23, 2021 at 7:34 AM Vincent Schut 
wrote:

> Thanks for confirming, Even. That doesn't look too difficult. I'll give it
> a try.
>
> On 4/23/21 3:09 PM, Even Rouault wrote:
>
> I guess you should add something similar to
> https://github.com/OSGeo/gdal/commit/3623f9c91a2c513af204d30fe25314dbe5c7b9be
> for /vsigs
> Le 23/04/2021 à 14:59, Vincent Schut a écrit :
>
> On 4/23/21 2:17 PM, Vincent Schut wrote:
>
> Hi, how should I specify a tiledb dataset's url that resides on gcs
> (google cloud storage) to gdal? I've tried several combinations of gcs://,
> /vsigs/, prefixed with TILEDB:// or not, but no luck. I've looked in the
> driver source, and apparently there is only a /vsis3/ -> tiledb uri
> translation, but no equivalent gcs one?
>
> To clarify this a bit: writing works:
>
> gdal_translate -of tiledb -co COMPRESSION=ZSTD
> S2B_MSIL1C_20210227T032659_N0209_R018_T47MQV_20210227T072433.tif
> gcs://s11-dev-vincent-tiledb-test-public/S2B_MSIL1C_20210227T032659_N0209_R018_T47MQV_20210227T072433.tiledb
>
> and the file is correctly created in the bucket. However, when I want to
> open it, it fails:
>
> gdalinfo -if tiledb
> gcs://s11-dev-vincent-tiledb-test-public/S2B_MSIL1C_20210227T032659_N0209_R018_T47MQV_20210227T072433.tiledb
> ERROR 4:
> gcs://s11-dev-vincent-tiledb-test-public/S2B_MSIL1C_20210227T032659_N0209_R018_T47MQV_20210227T072433.tiledb:
> No such file or directory
> gdalinfo failed - unable to open
> 'gcs://s11-dev-vincent-tiledb-test-public/S2B_MSIL1C_20210227T032659_N0209_R018_T47MQV_20210227T072433.tiledb'.
>
> Is there any way to correctly specify this tiledb url on gcs for gdal?
>
> Vincent.
>
> --
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Registered Content-Type for VRT?

[Sean Gillies via gdal-dev]

After re-reading https://tools.ietf.org/html/rfc6838, I remember that
"applications/gdalvrt+xml" would necessarily be a "standards tree" media
type and would need to be published in an RFC. The "vendor tree" is more
open. So something like "application/vnd.gdalvrt+xml" is more appropriate.
If you want this, Howard, you should fill out the form linked in RFC 6838.
It doesn't take long to get feedback and approval on vendor tree media
types.

On Tue, Apr 20, 2021 at 10:09 AM Even Rouault 
wrote:

> application/gdalvrt+xml would just mean what the organization that defines 
> the format is. People could reimplement that from scratch
>
> Note that there is potentially the OGR VRT format (that could go as 
> application/ogrvrt+xml), but that is a separate schema and likely less used 
> than its raster counterpart.
>
> Le 20/04/2021 à 18:04, Sean Gillies via gdal-dev a écrit :
>
> Hi Jukka,
>
> It's possible that ArcGIS has independently implemented VRT, but I think
> it's more likely that it delegates to GDAL. I think it would be possible to
> craft a VRT doc that would reveal some details -- such as one that uses
> GDAL's embedded Python pixel functions to dump some system information --
> but I don't have access to ArcGIS to try it.
>
> On Mon, Apr 19, 2021 at 4:01 PM jratike80 <
> jukka.rahko...@maanmittauslaitos.fi> wrote:
>
>> Hi,
>>
>> Do you count ArcMap or ArcGIS Pro as self standing software or rather as
>> applications that use GDAL? They read VRTs pretty well but for writing
>> some
>> developer tools are required
>>
>> https://pro.arcgis.com/en/pro-app/latest/help/data/imagery/supported-raster-dataset-file-formats.htm
>> .
>>
>> -Jukka Rahkonen-
>>
>>
>> GDAL - Dev mailing list wrote
>> > Maybe application/gdalvrt+xml? As far as I know, GDAL is the only
>> software
>> > that plays VRTs, and probably the only software that ever can,
>> considering
>> > the extensions and embedded Python stuff.
>> >
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Registered Content-Type for VRT?

[Sean Gillies via gdal-dev]

Hi Jukka,

It's possible that ArcGIS has independently implemented VRT, but I think
it's more likely that it delegates to GDAL. I think it would be possible to
craft a VRT doc that would reveal some details -- such as one that uses
GDAL's embedded Python pixel functions to dump some system information --
but I don't have access to ArcGIS to try it.

On Mon, Apr 19, 2021 at 4:01 PM jratike80 <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi,
>
> Do you count ArcMap or ArcGIS Pro as self standing software or rather as
> applications that use GDAL? They read VRTs pretty well but for writing some
> developer tools are required
>
> https://pro.arcgis.com/en/pro-app/latest/help/data/imagery/supported-raster-dataset-file-formats.htm
> .
>
> -Jukka Rahkonen-
>
>
> GDAL - Dev mailing list wrote
> > Maybe application/gdalvrt+xml? As far as I know, GDAL is the only
> software
> > that plays VRTs, and probably the only software that ever can,
> considering
> > the extensions and embedded Python stuff.
> >
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC80

[Sean Gillies via gdal-dev]

Hi all,

I was unable to comment on https://github.com/OSGeo/gdal/pull/3682 until
now. Sorry about that.

I think we should consider removing the 2nd paragraph under "Advisory
Board" to help keep the maintenance sponsorship and new feature development
concerns separated. It would be enough, I think, for GDAL maintainers to
present to a board several times a year to keep themselves accountable to
sponsors, and that new feature brainstorming -- which is out of scope --
could be done elsewhere.

On Fri, Apr 16, 2021 at 8:50 AM Even Rouault 
wrote:

> Hi,
>
> I hereby motion to adopt RFC 80:
>
> https://github.com/OSGeo/gdal/pull/3682
>
> This also implies adopting the "GDAL Sponsorship Prospectus"  at
>
>
> https://docs.google.com/document/d/1yhMWeI_LgEXPUkngqOitqcKfp7ov6WsS41v5ulz-kd0/edit?ts=60777468#
> whose a PDF version will be uploaded on the website
> (https://github.com/OSGeo/gdal/pull/3681 to be updated)
>
> and the proposed answers to the NumFOCUS application form are at:
>
> https://docs.google.com/document/d/1bc5jdpCe1axdyBHxbJnun7e0DTyDoZI_eFYgJYnOhB8/edit
>
> which will be submitted consequently.
>
> Starting with my +1
>
> Even
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Registered Content-Type for VRT?

[Sean Gillies via gdal-dev]

I haven't seen a VRT media type in the wild, nor is there one on
https://www.iana.org/assignments/media-types/media-types.xhtml.

Maybe application/gdalvrt+xml? As far as I know, GDAL is the only software
that plays VRTs, and probably the only software that ever can, considering
the extensions and embedded Python stuff.

On Mon, Apr 19, 2021 at 7:58 AM Howard Butler  wrote:

> Is there a content being used in the wild for VRT? What would be a
> reasonable one to register?
>
> The scenario where I would find this useful is to have an application such
> as QGIS be able to register to open VRTs that were downloaded from a
> server.
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: RFC 78: gdal-utils package

[Sean Gillies]

It looks to me like this RFC commits the project to spending more energy on
language bindings for the next few versions. I'm not enthusiastic about
that, but am not super opposed. So I'm a -0.

On Wed, Mar 24, 2021 at 1:58 PM Kurt Schwehr  wrote:

> +0  KurtS
>
> I have some vague undefined unease about this.  Even's comments help
> diminish that some.  I don't see strong enough arguments for me to vote for
> it.
>
> On Wed, Mar 24, 2021 at 12:03 PM Even Rouault 
> wrote:
>
>> Hi Idan,
>>
>>
>> Motion:
>>
>> Adopt RFC 78: gdal-utils package
>> <https://github.com/talos-gis/gdal/blob/Branch_rfc78_py_modules/gdal/doc/source/development/rfc/rfc78_gdal_utils_package.rst>
>>  (formatted
>> version).
>>
>> +1.
>>
>> Sorry for the late reply.
>>
>> I'd like some efforts on the documentation front regarding this new
>> addition. The new doc page hhttps://gdal.org/api/python.html (initiated
>> from the swig/python/README.rst. not sure how we could avoid duplication of
>> content in the tree. perhaps some inclusion of README.rst from the docs)
>> would need to be enriched with details from the RFC, to explain how the 2
>> pypi packages relate, and the potential issues in the workflows you
>> indicate.
>>
>> The HOWTO-RELEASE procedure will likely need to be adjusted.
>>
>> I'll appreciate too that you actively support users in github issues /
>> email traffic on this topic once this is deployed in a release.
>>
>> Best regards,
>>
>> Even
>>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: RFC 78: gdal-utils package

[Sean Gillies via gdal-dev]

Hi,

On Wed, Mar 24, 2021 at 2:51 PM Alan Snow  wrote:

> One recommendation I have for this RFC would be to remove gdal_utils
> entirely from the main GDAL repository and into its own repository.
> The main reason would be to test against multiple versions of GDAL to
> ensure compatibility. Compatibility across versions is a main goal of this
> RFC if I understand correctly, so that is why I bring it up.
>
> Hope that is helpful,
> Alan
>

Thank you for bringing this up and joining the discussion, Alan!. I agree
that gdal_utils is a lot like projects we work on together in this respect.

I scanned the RFC and am confused about what I read in
https://github.com/talos-gis/gdal/blob/Branch_rfc78_py_modules/gdal/doc/source/development/rfc/rfc78_gdal_utils_package.rst#how-to-upgrade-the-utils-without-upgrading-the-bindings
.

> pip install a wheel overwrites whichever files already exist (even if
installed by a different package) If you pip install gdal then pip install
gdal-utils you'd get the utils from gdal-utils. If later you do again pip
install gdal with a different version then you'd get the utils from gdal
again, and so on. (it doesn't seem that it matters which version is a
bigger number, just which one you installed later)

How certain are we of this? I am not 100% certain that this is true or that
the behavior of pip here is totally specified. Even if it is true, it seems
like there is a lot of potential for confusion. I think it would be much
better for users if we didn't have 2 distributions contending for the same
namespace.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] gdal_rasterize expected behaviour

[Sean Gillies via gdal-dev]

Hi Hug,

GDALRasterizeGeometries takes an array of geometries and iterates over them
from start to end, burning them into the raster one at a time. With a
strictly ordered vector layer, you can expect the later shapes to be burned
over the earlier ones.

On Mon, Mar 8, 2021 at 7:00 AM Hugues François 
wrote:

> Hello,
>
> I don't know exactly where I should ask this question and I wasn't able to
> find the answer when searching on the internet (maybe I didn't use the
> right keywords).
>
> I have to rasterize a vector layer, made of polygons, which resolution is
> finer than the destination raster I have to use so that I want to use the
> "all_touched" option. But since my the raster resolution is coarser than
> the vector one, it may happens sometimes that a single pixel is touched by
> several polygons (and sometime none of them encompass the pixel center
> point). In this case, how does the value burned to the raster is chosen
> among the candidates?
>
> Best regards,
> Hug
>
>

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion (V2): remove and deprecate a few drivers

[Sean Gillies via gdal-dev]

+1

On Thu, Mar 4, 2021 at 9:33 AM Even Rouault 
wrote:

> Hi,
>
> Updating my yesterday motion with the feedback received (only second
> bullet updated with a more restricted set of drivers)
>
> Motion:
>
> - remove the vector drivers BNA, AeronavFAA, HTF, OpenAir, SEGUKOOA,
> SEGY, SUA, XPlane and raster drivers BPG, E00GRID, EPSILON,
> IGNFHeightASCIIGrid, NTv1. They have all been authored by myself and I'm
> not aware of them having been much used or being still in use.
> Implemented in https://github.com/OSGeo/gdal/pull/3373. They (driver
> code, doc and tests) have been moved to the
> https://github.com/OSGeo/gdal-extra-drivers
>
> - deprecate the raster drivers DODS, JPEG2000 (superseded per
> JP2OpenJPEG), JPEGLS, MG4LIDAR, FUJIBAS, IDA, INGR, ZMAP and vector
> driver ARCGEN, ArcObjects, CLOUDANT, COUCHDB, DB2, DODS, FME, GEOMEDIA,
> GTM, INGRES, MONGODB (superserded per MongoDBV3), REC, WALK . They will
> now be disabled at runtime by default, with an explicit error message
> when they are triggered, unless the
> GDAL_ENABLE_DEPRECATED_DRIVER_{drivername}
> configuration option is set to YES, and will be removed in GDAL 3.5.
> Implemented in https://github.com/OSGeo/gdal/pull/3505
>
> Starting with my +1
>
> Even
>
> --
> http://www.spatialys.com
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for discussion on RFC 79: Listing of Service Providers on GDAL website

[Sean Gillies via gdal-dev]

Hi Jukka,

On Wed, Feb 24, 2021 at 12:20 AM jratike80 <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> GDAL - Dev mailing list wrote
> > Hi Even,
> >
> > On Tue, Feb 23, 2021 at 2:56 AM Even Rouault 
>
> > even.rouault@
>
> > 
> > wrote:
> >
> >> Hi,
> >>
> >> Please find https://github.com/OSGeo/gdal/pull/3473 which proposes to
> >> list
> >> service providers offering GDAL related services on the GDAL website. As
> >> mentioned in the RFC, this is a straightfoward port of the equivalent
> >> adopted
> >> RFC of the Mapserver project.
> >>
> >> Even
> >>
> >
> > I'd like to hear more about the Mapserver project service provider list.
> > Does it work? Is it kept up to date or does it get stale? Does the
> > community find that it effectively steers $$ to consultants that fix bugs
> > and maintain the project? With the difference in image/logo sizes, is the
> > intent to make this look a bit like a conference sponsor page?
> >
> > --
> > Sean Gillies
>
>
> Hi,
>
> Maybe the number of new potential customers searching for consultants to do
> Mapserver development is not so huge at the moment and the old ones know
> their contacts already. However, I feel that the service provider list does
> work for the Geoserver project. Probably mostly not so that users start by
> browsing the list and pick a contractor. It is rather so that core
> Geoserver
> developers know that often there is a minimal chance that some bug will be
> fixed or some feature request will be implemented just because there is an
> open ticket about that. Then they can inform the user about the reasonable
> alternatives like in this example that I picked from a mailing list:
>
> "That does not mean we cannot work around a deficiency in a rogue server,
> but be prepared to issue your PR or fund development though commercial
> services: http://geoserver.org/support/;
>
> For my mind that is also fair for the users and gives more hope than the
> comment on the OSSIM site, that is otherwise my absolute favorite:
>
> "We encourage users to use the software as-is or become an active
> contributor."
>
> -Jukka Rahkonen-
>

Thanks for the explanation. I agree that we can't fairly tell new people
that they may have to pay for help without showing them who they may hire.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for discussion on RFC 79: Listing of Service Providers on GDAL website

[Sean Gillies via gdal-dev]

Hi Even,

On Tue, Feb 23, 2021 at 2:56 AM Even Rouault 
wrote:

> Hi,
>
> Please find https://github.com/OSGeo/gdal/pull/3473 which proposes to
> list
> service providers offering GDAL related services on the GDAL website. As
> mentioned in the RFC, this is a straightfoward port of the equivalent
> adopted
> RFC of the Mapserver project.
>
> Even
>

I'd like to hear more about the Mapserver project service provider list.
Does it work? Is it kept up to date or does it get stale? Does the
community find that it effectively steers $$ to consultants that fix bugs
and maintain the project? With the difference in image/logo sizes, is the
intent to make this look a bit like a conference sponsor page?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] zarr driver PR

[Sean Gillies]

Hi all,

Also deep in our recent long thread about removing drivers from GDAL is
mention of a new PR for a "zarr" driver.

https://github.com/OSGeo/gdal/pull/3411

I'd like to see more discussion of the scope of this work and plans for
future development and support before a zarr driver is added to GDAL.

Is the zarr driver not going to support version 2 of the zarr spec? I that
v2 would be difficult to implement as a GDAL driver as it is extremely
flexible and extensible. For example, I've written custom compressor and
storage for use in a project at work and can't imagine how a driver written
in C++ would understand it.

Is the zarr driver based on
https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html?
That work seems not yet finished. Is it premature to add a driver to GDAL
for a format that isn't yet specified and may yet be changed? Or should we
expect only minor changes at this point?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Requiring RFCs for new format drivers

[Sean Gillies]

Hi all,

Buried in a recent thread are some comments in support of requiring a GDAL
RFC for new format drivers. On the surface it seems one way to keep GDAL's
mass and surface area from growing without bounds. It's also been pointed
out that at least one of GDAL's formats was never intended for use.
Requiring RFCs might also help prevent proliferation of needlessly
different web API clients and JSON formats. Or it might not, I'd love to
read what others have to say about this.

Requiring an RFC could hurt developers, though, since pay for format driver
development (and other new features) is what's largely been subsidizing bug
fixes and general maintenance of the project. I'm sure that others of you
can think of consequences and side effects that I haven't.

I suppose we should probably have an RFC about RFCs if this idea isn't
immediately shot down. Kurt Schwehr has made a template for what a driver
RFC might look like and I suggest we discuss that as well.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Sean Gillies via gdal-dev]

Hi Tamas,

Are you suggesting that a RFC be required for a new driver? I would support
this 100%.

On Wed, Jan 27, 2021 at 2:17 PM Tamas Szekeres  wrote:

> David,
>
> Up to this time the driver writers were highly welcomed to author new
> drivers for the project and these effort didn't require a separate RFC in
> terms of the Project Management Committee Guidelines
> <https://gdal.org/development/rfc/rfc1_pmc.html> document. Adding new
> drivers hasn't been considered to be substantial changes or something that
> causes to change the API or brings in backwards incompatibility issues.
> However in a real deployment situation the drivers may cause issues for
> the developers, end users and package maintainers from several aspects like
> so:
>
> 1. The drivers may depend on external libraries and we don't want to link
> against those libraries in all cases.
> 2. The external libraries may cause license incompatibilities, ie linking
> against that may change the license terms of the overall project.
> 3. Not all of the drivers are required in a particular solution (that
> applies to the built in drivers, too). In a real situation the application
> may use only a limited set of drivers and the existence of the unwanted
> drivers may cause some performance degradation.
> 4. Implementing custom compilations (and omitting unwanted drivers from
> the build) may be problematic for most of the users or projects utilizing
> gdal as a sub-project.
>
> In my opinion, the current solution of incubating new drivers is fairly
> minimalistic, we might probably want to know what kind of format is being
> addressed, who is the audience, how amount of user might be of interest,
> how the licensing of the dependent project is looking like, is the
> dependent project (if any) maintained well enough and can be compiled to
> each supported platforms etc.
>
> But replying to the question, I think you shoud continue the effort, but
> consider to implement a plugin build for that. There are several existing
> drivers are compiled as plugins at the moment, so it should not be so
> difficult.
>
>
> Best regards,
>
> Tamas
>
>
> David Brochart  ezt írta (időpont: 2021. jan.
> 27., Sze, 15:29):
>
>> I am currently trying to add a Zarr driver to GDAL (see
>> https://github.com/OSGeo/gdal/pull/3411), but from what I can see the
>> trend is to remove drivers, so I'm now wondering it I should pursue this
>> effort. I'm relatively new to GDAL, but from my point of view supporting a
>> lot of formats is part of GDAL's success, so maybe the real focus should be
>> on implementing a plugin mechanism that would allow driver development
>> independently from core GDAL. Again, I might not have enough background to
>> give valuable feedback.
>>
>> Regards,
>>
>> David.
>>
>> On Wed, Jan 27, 2021 at 12:34 PM thomas bonfort 
>> wrote:
>>
>>>
>>>
>>> On Tue, Jan 12, 2021 at 11:36 PM Even Rouault <
>>> even.roua...@spatialys.com> wrote:
>>>
>>>>
>>>> The issue with esoteric/legacy drivers is not that much maintenance of
>>>> the
>>>> actual code of the drivers, in the sense of dealing with bug reports,
>>>> questions, etc. (pretty sure they are none for the ones I listed). Most
>>>> of
>>>> them must work reasonably well and be feature complete, and most
>>>> vulnerabilities have now been fixed. My concern is that this legacy
>>>> code has
>>>> indirect costs on other GDAL developers and users. The psychological
>>>> cost I
>>>> mentionned. Let's say someone want to turn on higher warning levels,
>>>> and that
>>>> this breaks in tens of drivers. Would he have to ping every maintainer
>>>> and
>>>> wait for them to address the issue ? Or maybe he will just give up.
>>>> Similarly
>>>> for breaking changes in the driver API. As Sean mentionned, this is
>>>> probably a
>>>> serious obstacle to growing up the core development team.
>>>>
>>>
>>> Given that we have a relatively easy way to disable individual drivers
>>> at compile time, could we expand on this mechanism to mark problematic
>>> drivers as "orphaned" in this case? The code would still live in the repo
>>> but would be effectively disabled until someone with sufficient interest
>>> invests the time/money to update the problematic code and re-enable it.
>>> This will of course create some overhead to keep track of which drivers
>>> have been orphaned from one release to the next, create github
>>> issues/labels to list which drivers need work to be re-enabled, etc... But
>>> it shifts the burden of maintaining the codebase of 250 drivers from the
>>> core team over to the people/companies who actually need them. I'd be
>>> willing to contribute the scripts that could ease the process of
>>> orphaning/reintegrating a specific driver.
>>>
>>> Regards,
>>> Thomas
>>>
>>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Mapbox Vector Tiles Driver Support for Styles

[Sean Gillies via gdal-dev]

Hi,

No, the MVT driver is only concerned with features in a protobuf file:
https://github.com/mapbox/vector-tile-spec/tree/master/2.1. It doesn't know
about Mapbox styles.

On Wed, Jan 20, 2021 at 2:53 PM Miller, Doug  wrote:

> How does the Mapbox Vector Tiles Driver take into account styles (i.e.
> style.json) when reading?  Does it even have that functionality?


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Driver maintenance - long-term solution ?

[Sean Gillies via gdal-dev]

Hi Howard and all,

On Wed, Jan 13, 2021 at 3:58 PM Howard Butler  wrote:

>
>
> On Jan 13, 2021, at 4:28 PM, Nyall Dawson  wrote:
>
> On Thu, 14 Jan 2021 at 06:24, David Strip  wrote:
>
> What is the path forward?  One path Howard suggests is establishing a
> foundation similar to that behind Qgis. Another alternative, probably far
> more controversial, is a license change.
>
>
> I'm pretty clueless regarding licenses -- but this is an interesting
> thought. I wonder if any new drivers added to GDAL could be done with
> a dual-licensing under both GPL + some other license which requires
> ongoing sponsorship of the GDAL project?
>
>
> License monkey business isn't viable in any way with GDAL. It would just
> create confusion and erode trust, which we can't get back if broken.
>

> The big organizations running 100,000,000s of CPU hours extracting
> information from imagery they're reading in COGs with GDAL need to be
> donating substantial resources into an organization that provides
> coordination. The last time I did a fund raise with gdalbarn.com I was
> called out for naming some of these organizations and expressing my
> disappointment they couldn't find a way to participate or simply ignored
> the request.  Maybe they will step forward this time around.
>

Maybe? Getting shamed feels bad and makes people bitter. The good news is
that the attrition rate in big organizations is pretty high and those
people may have left and been replaced by new people who we can approach
again. This turnover is also the bad news: people who have been
stakeholders in the past may have moved on as well.


> Whether it is in a new foundation or an existing one like NumFocus,
> substantial resources need to be dumped in a pot that are earmarked for
> supporting work that generates value for the project. Chasing new feature
> work to subsidize project maintenance activities is not sustainable in two
> directions – burn out for the maintainer and creeping feature-itis for the
> project.
>

Agreed. I'd like to try to improve this situation in 2021.


>
> It's clear what's happened in the past is a combination of luck and
> graciousness by both Frank and Even.
>
> Howard
>

Howard, you're so right. Not every industry enjoys a resource like GDAL.
It's a special project run by uniquely dedicated people.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Sean Gillies via gdal-dev]

Hi Even,

On Mon, Jan 11, 2021 at 7:44 AM Even Rouault 
wrote:

> Hi,
>
> trying to answer the different points raised up to now:
>
> - SVG: let's keep it as it is used. This is exactly the feedback I'm
> seeking
> for. I had developed this as a toy, crazy me, I won't do it anymore. No
> idea
> anyone was using it actually.


> - making those driver plugins. This would require significant work, and
> the
> purpose is to save work. Our current build infrastructure is not ready for
> easy plugification. And announcing they will be unmaintained is probably
> not
> efficient to know in advance the impact of the planned breakage. People
> don't
> read documentation. The only way to force people to react is to break them
> in
> some way.
>
> - regarding Python, I'm not sure what the question is exactly. If it was
> how
> to still enable the drivers candidate for retirement to work, then it is
> just
> gdal.SetConfigOption('GDAL_ENABLE_DRIVER_FOO', 'YES')
>

Making users explicitly turn these formats back on feels good to me. I
don't see a downside to it. We might want to consider one round of warnings
before setting this option's default to NO? Doing so would take care of the
operators of deployed applications who *do* pay attention to warnings.

Would it make any sense to retire read and write of formats on a different
schedule? The fewer SDTS files written in the next 6-12 months, the less
impact there will be when we remove the driver.

When the change is made to GDAL, I might bikeshed about the config option
name a bit.


> - once we have decided which drivers should be retired, I don't find
> moving
> them to some cemetery repository very useful. Because that would lack part
> of
> the build scripts. What would be more useful is to add a paragraph in the
> documentation that drivers FOO, BAR, BAZ were retired in GDAL 3.5. That
> way
> people know they have to download a GDAL tarball or checkout a git tag
> before
> that release, or download a past OSGeo-Live VM, or Conda package, etc...
> And
> they will get (hopefully) functional code. Much better than the cemetery
> approach.
>
> - regarding schedule:
>* GDAL 3.3, May 2021: version with drivers semi-retired
>* GDAL 3.4, November 2021: still with drivers semi-retired
>* GDAL 3.5, May 2022: retired drivers are gone
>  So we won't get much feedback from Ubuntu LTS april 2022, as at that
> date,
> the drivers will have to be retired for the 3.5 release.
>
> - Where are the costs in keeping these drivers ?
>* monetary: there is one, but I'm unable to quantify it
>* environmental: burn CPU cycles each time someone does a GDAL build
>* psychological: prevent developers from doing modernization in GDAL
> internals. When you know you have to change 250 drivers, you think twice
> before doing the change, and generally you conclude it is not worth it, or
> fallback to hacks to limit the amount of code change. We should probably
> trim
> down the list to 20 raster and vector drivers to have a real effect
> regarding
> that. For a next time :-)
>
> Even
>

Would we disable default compilation of these drivers in 3.3 as well? I'd
be in favor of that.

Thanks for writing that third bullet point. I've thought twice about
changes for exactly that reason.

Formats come and go. We can't expect the GDAL project and its maintainers
(mostly you!) to be the curators forever of all data.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] How to wrap a C++ library using GDAL in a Python library?

[Sean Gillies via gdal-dev]

Hi Mateusz,

The OSGeo sysadmins found something to correct in mailman configuration,
I'm giving that a test here.

On Wed, Dec 9, 2020 at 1:32 AM Mateusz Loskot  wrote:

> On Tue, 8 Dec 2020 at 23:16, Sean Gillies  wrote:
> >
> > The state of the art for very thin Python bindings for a C++ project
> seems to be pybind11. It's used for numpy's FFT module, based on pocketfft.
>
> My observations are similar to that and pybind11 is the currently
> popular solution.
>
> I personally like to stick to vanilla Python API though, not easy, but
> not a terrible experience either.
>
> > If some of you are surprised to see me post again on this thread, it's
> because my s...@mapbox.com address is getting marked as spam.
>
> FYI, your messages from that address always landed in my GMaill inbox
> marked as spam.
> That actually is the case for every message from @mapbox.com (e.g.
> from OSRM folks).
>
> Best regards,


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] How to wrap a C++ library using GDAL in a Python library?

[Sean Gillies]

Hi,

Rasterio is different, though. It spreads Python very thickly on top of
GDAL. The state of the art for very thin Python bindings for a C++ project
seems to be pybind11. It's used for numpy's FFT module, based on pocketfft.

If some of you are surprised to see me post again on this thread, it's
because my s...@mapbox.com address is getting marked as spam. I've been
having a private conversation with Even on this list that no one else has
seen. And I thought it was because the clarity and insight of my answers
rendered everyone incapable of comment :)

On Tue, Dec 8, 2020 at 3:06 PM Sean Gillies  wrote:

>
>
> -- Forwarded message -
> From: Brendan Ward 
> Date: Tue, Dec 8, 2020 at 11:25 AM
> Subject: Re: [gdal-dev] How to wrap a C++ library using GDAL in a Python
> library?
> To: Alex HighViz 
> Cc: gdal-dev@lists.osgeo.org 
>
>
> Alex,
>
> see rasterio: https://github.com/mapbox/rasterio
>
> It uses Cython (https://cython.readthedocs.io/en/latest/index.html) to
> wrap GDAL rasters and related functions for use in Python.  You can use
> Cython to wrap C / C++ libraries for use in Python; how much you wrap
> depends on the interface you want between the two.
>
> Hope that helps,
>
> Brendan
>
> On Tue, Dec 8, 2020 at 10:17 AM Alex HighViz 
> wrote:
>
>> Hi Paul,
>>
>> Yes there is a generic, "how do I expose a  C++ library to Python users"
>> question. But there is a GDAL specific issue that the main inputs and
>> outputs in my library are raster layers, and I am not sure how to pass
>> those. Especially if my library is using GDAL and the user is using GDAL
>> through Python. I can make my interface completely based on filenames, but
>> that seems inefficient. Is this not a more commonly encountered problem?
>>
>> Thanks, Alex
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> --
>> *From:* Paul Harwood 
>> *Sent:* Tuesday, December 8, 2020 4:46:07 PM
>> *To:* Alex HighViz 
>> *Cc:* gdal-dev@lists.osgeo.org 
>> *Subject:* Re: [gdal-dev] How to wrap a C++ library using GDAL in a
>> Python library?
>>
>> I may have misunderstood but I think you are asking the wrong community.
>>
>> You can take your own C++ library and make it available to a Python
>> library - see https://docs.python.org/3/extending/extending.html etc -
>> but this is not the community to ask for advice about that. You can, of
>> course, access GDAL in that c++ library using the c++ API but I don't think
>> that doing so would change how you expose the API from your library in
>> Python ...
>>
>> Perhaps you need to make your question more specific?
>>
>> On Tue, 8 Dec 2020 at 12:37, Alex HighViz 
>> wrote:
>>
>> Hello,
>>
>> Could somebody please put me on the right track with the following
>> problem?
>>
>> I have a C++ library that makes use of GDAL for processing raster maps
>> and I would like to wrap some of its features into a Python library to make
>> it accessible to a wider community.
>>
>> I would like my library to present functions that take rasters as input
>> and produce rasters as outputs.  I can modify the C++ side to have the
>> functions work on GDALRasterBands, GDALDatasets, or just on filenames. From
>> my perspective I'd prefer to write any wrapping / boiler plate in C++ and
>> have the Python parts as small as possible.
>>
>> I know this question has been asked before here, but I don't recall the
>> answer and could not find it again. A simple example of "best practice"
>> would be ideal.
>>
>>
>> With many thanks, Alex
>>
>>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] How to wrap a C++ library using GDAL in a Python library?

[Sean Gillies]

Hi Alex,

I observe more people using pybind11 these days.
https://github.com/pybind/pybind11.

Yours,


On Tue, Dec 8, 2020 at 5:37 AM Alex HighViz  wrote:

> Hello,
>
> Could somebody please put me on the right track with the following
> problem?
>
> I have a C++ library that makes use of GDAL for processing raster maps and
> I would like to wrap some of its features into a Python library to make it
> accessible to a wider community.
>
> I would like my library to present functions that take rasters as input
> and produce rasters as outputs.  I can modify the C++ side to have the
> functions work on GDALRasterBands, GDALDatasets, or just on filenames. From
> my perspective I'd prefer to write any wrapping / boiler plate in C++ and
> have the Python parts as small as possible.
>
> I know this question has been asked before here, but I don't recall the
> answer and could not find it again. A simple example of "best practice"
> would be ideal.
>
>
> With many thanks, Alex
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Querying RasterBands near edges

[Sean Gillies]

Hi Felix,

The rasterio Python API has a concept of a "boundless" read (see boundless
keyword under
https://rasterio.readthedocs.io/en/latest/api/rasterio._io.html#rasterio._io.DatasetReaderBase.read)
which sounds similar to what you're looking for. This is implemented using
a VRT and you might be able to do the same kind of thing with the GDAL
Python bindings.

1. Create a VRT with ample extra space around your raster and with a
suitable background value
2. Read from the VRT instead of the raster

You could even tile the VRT with multiple virtual copies of your raster if
you wanted a true wrap-around.

Hope this helps!

On Mon, Dec 7, 2020 at 1:59 PM Felix 
wrote:

> Hey,
>
> Our application requires us to fetch for many pairs of (latitude,
> longitude) all existing values within some radius from an existing raster
> dataset. Is it possible to read values from a GDALRasterBand given some
> general bounding box, even if it is at the edge of a dataset?
>
> One can easily convert between bounding boxes in geographical coordinates
> and pixels using Dataset::GetGeoTransform() and InvGeoTransform(), but
> GDALRasterBand/GDALMDArray/the Python bindings seems to not have a method
> for being queried by it. Of course, simple boxes can be read e.g. using the
> Python methods Band::ReadAsArray/Band::ReadRaster. But this does not work
> once queries get to the boundaries, as all of these methods do not support
> "wrapping around" to the other side of the dataset. As an example:
>
> Dataset: 1000 x 1000 pixels
> Bounding box in pixels, given as two corner points in (x, y) pixels: (950,
> 950) and (50, 50)
> This should query all 50 by 50 pixel corners of the dataset and result in
> a 100 by 100 pixel result
> It arises when a point at the edge of a dataset is queried with a radius
> of 50 pixels
> However, numpy and the above methods do not allow to express such
> wrap-around indexing
>
> Is there some method that I'm overlooking or is this intentionally
> something that users should implement for themselfs? It feels like such
> indexing would be required in a lot of other places too.
>
> I've posted here last week about some suggestions on how to query raster
> bands at non-structured points. Thanks for the useful replies, Christoph &
> Joaquim! However, it seems like we have to code that ourselfs, thus the
> above digging.
>
> Cheers
> Felix D.
> _______
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev