from:"Even Rouault"

Re: [gdal-dev] Reading NetCDF on S3 with /vsis3/

2023-12-03 Thread Even Rouault via gdal-dev


Scott,

Le 03/12/2023 à 03:09, Scott Staniewicz via gdal-dev a écrit :
Are there any known limitations that prevent NetCDF datasets from 
being read using `/vsis3/`?


Yes. This requires Linux + sufficient permissions to run the userfaultfd 
Linux system call, which can be an issue when running in a container. Cf 
https://gdal.org/drivers/raster/netcdf.html#vsi-virtual-file-system-api-support.


The following works for me:

$ docker run --rm -it --security-opt seccomp=unconfined -e 
CPL_AWS_CREDENTIALS_FILE=/path/to/.aws/credentials -v $HOME:$HOME 
ghcr.io/osgeo/gdal:alpine-normal-3.8.1 gdalinfo 
/vsis3/my_bucket/longitude_latitude.nc


Driver: netCDF/Network Common Data Format
Files: /vsis3/my_bucket/longitude_latitude.nc
Size is 4, 3
Origin = (-180.0416657,-79.791679382324219)
Pixel Size = (0.083,-0.083328247070312)
Corner Coordinates:
Upper Left  (-180.0416667, -79.7916794)
Lower Left  (-180.0416667, -80.0416641)
Upper Right (-179.708, -79.7916794)
Lower Right (-179.708, -80.0416641)
Center  (-179.875, -79.9166718)
Band 1 Block=4x1 Type=Float32, ColorInterp=Undefined
  NoData Value=9.96921e+36
  Metadata:
    NETCDF_VARNAME=myvar

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] help please with close/reopen and GDALSubdatasetInfo

2023-12-03 Thread Even Rouault via gdal-dev


Hi Michael,

It was most missing a "break;" statement when the subdataset of interest 
has been found to exit the loop. Without the break, the loop will 
continue to read papszSubdatasets[] which has been invalidated by the 
poSrcDS->ReleaseRef(). Ah, memory unsafe languages :-). You also had a 
few memory leaks.


Fixed version:

#include "gdal.h"
#include "gdal_priv.h"
#include 
int main(int argc, char **argv) {
  GDALAllRegister();

  auto poSrcDS =
    GDALDataset::Open(argv[1], GDAL_OF_RASTER, nullptr, nullptr, nullptr);
  if (poSrcDS == nullptr)
  {
    return 0;
  }
  char **papszSubdatasets = poSrcDS->GetMetadata("SUBDATASETS");
  int nSubdatasets = CSLCount(papszSubdatasets);
  if (nSubdatasets > 0)
  {
    for (int j = 0; j < nSubdatasets; j += 2)
    {
  char* pszSubdatasetSource = CPLStrdup(strstr(papszSubdatasets[j], 
"=") + 1);
  GDALSubdatasetInfoH info = 
GDALGetSubdatasetInfo(pszSubdatasetSource);
  char* component = info ? 
GDALSubdatasetInfoGetSubdatasetComponent(info) : NULL;

  const bool bFound = component && EQUAL(argv[2], component);
  CPLFree(component);
  GDALDestroySubdatasetInfo(info);
  if ( bFound) {
    std::cout << pszSubdatasetSource << "\n";
    poSrcDS->ReleaseRef();
    poSrcDS = GDALDataset::Open(pszSubdatasetSource, 
GDAL_OF_RASTER, nullptr, nullptr, nullptr);

    CPLFree(pszSubdatasetSource);
    break;
  }
  else {
    CPLFree(pszSubdatasetSource);
  }
    }
  }

  poSrcDS->ReleaseRef();
  return 1;
}

Even


Le 03/12/2023 à 05:10, Michael Sumner via gdal-dev a écrit :

#include "gdal.h"
#include "gdal_priv.h"
#include 
int main(int argc, char **argv) {
  GDALAllRegister();

  auto poSrcDS =
    GDALDataset::Open(argv[1], GDAL_OF_RASTER, nullptr, nullptr, nullptr);
  if (poSrcDS == nullptr)
  {
    return 0;
  }
  char **papszSubdatasets = poSrcDS->GetMetadata("SUBDATASETS");
  int nSubdatasets = CSLCount(papszSubdatasets);
  char *pszSubdatasetSource = nullptr;
  if (nSubdatasets > 0)
  {
    for (int j = 0; j < nSubdatasets; j += 2)
    {
      pszSubdatasetSource = CPLStrdup(strstr(papszSubdatasets[j], "=") 
+ 1);
      GDALSubdatasetInfoH info = 
GDALGetSubdatasetInfo(pszSubdatasetSource);
      if ( EQUAL(argv[2], 
GDALSubdatasetInfoGetSubdatasetComponent(info))) {

        std::cout << pszSubdatasetSource << "\n";
        poSrcDS->ReleaseRef();
        poSrcDS = GDALDataset::Open(pszSubdatasetSource, 
GDAL_OF_RASTER, nullptr, nullptr, nullptr);

        CPLFree(pszSubdatasetSource);
        GDALDestroySubdatasetInfo(info);

      }
    }
  }

  poSrcDS->ReleaseRef();
  return 1;
}


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Motion: adopt RFC 98: Build requirements for GDAL 3.9

2023-12-01 Thread Even Rouault via gdal-dev


Hi,

Motion: adopt RFC 98: Build requirements for GDAL 3.9

https://github.com/OSGeo/gdal/pull/8802

Starting with my +1,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [External] : Re: GDAL driver Open() calls

2023-11-30 Thread Even Rouault via gdal-dev

ok, that should be fixed per https://github.com/OSGeo/gdal/pull/8885 
(and factorization of similar code of gdal_translate_lib.cpp and 
gdaldriver.cpp)


Le 30/11/2023 à 20:59, Fengting Chen a écrit :


Hi Even,

Here is what found out from debugging: There are two Open() calls. 
Your fix skipped one. Here is the stack trace of the second one:


Thanks!

*From: *Even Rouault 
*Date: *Thursday, November 30, 2023 at 8:40 AM

Le 30/11/2023 à 04:47, Fengting Chen a écrit :

This doesn’t seem to fix the problem. How do I print out the stack
trace in the driver?

This is not something the driver can do. You have to use the 
interactive debugger of your dev environment, attach it to your 
gdal_translate command, set a breakpoint in the GeoRasterDataset::Open 
method, and use the appropriate function of your debugger to display 
the call stack. I'm sure you'll find plenty of online resources for 
whatever debugger you use.


Even



--
http://www.spatialys.com  
<https://urldefense.com/v3/__http:/www.spatialys.com__;!!ACWV5N9M2RV99hQ!LdunXkjoOPFddx-jqp6_eWkk746TTCmRdAPcBj35cB2LBYcxMogtL6zHZEup8CJ2DN0T62IvlW7uYshKVkU9PsDwZbPE$>
My software is free, but my time generally not.


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] How to determine which creation options are supported by a driver?

2023-11-30 Thread Even Rouault via gdal-dev



Le 30/11/2023 à 21:19, Meyer, Jesse R. (GSFC-618.0)[SCIENCE SYSTEMS AND 
APPLICATIONS INC] via gdal-dev a écrit :


Hi,

I have gdal 3.8.1 installed via Brew which includes liblerc and zstd 
as part of their formulae, so I assumed I could use the creation 
options “COMPRESS=LERC_ZSTD” and “MAX_Z_ERROR=x” as documented on the 
GTIFF page when creating a new gtif image.


But when I do GDAL states:

“Warning 6: 'LERC_ZSTD' is an unexpected value for COMPRESS creation 
option of type string-select.


Warning 6: driver GTiff does not support creation option MAX_Z_ERROR”

"gdalinfo --format GTtiff" will report the options available in your 
GDAL build


So I am not sure if this is an issue with the driver itself or with 
the brew formula.  Please advise,


In the brew formula. In the libtiff one more exactly, as this capability 
comes from libtiff. Just checking 
https://github.com/Homebrew/homebrew-core/blob/b4698a581c0d752241353682fecd627d85ca6dd6/Formula/lib/libtiff.rb, 
it lacks a liblerc dependency.


You could try from Conda, since its libtiff includes lerc support (3.8.1 
hasn't yet landed there, but it should come soon)




Thanks,

Jesse


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] "RFC 98: Build requirements for GDAL 3.9" available for review

2023-11-30 Thread Even Rouault via gdal-dev


No further comments on this ? If not, I'll submit it to vote tomorrow.

Le 24/11/2023 à 12:15, Even Rouault via gdal-dev a écrit :


Hi,

Please find "RFC 98: Build requirements for GDAL 3.9" in 
https://github.com/OSGeo/gdal/pull/8802 for review & comments


Summary:

The document updates RFC68 with the new build requirements for GDAL 3.9:

  * C++ >= 17
  * CMake >= 3.16
  * PROJ >= 6.3.1

The minimum version for the following optional dependencies is also 
updated:


  * Python >= 3.8
  * GEOS >= 3.8
  * Poppler >= 0.86
  * libtiff >= 4.1
  * libcurl >= 7.68
  * libpng >= 1.6.0
  * libsqlite3 >= 3.31
  * libopenjp2 >= 2.3.1
  * libnetcdf >= 4.7 and built with NC4 enabled
  * libhdf5 >= 1.10

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.1 is released

2023-11-30 Thread Even Rouault via gdal-dev


Hi,

On behalf of the GDAL/OGR development team, I am pleased to announce the
release of the GDAL/OGR 3.8.1 bug fix version. All users that had already
updated to 3.8.0 are strongly encouraged to update to 3.8.1, which 
addresses a
few 3.8.0 specific regressions (particularly for ogr2ogr workflows with 
GeoPackage, FlatGeobuf, GeoParquet or TileDB as source datasets).


Consult the release notes for the list of issues addressed:
    https://github.com/OSGeo/gdal/blob/v3.8.1/NEWS.md

The sources are available at:

    https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1.tar.xz
    https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1.tar.gz
    https://download.osgeo.org/gdal/3.8.1/gdal381.zip

Best regards,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.1RC3 is available & motion to approve it

2023-11-30 Thread Even Rouault via gdal-dev

Motion passed with +1 from PSC members JukkaR, HowardB, JavierJS, SeanG 
and me.


Le 28/11/2023 à 16:32, Even Rouault via gdal-dev a écrit :

Hi,

well, another annoying 3.8.0 regression related to ogr2ogr GPKG -> 
Shapefile when field names are longer than 10 characters (ah 
Shapefile!!!) popped up today, so here's a RC3 (crossing fingers it's 
the good one):


  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc3.tar.xz
  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc3.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdal381rc3.zip

A snapshot of the gdalautotest suite is also available:

https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc3.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc3.zip

The changes since RC2 are:

* CMake: make GDAL_USE_LIBKML and GDAL_USE_OPENJPEG honor 
GDAL_USE_EXTERNAL_LIBS

* Detect failure in installation of the Python bindings
* Fix installation issue with Python 3.12 on Debian
* GDALOverviewDataset::IRasterIO(): use parent dataset when possible 
for more

  efficiency
* gdal_footprint: fix wrong taking into account of alpha band (#8834)
* gdal_footprint: fix taking into account of individual bands that 
have nodata
* COG: for JPEG compression, convert single band+alpha as single band 
JPEG + 1-bit mask band
* GTIFF SRS reader: include VertCRS name from EPSG in CompoundCRS name 
if there's no citation geokey
* ogr2ogr: fix GPKG -> Shapefile when field names are truncated 
(#8849, 3.8.0 regression)
* CSV writer: do not quote integer fields by default (only if 
STRING_QUOTING=ALWAYS is specified)

* GPX: make detection of extensions element more robust (#8827)
* Shapefile: recognize '  0' as a null date, fix writing an 
invalid "/00/00" date

* Python bindings: add a combineBands option to gdal.Footprint()


Withdrawing previous motion and:

==> Motion: adopt 3.8.1 RC3 as final 3.8.1 release

Starting with my +1,

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [External] : Re: GDAL driver Open() calls

2023-11-30 Thread Even Rouault via gdal-dev



Le 30/11/2023 à 04:47, Fengting Chen a écrit :


This doesn’t seem to fix the problem. How do I print out the stack 
trace in the driver?


This is not something the driver can do. You have to use the interactive 
debugger of your dev environment, attach it to your gdal_translate 
command, set a breakpoint in the GeoRasterDataset::Open method, and use 
the appropriate function of your debugger to display the call stack. I'm 
sure you'll find plenty of online resources for whatever debugger you use.


Even


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] How do GCP's in a VRT file work?

2023-11-29 Thread Even Rouault via gdal-dev

If you use the -tps flag of gdalwarp, it will strictly honour your GCPs. 
Cf https://gdal.org/programs/gdalwarp.html#cmdoption-gdalwarp-tps and 
https://gdal.org/api/gdal_alg.html#_CPPv424GDALCreateTPSTransformeriPK8GDAL_GCPi


"The thin plate spline transformer produces exact transformation at all 
control points and smoothly varying transformations between control 
points with greatest influence from local control points. It is suitable 
for for many applications not well modeled by polynomial transformations."


Le 29/11/2023 à 16:05, Joe Lovick via gdal-dev a écrit :
Could someone explain how GCP's placed into a VRT file work, i was 
expecting a linear/bi-linear, or polynomial fit between the points for 
the simplest EPSG:4326 projection.


however that is not what i see, adjusting a single point, effects the 
projection of all corner points.


for example,

a vrt file that starts...


    
Y="41.9822570038769" />
X="-93.636924110907" Y="41.9822308438819" />
Y="41.9822348674463" />
Y="41.9822605751492" />


.

if i adjust the 0,0 point to be further east, when i look at the 
resulting remaped image, all corner image points have shifted, rather 
than just the single corner location for 0,0


what am i missing here?

many thanks
Joe


via Newton Mail 




___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [External] : Re: GDAL driver Open() calls

2023-11-29 Thread Even Rouault via gdal-dev

Hi,

The logs didn't directly answer my question as they lacked the chain of
calls, but I believe I figured out the reason. Please try
https://github.com/OSGeo/gdal/pull/8865

Even

Le 29/11/2023 à 02:56, Fengting Chen a écrit :

The attached are two output logs with debug on. You can see that the
output from GDAL 3.7 has two extra queries at the beginning. These
queries only happen when the eAccess is GA_ReadOnly. The logs are
generated with the same command parameters to gdal_translate.

*From: *Even Rouault
*Date: *Tuesday, November 28, 2023 at 6:36 PM
*To: *Fengting Chen ,
gdal-dev@lists.osgeo.org

*Subject: *[External] : Re: [gdal-dev] GDAL driver Open() calls

Hi,

Please share the stack trace (on a debug build) of both instances
where GDALOpen() is called

Even

Le 28/11/2023 à 23:40, Fengting Chen via gdal-dev a écrit :

Hi,

I noticed from the GeoRaster driver that in GDAL 3.6 and 3.7
(maybe 3.8 too), there are two GDALOpen() calls on the driver
with GDALOpenInfo eAccess as GA_ReadOnly, even when the GeoRaster
is the output format in gdal_translate command. There are no such
calls in GDAL 3.4. I am wondering if these invocations of Open()
with eAccess as GA_ReadOnly are necessary when the driver is the
output driver. This has caused performance issue for the GeoRaster
driver when a user is trying to load the data into Oracle database
because it has two unnecessary reads from the database. If these
Open() calls are unavoidable, how to differentiate the calls
whether it is as input or as output in the gdal_translate command?

Thanks.

___

gdal-dev mailing list

gdal-dev@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev
<https://urldefense.com/v3/__https:/lists.osgeo.org/mailman/listinfo/gdal-dev__;!!ACWV5N9M2RV99hQ!IqVgHqcg5DBb3fu5ZpXizGqYnezdKvCzFALf06Gt-Vs7RiPPCXFACcefd_slweWtNCqHy-G-OsmoPt9XZqR4DOWs5UiD$>

--
http://www.spatialys.com
<https://urldefense.com/v3/__http:/www.spatialys.com__;!!ACWV5N9M2RV99hQ!IqVgHqcg5DBb3fu5ZpXizGqYnezdKvCzFALf06Gt-Vs7RiPPCXFACcefd_slweWtNCqHy-G-OsmoPt9XZqR4DFVEpZ99$>
My software is free, but my time generally not.

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL driver Open() calls

2023-11-28 Thread Even Rouault via gdal-dev


Hi,

Please share the stack trace (on a debug build) of both instances where 
GDALOpen() is called


Even

Le 28/11/2023 à 23:40, Fengting Chen via gdal-dev a écrit :


Hi,

I noticed from the GeoRaster driver that in GDAL 3.6 and 3.7 (maybe 
3.8 too),  there are two GDALOpen() calls on the driver with 
GDALOpenInfo eAccess as GA_ReadOnly, even when the GeoRaster is the 
output format in gdal_translate command. There are no such calls in 
GDAL 3.4. I am wondering if these invocations of Open() with eAccess 
as GA_ReadOnly are necessary when the driver is the output driver. 
This has caused performance issue for the GeoRaster driver when a user 
is trying to load the data into Oracle database because it has two 
unnecessary reads from the database.  If these Open() calls are 
unavoidable, how to differentiate the calls whether it is as input or 
as output in the gdal_translate command?


Thanks.


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.1RC3 is available & motion to approve it

2023-11-28 Thread Even Rouault via gdal-dev

FYI, Docker images ghcr.io/osgeo/gdal:alpine-small-3.8.1 , 
ghcr.io/osgeo/gdal:alpine-normal-3.8.1 , 
ghcr.io/osgeo/gdal:ubuntu-small-3.8.1 , 
ghcr.io/osgeo/gdal:ubuntu-full-3.8.1 have now been refreshed against RC3


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.1RC3 is available & motion to approve it

2023-11-28 Thread Even Rouault via gdal-dev


Hi,

well, another annoying 3.8.0 regression related to ogr2ogr GPKG -> 
Shapefile when field names are longer than 10 characters (ah 
Shapefile!!!) popped up today, so here's a RC3 (crossing fingers it's 
the good one):


  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc3.tar.xz
  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc3.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdal381rc3.zip

A snapshot of the gdalautotest suite is also available:

https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc3.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc3.zip

The changes since RC2 are:

* CMake: make GDAL_USE_LIBKML and GDAL_USE_OPENJPEG honor 
GDAL_USE_EXTERNAL_LIBS

* Detect failure in installation of the Python bindings
* Fix installation issue with Python 3.12 on Debian
* GDALOverviewDataset::IRasterIO(): use parent dataset when possible for 
more

  efficiency
* gdal_footprint: fix wrong taking into account of alpha band (#8834)
* gdal_footprint: fix taking into account of individual bands that have 
nodata
* COG: for JPEG compression, convert single band+alpha as single band 
JPEG + 1-bit mask band
* GTIFF SRS reader: include VertCRS name from EPSG in CompoundCRS name 
if there's no citation geokey
* ogr2ogr: fix GPKG -> Shapefile when field names are truncated (#8849, 
3.8.0 regression)
* CSV writer: do not quote integer fields by default (only if 
STRING_QUOTING=ALWAYS is specified)

* GPX: make detection of extensions element more robust (#8827)
* Shapefile: recognize '  0' as a null date, fix writing an invalid 
"/00/00" date

* Python bindings: add a combineBands option to gdal.Footprint()


Withdrawing previous motion and:

==> Motion: adopt 3.8.1 RC3 as final 3.8.1 release

Starting with my +1,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 97: OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn "sealing"

2023-11-27 Thread Even Rouault via gdal-dev


Hi,

I declare this motion passed with +1 from PSC members JukkaR, JavierJS 
and me.


(and friendly remainder that the GDAL 3.8.1RC2 motion is waiting for 
more votes)


Even

Le 22/11/2023 à 13:05, Even Rouault via gdal-dev a écrit :

Hi,

Motion: adopt RFC 97: OGRFeatureDefn, OGRFieldDefn and 
OGRGeomFieldDefn "sealing"


https://github.com/OSGeo/gdal/pull/8734

Pre-rendered view:

https://github.com/rouault/gdal/blob/rfc97_text/doc/source/development/rfc/rfc97_feature_and_fielddefn_sealing.rst 



Starting with my +1,

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] COPY statement failed with PG_USE_COPY=YES

2023-11-27 Thread Even Rouault via gdal-dev


Hi Matteo,

I've pushed a fix in https://github.com/OSGeo/gdal/pull/8840 for that. 
The null dates were formatted in a unusual way. I assume that file was 
produced for another dataset where null dates were set to "/00/00" 
rather than an appropriate null value, which ended in a non-typical null 
date being written.


Even

Le 27/11/2023 à 16:04, Matteo Ghetta a écrit :

Hi Even,

sorry for the delay. In attach you can find a shapefile and here the 
command I'm using:


ogr2ogr -progress --config PG_USE_COPY YES --config SHAPE_ENCODING 
LATIN1 -f PostgreSQL PG:"host=XX port=XX dbname=XX user=XX password=XX 
active_schema=gis" -lco DIM=4 /path/to/FOG_NODI_PT_GIS.shp 
FOG_NODI_PT_GIS -lco LAUNDER=NO -overwrite -nlt POINT -lco 
GEOMETRY_NAME=geom -lco FID=id -nln gis."FOG_NODI_PT_GIS" -s_srs 
EPSG:23032 -t_srs EPSG:4326


Cheers and thanks!

Matteo

On 11/21/23 12:24, Even Rouault wrote:

Matteo,

 From a quick test, I can't reproduce. Please provide a minimum 
shapefile that reproduces the issue


$ cat test.csv
id,dt
1,"/00/00"
$ cat test.csvt
Integer,Date
$ ogr2ogr date_null.shp test.csv
Warning 1: Invalid value type found in record 1 for field dt. This 
warning will no longer be emitted
$ ogr2ogr pg:dbname=autotest date_null.dbf --config PG_USE_COPY YES 
-overwrite

$ ogrinfo pg:dbname=autotest date_null -al
INFO: Open of `pg:dbname=autotest'
   using driver `PostgreSQL' successful.

Layer name: date_null
Geometry: None
Feature Count: 1
Layer SRS WKT:
(unknown)
FID Column = ogc_fid
id: Integer (9.0)
dt: Date
OGRFeature(date_null):1
   id (Integer) = 1
   dt (Date) = (null)

Even

Le 21/11/2023 à 09:24, Matteo Ghetta via gdal-dev a écrit :

Hi all,

I'm trying to import (many) shapefiles into a PG database. I'm 
facing the following error, but only when PG_USE_COPY is set to YES:


ERROR 1: COPY statement failed.
ERROR:  date/time field value out of range: "/00/00"
CONTEXT:  COPY TABLE, line 12682, column DATA_INI: "/00/00"

The value in the source shapefile is NULL for some rows. With 
PG_USE_COPY NO the import works perfectly.


I've found this old ticket, that seems to describe this exact issue, 
and is marked as resolved:


https://trac.osgeo.org/gdal/ticket/4265

Tested with:

* GDAL 3.2.2, released 2021/03/05
* GDAL 3.7.3, released 2023/10/30

Do you have any ideas about what is causing this?

Cheers and thanks

Matteo
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL Java binding problem

2023-11-27 Thread Even Rouault via gdal-dev



Le 27/11/2023 à 10:40, Raivo Rebane via gdal-dev a écrit :


Hi,

I want to use GDAL for GML parsing with Java.

I made an Eclipse project and got dependency - gdal-3.8.0.

My code compiles fine, but in debuging at line - gdal.AllRegister();

I got error :

Native library load failed.

java.lang.UnsatisfiedLinkError: no gdalalljni in java.library.path: 
[/home/Raivo/GML/lib]


Exception in thread "main" java.lang.UnsatisfiedLinkError: 'void 
org.gdal.gdal.gdalJNI.AllRegister()'


at org.gdal.gdal.gdalJNI.AllRegister(Native Method)

at org.gdal.gdal.gdal.AllRegister(gdal.java:712)

at My.Parser.Parser.main(Parser.java:16)

an in realility gdalalljni doesnt exist in my computer.

So I cloned GDAL sources from git, version 3.9.0, but I didn't find 
docs how to buil Java binding jar file


Cf 
https://gdal.org/development/building_from_source.html#java-bindings-options


I've just submitted in https://github.com/OSGeo/gdal/pull/8836 more 
detailed instructions:


"""


How to use the bindings
---

The result of the build of the Java bindings will be both a :file:`gdal.jar`
and a companion :file:`libgdalalljni.so` / :file:`libgdalalljni.dylib` /
:file:`gdalalljni.dll` native library. To limit potential compatibility 
problems,
you should ensure that gdal.jar and gdalalljni come from the same GDAL 
sources.


The native gdalalljni library, as well as the core libgdal library (and its
dependencies) should be accessible through the mechanism of the operating
system to locate shared libraries.
Typically on Linux, this means that the path to those libraries should 
be set
in the ``LD_LIBRARY_PATH`` environment variable (or in 
:file:`/etc/ld.so.conf`).

On MacOSX, it should be in the ``DYLD_LIBRARY_PATH`` environment variable.
And on Windows, in the ``PATH`` environment variable.

For example, to test on Linux that the bindings are working, you can lanch,
from the build directory:

::

    export LD_LIBRARY_PATH=$PWD:$PWD/swig/java:$LD_LIBRARY_PATH
    java -classpath $PWD/swig/java/gdal.jar:$PWD/swig/java/build/apps 
gdalinfo


On Windows:

::

    set PATH=%CD%;%CD%\swig\java;%PATH%
    java -classpath %CD%\swig\java\gdal.jar;%CD%\swig\java\build\apps 
gdalinfo


"""

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL Maintainers Meeting Minutes

2023-11-26 Thread Even Rouault via gdal-dev

Javier,

* use TILED=YES by default in GTiff driver

Was that decided? As some people pointed (including me) a big tiled
geotiff without overviews take ages to load in QGIS.

No, pretty much all items are still just ideas (well, perhaps besides
enabling Python exceptions by default, which I can't really imagine
postponing any further than a 4.0 :-)), some of them not necessarily
desirable or turning out to not be easily implementable. Most of them
will only make sense if we have sufficient community buy-in, as they'll
break things for existing users.

Your concern about big tiled geotiff without overviews is a real one.
One idea I just got to avoid that that just came to mind (possibly not a
good one as I'm detailing it, but anyway here's my brain dump :-)) could
be that on creation of a tiled GeoTIFF larger than let's say 10k x 10k
pixels, GDAL would automatically create a nearest-neighbour overview of
let's say at most 512 pixels in width and height and update it whenever
a tile is updated,. So the overview would likely have to be uncompressed
(not entirely clear if all GeoTIFF capable software around is able to
deal with different compression methods for overviews versus full
resolution image), so its size remains constant when tiles are updated.
And it would have to be discarded when explicit ones are created. And
the logic of selection of overviews would also have to be modified to
select that overview in a more aggressive way than the current logic
does (cf
https://gdal.org/api/gdaldataset_cpp.html#_CPPv4N11GDALDataset8RasterIOE10GDALRWFlagPvii12GDALDataTypeiPi8GSpacing8GSpacing8GSpacingP20GDALRasterIOExtraArg).
Perhaps an open option that a viewer such as QGIS could set? So quite a
bunch of extra complications in the GeoTIFF driver which is already
super complicated.

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL Maintainers Meeting Minutes

2023-11-26 Thread Even Rouault via gdal-dev

* C++17 is currently enabled in master. An RFC will be written in combination with RFC 98 to describe the consequences, requirements, usages, and API changes that may result of this update. Note that RFC 68 in 2017 was where we updated to C++11 in GDAL 2.3.0https://gdal.org/development/rfc/rfc68_cplusplus11.html  


Small clarification: RFC 98 intends to capture all consequences. No 
other RFC related to that for now :-)


Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-25 Thread Even Rouault via gdal-dev


Hi,

I got feedback from a user 
(https://mastodon.social/@le...@fosstodon.org/111470467316641321) that 
the driver is apparently still usable for some datasets that can be 
downloaded from the national map catalog 
(https://kartkatalog.geonorge.no<https://kartkatalog.geonorge.no>). So I 
guess we'll have to keep it alive for now. But the Norwegian community 
would be most welcome to step up to maintain it to address its shortcomings.


Even

Le 23/11/2023 à 18:00, Even Rouault via gdal-dev a écrit :


Hi,



However, I doubt that these organisations use the SOSI driver from 
GDAL/OGR to produce these files.


They can't: the driver is read-only

Discussing that topic with other developers today during GDAL 
maintainer meeting, we could proceed with how we did in the past with 
drivers we suspected to be no longer used but lacked data points. That 
is in 3.9 the driver will be still there, but when it detects a SOSI 
dataset, it will error out with a message indicating to the user that 
they have to define an environment variable to enable it, and invite 
them to file an issue on the bug tracker to make a case for keeping 
the driver, otherwise without feedback, the driver will be fully 
removed in 3.10


Dan also pointed to that QGIS ticket 
(https://github.com/qgis/QGIS/issues/50868) where a user reported a 
crash when trying to use a SOSI file, showing that there are issues in 
the driver and/or libfyba.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Creation of MVT vector tiles with fid

2023-11-25 Thread Even Rouault via gdal-dev


Hi,

add -preserve_fid to your ogr2ogr command line. On reading, you'll see 
it in the mvt_id field (on reading it is not set as a OGR feature id, 
since a given OGR feature can be split among several tiles, leading to 
several features when reading back, hence it is exposed as an attribute 
field to make sure the unicity of the OGR feature id)


Even

Le 25/11/2023 à 14:47, OpenDEM via gdal-dev a écrit :
Can someone please tell me how to prepare the data for the creation of 
MVT vector tiles with a fid?


ogr2ogr -f MVT outputfolder input.geojson (or PostGIS or ...) -dsco 
MINZOOM=0 -dsco MAXZOOM=6 -dsco COMPRESS=NO


Best regards,

Martin

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] slow translate with OVERVIEW_LEVEL=NONE when no overviews exist

2023-11-24 Thread Even Rouault via gdal-dev


Hi Michael,

Le 25/11/2023 à 00:44, Michael Sumner via gdal-dev a écrit :
When I translate this GeoTIFF to 10% original size, it's very very 
slow if OVERVIEW_LEVEL=NONE is set.


Fixed per https://github.com/OSGeo/gdal/pull/8819

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.1 RC1 is available, and motion to approve it

2023-11-24 Thread Even Rouault via gdal-dev

Le 24/11/2023 à 14:34, Sebastiaan Couwenberg via gdal-dev a écrit :

On 11/24/23 13:22, Hernán De Angelis via gdal-dev wrote:
May be this is of help: I have seen the case of the .py utilities not
been compiled and installed a couple of times when I mistakenly had
two Python versions present and acccessible.

There is an issue with setuptools silently failing with python3.12:

https://bugs.debian.org/1055682

This looks like a Debian specific issue. I can't replicate that locally
with a manually built python 3.12.0 (cf
https://gist.github.com/rouault/10096d152198a837ba3200af3cec04a9). It
also works fine on Fedora Rawhide: cf
https://github.com/OSGeo/gdal/actions/runs/6981942751/job/1994023?pr=8812

I've tried in a debian:unstable Docker image, and I indeed reproduce the
issue. I believe I've fixed that in
https://github.com/OSGeo/gdal/pull/8812 (as well as not ignoring the
installation failure). It seemed due to the fact that we previously add
to define SETUPTOOLS_USE_DISTUTILS=stdlib to make things work, and
apparently this is no longer needed. I'll let you cherry-pick those
changes
(https://github.com/OSGeo/gdal/pull/8812/commits/11cd5251e3166648ab51ca499468d0b144fcbfe9
and
https://github.com/OSGeo/gdal/pull/8812/commits/248d7809a46d2fefd7fe320b434182ebe2805ca7),
as I'll likely will not issue a RC3 for that.

Even

http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Problems compiling in Mac

2023-11-24 Thread Even Rouault via gdal-dev


Hi Carsten,

should be fixed per https://github.com/OSGeo/gdal/pull/8810

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.1 RC1 is available, and motion to approve it

2023-11-24 Thread Even Rouault via gdal-dev


Bas,
The Python utilities are no longer installed with the .py extension, 
this broke their use in QGIS when we removed this in the Debian 
package some time ago. Don't know if that's still the case. Either 
way, this doesn't seem like an appropriate change for a .1 patch release.


Are you sure about the .py utilities no longer be installed? Because, I 
see both /usr/bin/gdal2tiles and /usr/bin/gdal2tiles.py installed on 
Ubuntu 22.04 (before I reverted the change to add the launcher scripts)


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] "RFC 98: Build requirements for GDAL 3.9" available for review

2023-11-24 Thread Even Rouault via gdal-dev


Hi,

Please find "RFC 98: Build requirements for GDAL 3.9" in 
https://github.com/OSGeo/gdal/pull/8802 for review & comments


Summary:

The document updates RFC68 with the new build requirements for GDAL 3.9:

 * C++ >= 17
 * CMake >= 3.16
 * PROJ >= 6.3.1

The minimum version for the following optional dependencies is also updated:

 * Python >= 3.8
 * GEOS >= 3.8
 * Poppler >= 0.86
 * libtiff >= 4.1
 * libcurl >= 7.68
 * libpng >= 1.6.0
 * libsqlite3 >= 3.31
 * libopenjp2 >= 2.3.1
 * libnetcdf >= 4.7 and built with NC4 enabled
 * libhdf5 >= 1.10

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.1 RC2 is available (was Re: GDAL 3.8.1 RC1 is available, and motion to approve it)

2023-11-24 Thread Even Rouault via gdal-dev


Hi,

OK, I've reverted that change. I'll guess someone will have to figure 
out how to have both .py scripts and .exe/.sh launchers at the same time...


Pick up an archive among the following ones (by ascending size):

  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc2.tar.xz
  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc2.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdal381rc2.zip

A snapshot of the gdalautotest suite is also available:

https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc2.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc2.zip

Withdrawing previous motion and:

Motion: adopt 3.8.1 RC2 as final 3.8.1 release

Starting with my +1,

Even

Le 24/11/2023 à 06:34, Sebastiaan Couwenberg via gdal-dev a écrit :

On 11/23/23 22:56, Even Rouault via gdal-dev wrote:
 From a packaging point of view, the following change might have 
impacts on

build recipes:


The Python utilities are no longer installed with the .py extension, 
this broke their use in QGIS when we removed this in the Debian 
package some time ago. Don't know if that's still the case. Either 
way, this doesn't seem like an appropriate change for a .1 patch release.


usr/bin/__init__ is now also installed which not correct.

Kind Regards,

Bas


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.1 RC1 is available, and motion to approve it

2023-11-23 Thread Even Rouault via gdal-dev


Hi,

I have prepared a GDAL/OGR 3.8.1 release candidate.

Pick up an archive among the following ones (by ascending size):

  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc1.tar.xz
  https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc1.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdal381rc1.zip

A snapshot of the gdalautotest suite is also available:

https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc1.tar.gz
  https://download.osgeo.org/gdal/3.8.1/gdalautotest-3.8.1rc1.zip

The NEWS file is here:

  https://github.com/OSGeo/gdal/blob/v3.8.1RC1/NEWS.md

From a packaging point of view, the following change might have impacts on
build recipes:
* Python bindings: define entry_points.console_scripts. Affects in 
particular

  Windows where .exe launcher scripts are now installed in the Scripts
  subdirectory.

Motion: adopt 3.8.1 RC1 as final 3.8.1 release

Starting with my +1,

Best regards,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-23 Thread Even Rouault via gdal-dev


Hi,



However, I doubt that these organisations use the SOSI driver from 
GDAL/OGR to produce these files.


They can't: the driver is read-only

Discussing that topic with other developers today during GDAL maintainer 
meeting, we could proceed with how we did in the past with drivers we 
suspected to be no longer used but lacked data points. That is in 3.9 
the driver will be still there, but when it detects a SOSI dataset, it 
will error out with a message indicating to the user that they have to 
define an environment variable to enable it, and invite them to file an 
issue on the bug tracker to make a case for keeping the driver, 
otherwise without feedback, the driver will be fully removed in 3.10


Dan also pointed to that QGIS ticket 
(https://github.com/qgis/QGIS/issues/50868) where a user reported a 
crash when trying to use a SOSI file, showing that there are issues in 
the driver and/or libfyba.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Problems compiling in Mac

2023-11-23 Thread Even Rouault via gdal-dev





I thought that it was enough disabling GDAL_USE_EXTERNAL_LIBS (we
are not enabling explicitly arrow or kml). Am I wrong?


He might have to remove CMakeCache.txt because it could contain
the GDAL_USE_LIBKML=ON declaration that was set before turning off
GDAL_USE_EXTERNAL_LIBS

He started from an empty directory.

We are using conan, that is python, and setting 
GDAL_USE_EXTERNAL_LIBS=False . This is the result after searching for 
that string:


and what is the value of GDAL_USE_LIBKML and GDAL_USE_ARROW ? and what 
is the exact error message?



--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Problems compiling in Mac

2023-11-23 Thread Even Rouault via gdal-dev



Le 23/11/2023 à 14:53, Javier Jimenez Shaw via gdal-dev a écrit :

Hi

A colleague of mine is compiling in Mac. I am aware of the problems 
with arrow and libkml:

https://gdal.org/development/building_from_source.html#building-on-macos

But we are compiling with GDAL_USE_EXTERNAL_LIBS=OFF

However he has those problems with arrow and libkml, and had to 
unistall them from his computer.


I thought that it was enough disabling GDAL_USE_EXTERNAL_LIBS (we are 
not enabling explicitly arrow or kml). Am I wrong?


He might have to remove CMakeCache.txt because it could contain the 
GDAL_USE_LIBKML=ON declaration that was set before turning off 
GDAL_USE_EXTERNAL_LIBS





Thanks.
.___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ...  ._ .__

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Motion: adopt RFC 97: OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn "sealing"

2023-11-22 Thread Even Rouault via gdal-dev


Hi,

Motion: adopt RFC 97: OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn 
"sealing"


https://github.com/OSGeo/gdal/pull/8734

Pre-rendered view:

https://github.com/rouault/gdal/blob/rfc97_text/doc/source/development/rfc/rfc97_feature_and_fielddefn_sealing.rst 



Starting with my +1,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Advanced 3.8.1 release

2023-11-21 Thread Even Rouault via gdal-dev


Hi,

yesterday was a very prolific day where several regressions of 3.8.0 
were reported:


- https://github.com/OSGeo/gdal/issues/8755: Python: Filtered 
parquet/arrow layers sometimes produce invalid Map arrays


- https://github.com/OSGeo/gdal/issues/8757: Conversion from GPKG using 
ogr2ogr fails in 3.8.0 when there are more than 125 columns


- https://github.com/OSGeo/gdal/issues/8759: Possible regression in date 
parsing in GDAL 3.8.0


- https://github.com/OSGeo/gdal/issues/8761: Conversion from GPKG with 
the -preserve_fid flag fails when using ogr2ogr in 3.8.0


- https://github.com/OSGeo/gdal/issues/8767:  Spurious warnings when 
reading shapefiles (actually a 3.7.3 regression)


They have now been fixed. I'm considering issuing a 3.8.1 release 
candidate end of this week.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] COPY statement failed with PG_USE_COPY=YES

2023-11-21 Thread Even Rouault via gdal-dev


Matteo,

From a quick test, I can't reproduce. Please provide a minimum 
shapefile that reproduces the issue


$ cat test.csv
id,dt
1,"/00/00"
$ cat test.csvt
Integer,Date
$ ogr2ogr date_null.shp test.csv
Warning 1: Invalid value type found in record 1 for field dt. This 
warning will no longer be emitted
$ ogr2ogr pg:dbname=autotest date_null.dbf --config PG_USE_COPY YES 
-overwrite

$ ogrinfo pg:dbname=autotest date_null -al
INFO: Open of `pg:dbname=autotest'
  using driver `PostgreSQL' successful.

Layer name: date_null
Geometry: None
Feature Count: 1
Layer SRS WKT:
(unknown)
FID Column = ogc_fid
id: Integer (9.0)
dt: Date
OGRFeature(date_null):1
  id (Integer) = 1
  dt (Date) = (null)

Even

Le 21/11/2023 à 09:24, Matteo Ghetta via gdal-dev a écrit :

Hi all,

I'm trying to import (many) shapefiles into a PG database. I'm facing 
the following error, but only when PG_USE_COPY is set to YES:


ERROR 1: COPY statement failed.
ERROR:  date/time field value out of range: "/00/00"
CONTEXT:  COPY TABLE, line 12682, column DATA_INI: "/00/00"

The value in the source shapefile is NULL for some rows. With 
PG_USE_COPY NO the import works perfectly.


I've found this old ticket, that seems to describe this exact issue, 
and is marked as resolved:


https://trac.osgeo.org/gdal/ticket/4265

Tested with:

* GDAL 3.2.2, released 2021/03/05
* GDAL 3.7.3, released 2023/10/30

Do you have any ideas about what is causing this?

Cheers and thanks

Matteo
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Resampling a raster with gdalwarp

2023-11-19 Thread Even Rouault via gdal-dev



Le 19/11/2023 à 03:08, Giovanni Anconitano a écrit :

Thank you very much.

Just to be sure, gdalwarp takes the absolute value of the second tr 
value independently from the units of the pixel size.
It can be either meters or degrees depending on the coordinate 
reference system. Is that correct?


yes: 
https://github.com/OSGeo/gdal/blob/208d0cdb35cc4f790f516eb7a3a88a1d9149a2ed/apps/gdalwarp_lib.cpp#L5411



--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Resampling a raster with gdalwarp

2023-11-18 Thread Even Rouault via gdal-dev


Giovanni,

gdalwarp takes the absolute value of the second tr value, so -tr 10 -10 
is understood as -tr 10 10


It is expected that gdalinfo reports a negative value for the pixel size 
factor for the vertical axis, as in GeoTIFF when the image is north-up, 
the first line of the image is intended to be the one displayed at the 
top of your screen, and as the line number of your screen increase from 
the top of your screen to the bottom, northings or latitudes increase 
from the bottom to the top, hence the negative sign...


More detailed equivalent explanation at 
https://gis.stackexchange.com/a/229807/218992 (you can ignore the 
paragraph "As to why the image flipped..." which is specific to the 
question in that stackexchange thread)


Even

Le 19/11/2023 à 02:15, Giovanni Anconitano via gdal-dev a écrit :

Hello,

I have a question about how to properly set the -tr flag when using 
gdalwarp.
In particular, I'm using gdalwarp for resampling a GeoTiff from its 
original resolution to a target resolution.
Let's consider, for simplicity, the original pixel size equal to (5m, 
5m) and the target resolution equal to (10m, 10m).


This is the command I use: gdalwarp -tr 10 -10 input.tif output.tif

I'm using a negative  because when I open the GeoTiff file in 
QGIS, or if I use gdalinfo, I get a pixel size equal to (5, -5) for 
the source raster.


Is it correct to use a negative  with gdalwarp? I also tried -tr 
10 10, but the result seems to be the same as the case in which I use 
-tr 10 -10.

In both cases gdalinfo and QGIS display (10,-10) as the pixel size.

How does gdalwarp treat a negative yres value?

Thank you

Giovanni

*
*Fai crescere le giovani ricercatrici e i giovani ricercatori*
*
*con il 5 per mille alla Sapienza
*
Scrivi il codice fiscale dell'Università *80209930587
Cinque per mille *





___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Patches to fix build errors against libxml2 2.12

2023-11-18 Thread Even Rouault via gdal-dev


Hi,

Just to notify (mostly packagers) that libxml2 2.12 has been released 2 
days ago, and GDAL needs the following 2 patches to build against it:


- 
https://github.com/OSGeo/gdal/commit/cbed9fc91dffba30d0f9a6a06a412a04d9cd36fa 



- 
https://github.com/OSGeo/gdal/commit/ec33f6d6dfe944f59dc5454d01b4d000d9479c02


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GeoPDF and neatline

2023-11-17 Thread Even Rouault via gdal-dev


Hi,


_My question :_
Is there a way to gdal_translate / gdalwarp the PDF file without 
EXPLICITLY extracting the neatline ?

No. Your above method is the nominal one.


I tried to use the "*-oo NEATLINE=something*" option :
gdal_translate -of GTiff -oo NEATLINE="POLYGON ((467455.095191925 
4427281.84826825,467455.105581267 4442204.97588751,479233.89727774 
4442204.58886474,479234.013428373 4427281.65662041,467455.095191925 
4427281.84826825)) Boulder_400010515_FSTopo.pdf 
Boulder_400010515_FSTopo.pdf.tif


This won't work. As documented, this must be a name of a "Measure" PDF 
object. This is in the case of a PDF with multiple georeferenced areas 
(like US with an Alaska or Hawaai inset map). And by itself, this will 
not crop the raster, but just expose the pointed Measure as the NEATLINE 
metadata.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

2023-11-17 Thread Even Rouault via gdal-dev


Hi,

Motion passed with +1 from PSC members KurtS, HowardB, JukkaR, JavierJS 
and me


Even

Le 15/11/2023 à 10:51, Even Rouault via gdal-dev a écrit :

Hi,

Motion: adopt RFC 96: Deferred C++ plugin loading

https://github.com/OSGeo/gdal/pull/8648

Pre-rendered view:

https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst 



Starting with my +1,

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] RFC97 OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn "sealing" available for review

2023-11-16 Thread Even Rouault via gdal-dev


Hi,

Please review RFC 97: OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn 
"sealing"


at https://github.com/OSGeo/gdal/pull/8734

Summary:

This RFC aims at avoiding common misuse of the setter methods of the 
OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn classes. Indeed, the 
setter methods of those classes should not be used directly by user code 
(that is non driver implementations), on instances that are owned by a 
OGRLayer. It is quite frequent for users (even seasoned ones) to neglect 
that constraint. Hence this RFC introduces an optional "sealing" 
capability that drivers can enable to avoid users modifying instances 
that they should not.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Converting parquet to geopackage

2023-11-16 Thread Even Rouault via gdal-dev


Michael,

this error comes from libarrow-cpp.

Several potential causes:

- one of the parquet file is corrupted

- one of the parquet files is valid but uses "something" that 
libarrow-cpp can't understand. I mention this because we have seen an 
interoperability issues between files generated by the go implementation 
of parquet and libarrow-cpp. Not clear which end is the culprit. But the 
error message was different than yours


- there's an I/O error when getting data from S3

- some other bug in libarrow-cpp...

Perhaps run in --debug on mode to see in the traces which parquet file 
cause the issue ? (assuming that the error can also be reproduced when 
reading one of the component parquet files, and not just when reading 
the whole dataset...)


https://github.com/search?q=repo%3Aapache%2Farrow+TProtocolException%3A+Exceeded+size+limit=issues 
shows a number of issues where this error message pops up


Even

Le 16/11/2023 à 14:37, Smith, Michael ERDC-RDE-CRREL-NH CIV via gdal-dev 
a écrit :


Using gdal3.8 (ghcr.io/osgeo/gdal:ubuntu-full-3.8.0) , got an error I 
haven’t seen before:


ReadNext() failed: Couldn't deserialize thrift: TProtocolException: 
Exceeded size limit


Deserializing page header failed.

This happened at 92%

Command: ogr2ogr -f gpkg /data/overturemaps_2023_11_14.gpkg 
/vsis3/overturemaps-us-west-2/release/2023-11-14-alpha.0/theme=buildings/ 
theme=buildings -progress -NLT MULTIPOLYGON


So anyone know what this means and what caused it and any workarounds?

Mike

--

Michael Smith

US Army Corps of Engineers

Remote Sensing/GIS Center


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

2023-11-16 Thread Even Rouault via gdal-dev


Hi Sean,


I think this makes great sense for the project. I don't yet understand 
what it means for an enterprise like Rasterio's PyPI wheels.


I'd say it probably changes nothing. The RFC just postpones the time 
where the plugins are loaded, but the fact that they are dlopen()'ed 
(early or late) probably makes them non discoverable by delocate, since 
libgdal doesn't link to them in a way that is advertised in its shared 
library metadata. If your plan is to still have a rasterio wheel with a 
monolithic GDAL, then you don't need to build GDAL drivers as plugins 
and this RFC doesn't change anything to the status quo.


I'm not familiar at all with the wheel Python packaging tools, but if 
you'd want to have GDAL plugins in separate package(s) then you'd need 
to have a way of having the gdal_XXX.so / ogr_XXX.so be put in some 
known location that can be advertized to libgdal core with GDAL_DRIVER_PATH.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Export image binary data from GDB dataset

2023-11-15 Thread Even Rouault via gdal-dev

Andrea,

looking with an hexadecimal editor, one sees the strings "Fotografia di
Photo Editor", "MSPhotoEd.3", so it seems to be Microsoft Photo Editor
format. According to
https://stackoverflow.com/questions/43902205/microsoft-photo-editor-conversion-file-signatures
, it might be a OLE based format containing JPEG files. So you might try
tools that extract the content from OLE, and/or look for content between
the FF D8 marker for the start of JPEG content and FF D9 for end of JPEG
file

Even

Le 15/11/2023 à 12:49, andy via gdal-dev a écrit :

Hi,
I have a Binary field in a GDB data source.

I have no other metadata about it. I know only that the field name is
"image", then it should contain images.

But I do not know the original file format.

I have extract one row (and then one field) as string it, using

ogr2ogr -f CSV /vsistdout/ -fieldTypeToString Binary data.gdb
SITI_ARCHEO -where "OBJECTID=277" >out.csv

And I have a long string as the one below.
I have tried to use "xxd -r -p" to convert to binary and then use
"file" to interpret its file format, but I don't get anything useful.

How could I do some reverse extraction of these images in their
original format?

Thank you

151C3B0002001B000C0014002F00466F746F6772616669612064692050686F746F20456469746F72004D5350686F746F45642E3300010502000C004D5350686F746F45642E330E0900D0CF11E0A1B11AE13E000300FEFF090006000A00011002000100FEFF040005000600070045024602470248024902FFF
...

--
___

Andrea Borruso
website: https://medium.com/tantotanto
38° 7' 48" N, 13° 21' 9" E, EPSG:4326
___

"cercare e saper riconoscere chi e cosa,
in mezzo all’inferno, non è inferno,
e farlo durare, e dargli spazio"

Italo Calvino

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

2023-11-15 Thread Even Rouault via gdal-dev


Hi,

Motion: adopt RFC 96: Deferred C++ plugin loading

https://github.com/OSGeo/gdal/pull/8648

Pre-rendered view:

https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst 



Starting with my +1,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-14 Thread Even Rouault via gdal-dev

Robet,

Can you confirm the information I got that the driver can only be used 
to convert legacy datasets (can they still be downloaded from web 
sites?) and not recent ones ? How much time still do you plan to use it ?

Le 14/11/2023 à 20:53, nagyrobi_r--- via gdal-dev a écrit :

Hi all!

I use SOSI gdal conversion. So in case you are wondering, there is at 
least one user.

Kind regards,
Robet Nagy

Sent from Yahoo Mail on Android 
<https://mail.onelink.me/107872968?pid=nativeplacement=Global_Acquisition_YMktg_315_Internal_EmailSignature_sub1=Acquisition_sub2=Global_YMktg_sub3=_sub4=10604_sub5=EmailSignature__Static_>

On Tue, Nov 14, 2023 at 8:41 PM, Sebastiaan Couwenberg via gdal-dev
 wrote:
On 11/14/23 20:05, Even Rouault via gdal-dev wrote:
> Le 14/11/2023 à 19:48, Sebastiaan Couwenberg via gdal-dev a écrit :
>> On 11/14/23 19:16, Even Rouault via gdal-dev wrote:
>>> ==> Are there users still using the existing OGR SOSI driver ?
>>
>> We don't have metrics for the OGR driver, but libfyba0 still
has 513
>> users in popcon.
>>
>> https://qa.debian.org/popcon.php?package=fyba
>
> Looking in the same column in
> https://qa.debian.org/popcon.php?package=gdal shows that
gdal-bin would
> have only 520 users...  which seems highly suspiciously low
compared to
> the super specialized and low level libfyba.  I would bet that >
99% of
> installations of libfyba0 are because libgdal links to it, which
doesn't
> mean that it is used for useful stuff (the other reverse
dependency of
> libfyba in Debian is sosi2osm, which according to
> https://qa.debian.org/popcon.php?package=sosi2osm, has only 3 users)

sosi2osm was removed from Debian during the bookworm development
cycle.

The only rdep of fyba in Debian is gdal now.

Almost all installations of libfyba are due to it being a
dependency of
gdal, that's why it has 23911 installations but only 513 votes
(regular
users).

> What if people build a Docker image of GDAL from source and
install its
> dependencies with apt install libfyba-dev: will that trigger a
hit in
> popcorn?

popcon is opt-in, there are only ~23 systems running popcon
out of
millions of Debian installations. Hence it's mostly usable to provide
insight into relative popularity compared to other packages in
Debian.
Container images are unlikely to enable popcon, the Docker images
don't
even have it installed.

> I'm more interested in Norwegian flesh and bones users
testifying about
> their use or non-use of SOSI.

Me too, if there are none left we can also remove the fyba package
from
Debian.

Kind Regards,

Bas

-- 
  GPG Key ID: 4096R/6750F10AE88D4AF1

Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-14 Thread Even Rouault via gdal-dev



Le 14/11/2023 à 19:48, Sebastiaan Couwenberg via gdal-dev a écrit :

On 11/14/23 19:16, Even Rouault via gdal-dev wrote:

==> Are there users still using the existing OGR SOSI driver ?


We don't have metrics for the OGR driver, but libfyba0 still has 513 
users in popcon.


 https://qa.debian.org/popcon.php?package=fyba


Bas,

Looking in the same column in 
https://qa.debian.org/popcon.php?package=gdal shows that gdal-bin would 
have only 520 users...  which seems highly suspiciously low compared to 
the super specialized and low level libfyba.  I would bet that > 99% of 
installations of libfyba0 are because libgdal links to it, which doesn't 
mean that it is used for useful stuff (the other reverse dependency of 
libfyba in Debian is sosi2osm, which according to 
https://qa.debian.org/popcon.php?package=sosi2osm, has only 3 users)


What if people build a Docker image of GDAL from source and install its 
dependencies with apt install libfyba-dev: will that trigger a hit in 
popcorn?


I'm more interested in Norwegian flesh and bones users testifying about 
their use or non-use of SOSI.


Even

--

http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-14 Thread Even Rouault via gdal-dev


Hi,

During the OSGeo code sprint, it was mentioned to me that the OGR SOSI 
driver (https://gdal.org/drivers/vector/sosi.html) is probably no longer 
relevant. Kartverket, the Norwegian mapping agency, which is the entity 
delivering datasets in the SOSI standard and which contributed the 
driver in 2010 has changed the format a few years ago and libfyba + the 
OGR SOSI driver haven't been updated to deal with it. It seems unlikely 
that Kartverket will provide updates to libfyba + OGR SOSI driver, given 
that the updated SOSI format is going to be totally retired in a near 
future. Apparently the future/new standard will be GML based, indeed 
totally different of the old custom SOSI format.


It thus seems its only use now would be limited to SOSI datasets 
produced up to a few years ago, which is a pretty limited use case. If 
there's no longer any use of it, it could be a good candidate for being 
removed.


==> Are there users still using the existing OGR SOSI driver ?

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-13 Thread Even Rouault via gdal-dev


Hi,

The OSGeo code sprint last week has been the opportunity to advance the 
candidate implementation of the RFC. It is now available at 
https://github.com/OSGeo/gdal/pull/8695 :


 * All drivers that can be built as plugins and depend on external
   libraries have been converted to use the deferred plugin loading
   capability. Plus a few other drivers that only depend on core
 * The PNG, JPEG, GIF, MRF, NITF and planetary (PDS, PDS4, ISIS2,
   ISIS3, VICAR) drivers can also now be built as plugins (when not
   using the in-tree vendored libraries such as libpng, libjpeg, etc.),
   and use deferred plugin loading

On my full configuration with all drivers that can be built as plugins 
effectively built as plugins - that is those that only depend on 
libgdal, with -DGDAL_ENABLE_PLUGINS_NO_DEPS=ON and those that depend on 
third-party libraries, with -DGDAL_ENABLE_PLUGINS=ON -, the 
initialization time has dropped from ~ 300 ms to 70 ms. If building only 
as plugins drivers that depend on third-party libraries with 
-DGDAL_ENABLE_PLUGINS=ON (which is the config that makes most sense), it 
drops to 26 ms, with all 55 plugins in deferred loading mode. One of the 
biggest culprit identified during the conversion process was the DGNv8 / 
DWG driver wich links to gazillion shared libraries of the Teigha SDK.


I've also added a bit of extra user friendliness:

   When a plugin driver is known of core libgdal, but not available as 
a plugin at
    runtime, GDAL will inform the user that the plugin is not 
available, but could
    be installed. It is possible to give more hints on how to install a 
plugin

    by setting the following option:

    .. option:: 
GDAL_DRIVER__PLUGIN_INSTALLATION_MESSAGE:STRING


    .. option:: OGR_DRIVER__PLUGIN_INSTALLATION_MESSAGE:STRING

    Custom message to give a hint to the user how to install a 
missing plugin



    For example, if doing a build with::

    cmake .. -DOGR_DRIVER_PARQUET_PLUGIN_INSTALLATION_MESSAGE="You 
may install it with with 'conda install -c conda-forge 
libgdal-arrow-parquet'"


    and opening a Parquet file while the plugin is not installed will 
display the

    following error::

    $ ogrinfo poly.parquet
    ERROR 4: `poly.parquet' not recognized as a supported file 
format. It could have been recognized by driver Parquet, but plugin 
ogr_Parquet.so is not available in your installation. You may install it 
with with 'conda install -c conda-forge libgdal-arrow-parquet'


I'll submit soon the RFC to vote if there are no further remarks.

Even

Le 06/11/2023 à 14:50, Even Rouault via gdal-dev a écrit :

Hi,

I've revised a bit the RFC 
(https://github.com/OSGeo/gdal/pull/8648/commits/b7053db2f433275edf16286596e71f3ceb9738a5) 
to unify the way the plugin driver proxy and the actual driver declare 
their metadata. This avoids the new GDALPluginDriverFeatures class, 
and will limit the risk of omissions or inconsistencies between the 
proxy and actual drivers.


Even

Le 02/11/2023 à 12:59, Even Rouault via gdal-dev a écrit :

Hi,

I'm seeking for feedback and review on a new RFC (RFC 96: Deferred 
in-tree C++ plugin loading),

detailed in https://github.com/OSGeo/gdal/pull/8648, whose summary is:

This RFC adds a mechanism to defer the loading of in-tree C++ plugin 
drivers to
the point where their executable code is actually needed, and 
converts a number
of relevant drivers to use that mechanism. The aim is to allow for 
more modular

GDAL builds, while improving the performance of plugin loading.

(This is material only for GDAL 3.9 of course)

Even


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.0 is released

2023-11-13 Thread Even Rouault via gdal-dev


Hi,

On behalf of the GDAL/OGR development team and community, I am pleased to
announce the release of GDAL/OGR 3.8.0. GDAL/OGR is a C++ geospatial data
access library for raster and vector file formats, databases and web 
services.

It includes bindings for several languages, and a variety of command line
tools.

http://gdal.org/

The 3.8.0 release is a new feature release with the following highlights:

* Add [JSONFG](https://gdal.org/drivers/vector/jsonfg.html) read/write 
vector driver for OGC Features and Geometries JSON.
* Add [PMTiles](https://gdal.org/drivers/vector/pmtiles.html) read/write 
vector driver for PMTiles v3 datasets containing MVT PBF tiles
* Add [S102](https://gdal.org/drivers/raster/s102.html) raster read-only 
driver for S-102 bathymetric products (depends on libhdf5)
* Add [gdal_footprint](https://gdal.org/programs/gdal_footprint.html) 
utility: compute the footprint of a raster file, taking into account 
nodata/mask band, and generating polygons/multipolygons corresponding to 
areas where pixels are valid (#6264)
* Python bindings: various enhancements to reduce the number of 
"gotchas" related to inter-object ownership relationships, and a few 
syntaxic sugar enhancements
* Arrow interface: improve spatial and attribute filtering on read side; 
add write side with OGRLayer::WriteArrowBatch()

* GeoPackage: much faster spatial index creation (~ 3-4 times faster)
* ARG driver deprecated: will be removed in 3.9.0
* Bump of shared lib major version

More complete information on the new features and fixes in the 3.8.0 
release can be found at:


https://github.com/OSGeo/gdal/blob/v3.8.0/NEWS.md

The release can be downloaded from:
  * https://download.osgeo.org/gdal/3.8.0/gdal380.zip - source as a zip
  * https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0.tar.gz - source as 
.tar.gz
  * https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0.tar.xz - source as 
.tar.xz
  * https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0.tar.gz - 
test suite
  * https://download.osgeo.org/gdal/3.8.0/gdal380doc.zip - 
documentation / website


Best regards,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: adopt GDAL 3.8.0RC2 as 3.8.0 release

2023-11-13 Thread Even Rouault via gdal-dev

I declare this motion passed with +1 from PSC members JukkaR, JavierJS, 
HowardB and me


Even

Le 09/11/2023 à 11:12, Even Rouault via gdal-dev a écrit :

Retracting the motion to adopt 3.8.0RC1 and updating it to:

Motion:

Adopt GDAL 3.8.0RC2 as 3.8.0 release

Starting with my +1

Even

Le 08/11/2023 à 09:39, Even Rouault via gdal-dev a écrit :

Hi,

Motion:

Adopt GDAL 3.8.0RC1 as 3.8.0 release

Starting with my +1

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Motion: adopt GDAL 3.8.0RC2 as 3.8.0 release

2023-11-09 Thread Even Rouault via gdal-dev


Retracting the motion to adopt 3.8.0RC1 and updating it to:

Motion:

Adopt GDAL 3.8.0RC2 as 3.8.0 release

Starting with my +1

Even

Le 08/11/2023 à 09:39, Even Rouault via gdal-dev a écrit :

Hi,

Motion:

Adopt GDAL 3.8.0RC1 as 3.8.0 release

Starting with my +1

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.0 rc2 is available

2023-11-09 Thread Even Rouault via gdal-dev


Hi,

I have prepared a 3.8.0 RC2. The changes since RC1 are:
- Python utilities: make gdal_edit, gdal_pansharp, gdal_retile and 
ogr_layer_algebra return 0 error code when invoked with --version switch
- Include  for uint64_t in cpl_json.h (which broke compilation 
of GDAL reverse dependencies with gcc 13)


Pick up an archive among the following ones (by ascending size):

  https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0rc2.tar.xz
  https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0rc2.tar.gz
  https://download.osgeo.org/gdal/3.8.0/gdal380rc2.zip

Best regards,

Even

Le 06/11/2023 à 11:49, Even Rouault via gdal-dev a écrit :

Hi,

I have prepared a GDAL/OGR 3.8.0 release candidate.

NEWS at:

  https://github.com/OSGeo/gdal/blob/v3.8.0RC1/NEWS.md

(small change in it since the initial version I mentioned last week:
there is a new optional dependency: libaec, needed for some GRIB files)

Pick up an archive among the following ones (by ascending size):

  https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0rc1.tar.xz
  https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0rc1.tar.gz
  https://download.osgeo.org/gdal/3.8.0/gdal380rc1.zip

A snapshot of the gdalautotest suite is also available :

https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0rc1.tar.gz
  https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0rc1.zip

A snapshot of the documentation is at:

  https://download.osgeo.org/gdal/3.8.0/gdal380doc.zip

I'll call for a vote promoting it later this week if no serious 
problems are reported before.


The "release/3.8" branch has been created for all bug fixes related to 
3.8.x.

master is now targeting 3.9.0dev.

Best regards,

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Motion: Adopt GDAL 3.8.0RC1 as 3.8.0 release

2023-11-08 Thread Even Rouault via gdal-dev


Hi,

Motion:

Adopt GDAL 3.8.0RC1 as 3.8.0 release

Starting with my +1

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] tps - gdalwarp vs ogr2ogr

2023-11-07 Thread Even Rouault via gdal-dev


Hi,

I've ticketed your issue as https://github.com/OSGeo/gdal/issues/8672 . 
What you could try to reduce the execution time is to enable 
multithreading by adding -wo NUM_THREADS=ALL_CPUS to your gdalwarp 
command line


And yes the previous implementation of the raster TPS transformation was 
"wrong", at least not consistent with the forward direction of the TPS 
transformer as used by ogr2ogr. Both matched on the exact location of 
ground control points, but the result could be substantially different 
in-between. The new implementation solves that, but it implies 
substantial more computation cost as you found out...


Even

Le 07/11/2023 à 08:29, Stijn Tallir via gdal-dev a écrit :

Hi Jukka,

The transformation lasted 3h20min vs 9min (before the fix)!

The results for raster and vector tps transformation are very 
similar now and nothing changed for the vector result. Do I have to 
conclude the previous rasters tps transformation method was faulty then?


Stijn

Op ma 6 nov 2023 om 16:32 schreef Stijn Tallir 
mailto:stijn%2bgdal-...@strict.be>>:


Hi,

Does that mean the old raster tps transformation was "wrong" or
the old vector transformation?

I'm doing a test and trying to transform my raster image with the
latest dev version in osgeo4W but it takes forever to process now.
Don't know in how many days I will see the result :(

The vector transformation with the latest dev version was the same
(time and result).

Stijn

Op ma 6 nov 2023 om 14:48 schreef Rahkonen Jukka
:

Hi,

See the issue https://github.com/OSGeo/gdal/issues/8572
. Maybe your
problem is also resolved by
https://github.com/OSGeo/gdal/pull/8573. The fix is included
in the GDAL 3.8 RC1 version that was released 3 hour ago. Do
you have an option to make a test?

-Jukka Rahkonen-

*Lähettäjä:* Stijn Tallir 
*Lähetetty:* maanantai 6. marraskuuta 2023 14.47
*Vastaanottaja:* Rahkonen Jukka

*Kopio:* gdal-dev@lists.osgeo.org
*Aihe:* Re: [gdal-dev] tps - gdalwarp vs ogr2ogr

Hi Jukka,

I finally found the time to produce a test set.

You can download it here:

https://drive.google.com/file/d/1Y08Q-tIm5dxyKFKNdVqilvAO3H7FFFbx/view?usp=sharing

I started from an unreferenced raster file (raster2tps.tif)
with gcp's (gcp4tps.gcp) and transformed it with tps
(tpsraster.tif).

Then polygonized the unreferenced raster file (vector2tps.shp)
and transformed the result with  the same gcp's (gcp4tps.gcp)
and with tps (tpsvector.shp).

The vector2tps.shp polygons are "flipped" because of the
different Y-origin for rasters and vectors but this way both
datasets can use the exact same gcp's.

When you lay the tpsvector-result on top of the
tpsraster-result (in QGis for instance) you'll see the
differences in how both are transformed.

Kind regards,

Stijn

Op wo 16 aug 2023 om 13:16 schreef Stijn Tallir
mailto:stijn%2bgdal-...@strict.be>>:

Yes, I checked them visually for both raster and vector.

I compared the results also visually. The rasters are
transformed in a way that the end ponts of the gcp's align
exactly with the result so that is why I referred to it as
"right". The vector data result is in the neighbourhood of
the end points (sometimes a rather significant distance).

The result is different from order 1-3 transformations so
I presume the tps option isn't ignored.

Stijn

Op wo 16 aug 2023 om 11:52 schreef Rahkonen Jukka
:

Hi,

Did you check the ground control points? What is your
reference when you say that one result is right, and
another wrong? Have you used some other software for
comparison? Or do you only know that the results are
different?

-Jukka-

*Lähettäjä:* Stijn Tallir 
*Lähetetty:* keskiviikko 16. elokuuta 2023 12.27
*Vastaanottaja:* Rahkonen Jukka

*Kopio:* gdal-dev@lists.osgeo.org
*Aihe:* Re: [gdal-dev] tps - gdalwarp vs ogr2ogr

Hi Jukka,

I thought of the density as an option for the "error"
as you suggested and I made a point-file with a point
for every pixel in my original image and used this as
a source for the ogr2ogr transformation. So you could
say the desnity for both sources raster and vector)
are then alike.

The results were still the same (and wrong) ...

Stijn

Op

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-06 Thread Even Rouault via gdal-dev


Hi,

I've revised a bit the RFC 
(https://github.com/OSGeo/gdal/pull/8648/commits/b7053db2f433275edf16286596e71f3ceb9738a5) 
to unify the way the plugin driver proxy and the actual driver declare 
their metadata. This avoids the new GDALPluginDriverFeatures class, and 
will limit the risk of omissions or inconsistencies between the proxy 
and actual drivers.


Even

Le 02/11/2023 à 12:59, Even Rouault via gdal-dev a écrit :

Hi,

I'm seeking for feedback and review on a new RFC (RFC 96: Deferred 
in-tree C++ plugin loading),

detailed in https://github.com/OSGeo/gdal/pull/8648, whose summary is:

This RFC adds a mechanism to defer the loading of in-tree C++ plugin 
drivers to
the point where their executable code is actually needed, and converts 
a number
of relevant drivers to use that mechanism. The aim is to allow for 
more modular

GDAL builds, while improving the performance of plugin loading.

(This is material only for GDAL 3.9 of course)

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.0 rc1 is available

2023-11-06 Thread Even Rouault via gdal-dev


Hi,

I have prepared a GDAL/OGR 3.8.0 release candidate.

NEWS at:

  https://github.com/OSGeo/gdal/blob/v3.8.0RC1/NEWS.md

(small change in it since the initial version I mentioned last week:
there is a new optional dependency: libaec, needed for some GRIB files)

Pick up an archive among the following ones (by ascending size):

  https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0rc1.tar.xz
  https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0rc1.tar.gz
  https://download.osgeo.org/gdal/3.8.0/gdal380rc1.zip

A snapshot of the gdalautotest suite is also available :

https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0rc1.tar.gz
  https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0rc1.zip

A snapshot of the documentation is at:

  https://download.osgeo.org/gdal/3.8.0/gdal380doc.zip

I'll call for a vote promoting it later this week if no serious problems 
are reported before.


The "release/3.8" branch has been created for all bug fixes related to 
3.8.x.

master is now targeting 3.9.0dev.

Best regards,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Segfault with ogr2ogr and Parquet

2023-11-05 Thread Even Rouault via gdal-dev


Scott,

I don't reproduce anymore on master, and I strongly suspect this is the 
same issue as 
https://lists.osgeo.org/pipermail/gdal-dev/2023-November/057856.html 
which I fixed post-beta1


Even

Le 05/11/2023 à 19:57, Scott via gdal-dev a écrit :
ogr2ogr tst.gpkg 
PARQUET:"/vsicurl/https://overturemapswestus2.blob.core.windows.net/release/2023-10-19-alpha.0/theme=admins; 



--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Advice on `GDALCreateAndReprojectImage` and destination no-data value.

2023-11-03 Thread Even Rouault via gdal-dev


Simeon,

    Band 1 Block=210x39 Type=Byte, ColorInterp=Gray

The only way I've been able to get the behavior I need is to open the 
written dataset from the filesystem and set the no-data value 
explicitly in the band, which is inefficient and awkward.


You don't have much alternative than doing that if you use 
GDALCreateAndReprojectImage(). But GDALCreateAndReprojectImage() is a 
"simplistic" warp API that doesn't have all the bells and whistles of 
what gdalwarp lib. You could potentially submit a patch to 
GDALCreateAndReprojectImage() to set the nodata value, which would be 
reasonable, but perhaps you could just bind the GDALWarp() C function 
that is the heart of the gdalwarp binary. You'd get all the advanced 
heuristics of gdalwarp that you don't get with the less advanced warp 
functions.


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-11-03 Thread Even Rouault via gdal-dev





I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers.

Sorry no updated NEWS.md file yet

==> now available at https://github.com/OSGeo/gdal/blob/master/NEWS.md

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question: CPL minizip affected by CVE-2023-45853?

2023-11-03 Thread Even Rouault via gdal-dev


Hi James,

thanks for the notice. GDAL copy has diverged a bit, but I've just 
managed to apply the upstream fix per 
https://github.com/OSGeo/gdal/pull/8658


Even

Le 03/11/2023 à 16:17, James Addison via gdal-dev a écrit :

Hi folks,

I've arrived at the gdal mailing list after reading the security
policy[1] on the GitHub repository, but then decided that this is as
much a question as it is a bug, so I'm following the issue template
comment advice[2] to post here.

The Common Portability Library within gdal includes some code derived
from minizip / Info-ZIP, and while investigating Debian bug #1054290
I've been trying to figure out where else code affected by
vulnerability CVE-2023-45853 could exist.

Could a maintainer confirm whether the affected section of code[3] in
gdal/CPL is vulnerable too?  If so, there is a fix[4] from the zlib
repository (that hosts minizip) that may be straightforward to apply -
and I think that'd be license-compatible to cherry-pick but that's
probably worth confirming.

Thanks,
James
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.7.3 is released

2023-11-03 Thread Even Rouault via gdal-dev


Hi,

On behalf of the GDAL/OGR development team, I am pleased to announce the
release of the GDAL/OGR 3.7.3 bug fix version.

Consult the release notes for the list of issues addressed:
    https://github.com/OSGeo/gdal/blob/v3.7.3/NEWS.md

The sources are available at:

    https://download.osgeo.org/gdal/3.7.3/gdal-3.7.3.tar.xz
    https://download.osgeo.org/gdal/3.7.3/gdal-3.7.3.tar.gz
    https://download.osgeo.org/gdal/3.7.3/gdal373.zip

This will be the last bugfix version in the 3.7.x series.
A 3.8.0 release candidate should follow next week.

Best regards,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release

2023-11-03 Thread Even Rouault via gdal-dev


Motion passed with +1 from PSC members JukkaR, JavierJS, SeanG and me

Le 01/11/2023 à 12:13, Even Rouault via gdal-dev a écrit :

Hi,

Motion:

Adopt GDAL 3.7.3RC1 as 3.7.3 release

Starting with my +1

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Show isBigTIFF in image metadata

2023-11-02 Thread Even Rouault via gdal-dev


Jukka,



Does it feel reasonable? I know that overviews may be standard TIFFs 
while the main image is BigTIFF but maybe the information from the 
header would be enough. Or have I missed some existing tool? I thought 
that tiffinfo at least would report TIFF/BigTIFF but it doesn’t.



tiffdump does:

$ tiffdump byte.tif
byte.tif:
Magic: 0x4949  Version: 0x2a 

vs

$ tiffdump bigtiff.tif
bigtiff.tif:
Magic: 0x4949  Version: 0x2b 

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-02 Thread Even Rouault via gdal-dev


Howard,


I would like to see some language that describes the expectations for 
version compatibility of plugins (ie, what happens with a plugin built 
against x.0 is used against a main library of y.0).


This RFC actually doesn't change anything regarding this. See the 
mention of the GDAL_CHECK_VERSION macro in 
https://gdal.org/tutorials/raster_driver_tut.html#the-driver which 
covers this topic (I've just slightly tuned it to the below version 
which will be online soon):


"Note the use of GDAL_CHECK_VERSION macro. This is a macro that should 
be used by drivers that can be built as a plugin. As the GDAL C++ ABI 
may, and will, change between GDAL feature releases (for example from 
GDAL 3.x.0 to 3.y.0), it is necessary to recompile your driver against 
the header files of the GDAL feature version with which you want to make 
it work. The GDAL_CHECK_VERSION macro will check that the GDAL version 
with which the driver was compiled and the version against which it is 
running are compatible (checking that the major and minor version 
numbers are equal). The C++ ABI will however remain stable for releases 
of the same release branch (that is for bug fixes releases x.y.z of a 
given feature release x.y.0)."


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-11-02 Thread Even Rouault via gdal-dev


Hi Sean,
Rasterio's CI picked up a change to the AAIGrid driver in 3.8. The 3.7 
version driver used to have whitespace before a row and no whitespace 
after. It looks like this has flipped in 3.8. Is it intentional?


Yes: https://github.com/OSGeo/gdal/pull/8362

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-02 Thread Even Rouault via gdal-dev


Hi Rob,


This looks great from my perspective. Are there any downsides?


- A bit of additional coding complexity for driver development, but not 
that much


- As mentioned in the backwards compatibility paragraph, for people 
doing multi-step builds to build first libgdal and then plugins, that 
will require extra care and defining the 
-DGDAL/OGR_DECLARE_DRIVER__FOR_LATER_PLUGIN. Like the condaforge 
build recipee, but I would expect this to be more the exception, than 
what most distributions do. The recommended approach is still to do a 
single build and dispatch the artifacts in separate installable packages 
afterwards




I guess conceptually "Driver X depending on LibW clashes with Driver Z 
depending on LibY" cases are less likely to be hit during unit 
testing, since a particular test-runner/process won't (eventually) be 
loading the full set of drivers + dependencies? Usually that's a LibW 
vs LibY problem though, not a GDAL issue.


Yes that could somehow hide such issues, but as you said, there are 
integration issues, and GDAL can't be expected to reliably detect those. 
Clashes also happen generally at runtime, when you used clashing 
drivers. Just loading the libraries isn't sufficient per se to run into 
the clashes. At least in most situations on Linux.


I've considered yet-another config option like 
GDAL_DEFERRED_PLUGIN_LOADING=NO to disable the new behaviour (meaning to 
fully load the drivers at GDALAllRegister() time), but didn't go for it 
for now as the use cases for it are not obvious. People wanting to 
emulate it can just loop through drivers and call GetMetadata() on them, 
which will force plugin loading,  to simulate that.




Rob :)


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-02 Thread Even Rouault via gdal-dev


Hi,

I'm seeking for feedback and review on a new RFC (RFC 96: Deferred 
in-tree C++ plugin loading),

detailed in https://github.com/OSGeo/gdal/pull/8648, whose summary is:

This RFC adds a mechanism to defer the loading of in-tree C++ plugin 
drivers to
the point where their executable code is actually needed, and converts a 
number
of relevant drivers to use that mechanism. The aim is to allow for more 
modular

GDAL builds, while improving the performance of plugin loading.

(This is material only for GDAL 3.9 of course)

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] WFS: How to use feature ordering?

2023-11-02 Thread Even Rouault via gdal-dev


Craig,

remove the leading end-of-line and spaces in your ExecuteSQL

l = ds.ExecuteSQL(
"""SELECT *
FROM "open-data-platform:v_s_parcel_proposed"
ORDER BY "parcel_pfi" ASC
"""
)

Also fixed 
perhttps://github.com/OSGeo/gdal/commit/f6c7d95e2c66ba1f62f6ff17e31af37f7f8f6bc8

Even

Le 02/11/2023 à 02:50, Craig de Stigter via gdal-dev a écrit :

Hey folks

We're having a lot of trouble getting data out of a WFS server that 
doesn't have primary keys. (previous work on this) 




The URL that's causing us grief is this one: 
https://opendata.maps.vic.gov.au/geoserver/wfs?REQUEST=GetFeature=WFS=2.0.0=open-data-platform:v_s_parcel_proposed=0=2 



* we can't disable pagination, because the server only returns 5000 
records even if we ask for more.

* we can't paginate without ordering, because there's no PK.

So, adding a `=parcel_pfi` to the querystring fixes the issue. 
But I can't seem to get GDAL to do that for us.


There does seem to be code for applying an "ORDER BY" clause from 
ExecuteSQL to the WFS 1.1+ SORTBY field. However, I can't get it to work.


I made a test script 
 
to demonstrate the problem. It uses ExecuteSQL to apply an ORDER BY 
clause, but the resulting curl queries don't have a SORTBY parameter. 
Then it crashes after the server returns a 400 error due to the 
missing SORTBY.


I assume there's something I need to do to cause GDAL to use the 
SORTBY parameter but I'm not sure what it is. Any tips would be 
appreciated.


--
Regards,
Craig

Platform Engineer
Koordinates

+64 21 256 9488  / koordinates.com 
 / @koordinates 


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-11-01 Thread Even Rouault via gdal-dev


Hi Mike,

thanks for the report. Issue fixed in master. I've refreshed the Docker 
image


Note however that this Overture dataset contains 1,390,499,721 features, 
and with our estimate of 83 bytes / feature to build the FlatGeobuf 
RTree, that means you need at least 115 GB RAM on your machine 
(presumably more due to fragmentation etc)


You could potentially save a few minutes by not adding -progress since 
GetFeatureCount() on such dataset requires getting some bits in each of 
the ~1600 files, but I guess that for a process that's going to take a 
few hours, having some display progress justifies a few extra minutes :-)


Even

Le 01/11/2023 à 11:18, Michael Smith a écrit :


Using the ghcr.io/osgeo/gdal:ubuntu-full-latest docker image, I get a 
segfault when trying to convert overture maps to flatgeobuf


ogr2ogr -f flatgeobuf /data/overturemaps_2023_10_19.fgb 
/vsis3/overturemaps-us-west-2/release/2023-10-19-alpha.0/theme=buildings/ 
theme=buildings -progress


…

FlatGeobuf: Spatial index requested will write to temp file and do 
second pass on close


Segmentation fault (core dumped)

But no core file.

--

Michael Smith

Remote Sensing/GIS Center

US Army Corps of Engineers

*From: *gdal-dev  on behalf of Even 
Rouault via gdal-dev 

*Reply-To: *Even Rouault 
*Date: *Tuesday, October 31, 2023 at 1:14 PM
*To: *Rahkonen Jukka , 
"gdal-dev@lists.osgeo.org" 

*Subject: *Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

Le 31/10/2023 à 17:36, Rahkonen Jukka a écrit :

Hi,

I made a simple test with ogr2ogr and geopackage to geopackage on Windows. 
With GDAL 3.8.0dev-3e4dc710a2 (no arrow, old R-Tree) the timing was 36 minutes, 
with GDAL 3.8.0dev-6bbd2c080a the same conversion took 21 minutes. The gpkg 
file is 75 GB, with 120 layers.

Could it be that the new R-Tree implementation does not print anything into 
debug?

Indeed. Debug traces added back in https://github.com/OSGeo/gdal/pull/8640

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___ gdal-dev mailing list 
gdal-dev@lists.osgeo.org 
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release

2023-11-01 Thread Even Rouault via gdal-dev


Hi,

Motion:

Adopt GDAL 3.7.3RC1 as 3.7.3 release

Starting with my +1

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-10-31 Thread Even Rouault via gdal-dev






The ghcr.io/osgeo/gdal:ubuntu-small-latest, 
ghcr.io/osgeo/gdal:ubuntu-full-latest, 
ghcr.io/osgeo/gdal:alpine-normal-latest Docker images have been 
refreshed with 3.8.0beta1 (ghcr.io/osgeo/gdal:alpine-small-latest 
still building at time of writing).

All -latest images are now at 3.8.0beta1


(Note: the GDAL master conda builds mentioned at 
https://gdal.org/download.html#gdal-master-conda-builds have been 
broken for a couple weeks and are thus not usable to test beta1 
currently. I'm investigating)


Issue fixed. The procedure at 
https://gdal.org/download.html#gdal-master-conda-builds will now fetch 
3.8.0beta1



--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-10-31 Thread Even Rouault via gdal-dev



Le 31/10/2023 à 17:36, Rahkonen Jukka a écrit :

Hi,

I made a simple test with ogr2ogr and geopackage to geopackage on Windows. With 
GDAL 3.8.0dev-3e4dc710a2 (no arrow, old R-Tree) the timing was 36 minutes, with 
GDAL 3.8.0dev-6bbd2c080a the same conversion took 21 minutes. The gpkg file is 
75 GB, with 120 layers.

Could it be that the new R-Tree implementation does not print anything into 
debug?


Indeed. Debug traces added back in https://github.com/OSGeo/gdal/pull/8640

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: OSGeo Community Sprint Financial Support

2023-10-31 Thread Even Rouault via gdal-dev


+1 Even

Le 31/10/2023 à 17:59, Howard Butler via gdal-dev a écrit :

Dear PSC,

Sorry for the short notice, but I would like to motion that the GDAL 
Sponsorship Program financially support GDAL PSC members who wish to attend the 
OSGeo Community Sprint in Vienna [1] next week. Support level is capped at 
1000€ for European attendees and 1500€ for others.

Howard

[1] https://wiki.osgeo.org/wiki/OSGeo_Community_Sprint_2023
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.8.0beta1 available for testing

2023-10-31 Thread Even Rouault via gdal-dev


Hi,

I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers.

Sorry no updated NEWS.md file yet, but I'd in particular be interesting 
by testing of ogr2ogr workflows, since they have underwent significant 
changes in the underlying implementation:


- when the source layer is a layer implementing the ArrowStream API 
(that is  GeoPackage, FlatGeoBuf, Arrow or Parquet), and when no ogr2ogr 
options than -of, -where, -spat, -lco, -dsco, -gt, 
-append/-overwrite/-update are used (and -sql as well for GeoPackage). 
When enabling CPL_DEBUG, you'll see a "OGR2OGR: Using WriteArrowBatch()" 
trace when that new code path is taken. If specifying other options, the 
feature-iteration-based traditional implementation is used


- and/or when the output layer is GeoPackage (new layer), due to the 
revamped much faster spatial index creation.  This enhanced spatial 
index creation is not ogr2ogr specific and is actually available more 
generally for CreateLayer() + CreateFeature() or CreateLayer() + 
WriteArrowBatch() scenarios.


Point of attention would be when in situations with large files and/or 
with low RAM.


The ghcr.io/osgeo/gdal:ubuntu-small-latest, 
ghcr.io/osgeo/gdal:ubuntu-full-latest, 
ghcr.io/osgeo/gdal:alpine-normal-latest Docker images have been 
refreshed with 3.8.0beta1 (ghcr.io/osgeo/gdal:alpine-small-latest still 
building at time of writing).


(Note: the GDAL master conda builds mentioned at 
https://gdal.org/download.html#gdal-master-conda-builds have been broken 
for a couple weeks and are thus not usable to test beta1 currently. I'm 
investigating)


Source snapshots at:

- https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.gz

- https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.xz

- https://download.osgeo.org/gdal/3.8.0/gdal380beta1.zip

Autotest snapshots:

- https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.tar.gz

- https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.zip

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.7.3 RC1 available

2023-10-30 Thread Even Rouault via gdal-dev



Le 30/10/2023 à 18:11, Scott via gdal-dev a écrit :

Looking at the release notes did #8262 make it in?

https://github.com/OSGeo/gdal/pull/8262


no, this is 3.8.0 only material


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.7.3 RC1 available

2023-10-30 Thread Even Rouault via gdal-dev


Hi,

I have prepared a GDAL/OGR 3.7.3 release candidate.

Pick up an archive among the following ones (by ascending size):

  https://download.osgeo.org/gdal/3.7.3/gdal-3.7.3rc1.tar.xz
  https://download.osgeo.org/gdal/3.7.3/gdal-3.7.3rc1.tar.gz
  https://download.osgeo.org/gdal/3.7.3/gdal373rc1.zip

A snapshot of the gdalautotest suite is also available:

https://download.osgeo.org/gdal/3.7.3/gdalautotest-3.7.3rc1.tar.gz
  https://download.osgeo.org/gdal/3.7.3/gdalautotest-3.7.3rc1.zip

The NEWS file is here:

  https://github.com/OSGeo/gdal/blob/v3.7.3RC1/NEWS.md

I'll call for a vote promoting it to final later this week if no
serious problems are reported before.

Best regards,

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Tag for units in GeoTIFF

2023-10-26 Thread Even Rouault via gdal-dev



Le 26/10/2023 à 18:50, David Shean a écrit :

Hi Even and Javier,
For the special case of a DEM, would it make more sense to specify the 
units with a vertical CRS?  This approach would define the LENGTHUNIT 
and the associated vertical datum.


Example for EPSG:4979:

        AXIS["ellipsoidal height (h)",up,
            ORDER[3],
            LENGTHUNIT["metre",1]],

Example for EPSG:5703:

VERTCRS["NAVD88 height",
    VDATUM["North American Vertical Datum 1988"],
    CS[vertical,1],
        AXIS["gravity-related height (H)",up,
            LENGTHUNIT["metre",1]],
...

More complicated than specifying the generic UNITTYPE in raster 
metadata, but seems like best practice.


This is actually already implemented (deducing the value of 
GetUnitType() from the vertical component):


$ gdal_translate byte.tif out.tif -a_srs EPSG:4269+5703
$ gdalinfo out.tif | grep Unit
  Unit Type: foot

$ gdal_translate byte.tif out.tif -a_srs EPSG:4269+8228
$ gdalinfo out.tif | grep Unit
   Unit Type: foot



-David


On Oct 26, 2023, at 4:45 AM, Javier Jimenez Shaw via gdal-dev 
 wrote:


Thanks!

On Thu, 26 Oct 2023 at 12:27, Even Rouault via gdal-dev 
 wrote:


Javier,

Le 26/10/2023 à 11:59, Javier Jimenez Shaw via gdal-dev a écrit :

Hi

Using a GeoTIFF with a single band and float values can be very
useful to show any distribution over the terrain. One typical
example is DSM (digital surface model), where the value is the
elevation on every point.

Is there any standard (or accepted) metadata to say the units of
those values?


API wise in GDAL, you should use the
GDALRasterBand::SetUnitType() to set the unit

For TIFF, this will be encoded in the GDAL_METADATA TIFF tag like
the following:


  foo


Regarding the value itself, it is mostly unspecified by GDAL. It
could be a good practice to suggest to use strings recognized by
the "udunits" package (when possible), like done in the netCDF CF
conventions

(http://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#units)

Even



In the typical case of DSM it is usually "meter" or "foot". But
it can be something else, like "people/km2" for population, or
"mm" for rain precipitation, or "kg/ha" for agricultural yield.

Thanks
.___ ._ ..._ .. . ._. .___ .. __ . _. . __..  ...  ._ .__

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
http://www.spatialys.com

My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Tag for units in GeoTIFF

2023-10-26 Thread Even Rouault via gdal-dev


Javier,

Le 26/10/2023 à 11:59, Javier Jimenez Shaw via gdal-dev a écrit :

Hi

Using a GeoTIFF with a single band and float values can be very useful 
to show any distribution over the terrain. One typical example is DSM 
(digital surface model), where the value is the elevation on every point.


Is there any standard (or accepted) metadata to say the units of those 
values?


API wise in GDAL, you should use the GDALRasterBand::SetUnitType() to 
set the unit


For TIFF, this will be encoded in the GDAL_METADATA TIFF tag like the 
following:



  foo


Regarding the value itself, it is mostly unspecified by GDAL. It could 
be a good practice to suggest to use strings recognized by the "udunits" 
package (when possible), like done in the netCDF CF conventions 
(http://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#units)


Even



In the typical case of DSM it is usually "meter" or "foot". But it can 
be something else, like "people/km2" for population, or "mm" for rain 
precipitation, or "kg/ha" for agricultural yield.


Thanks
.___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ...  ._ .__

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression

2023-10-16 Thread Even Rouault via gdal-dev



Le 16/10/2023 à 17:14, Kurt Schwehr a écrit :

Thanks for the heads up!

Can you share that SHAs of the fix and cause?


master: d083af1ec9c38e79bfb0532885570de6e5b8a410

3.7 branch (should apply to 3.6 as well): 
b5858ed5bc5004c97f7cd6000674015bdc70b33b


cause: a worker thread of the multithreaded decoding could, in a 
situation where the block cache is full, cause a "dirty" (ie modified 
but not yet serialized to disk) block to be written, resulting in either 
a deadlock between the lock of the multithreaded decoder and the lock of 
the job queue mechanism, or other decoding threads could see their file 
handle being seek() by another thread.  In short: multithreading is hard.




On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev 
 wrote:


Hi,

For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff
compression+decompression, in particular within the same file, as for
example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow
can lead to deadlocks (process stalled forever) and/or data
corruption
(sometimes without errors at generation). Cf
https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The
fix is
in https://github.com/OSGeo/gdal/pull/8561

The issue is particularly visible on Windows, or more generally any
operating system (or file system where the output file is located)
which
has no VSIVirtualHandle::PRead() implementation, but it can also be
occasionally reproduced on Linux (at least as a deadlock).

Even

-- 
http://www.spatialys.com

My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression

2023-10-16 Thread Even Rouault via gdal-dev


Hi,

For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff 
compression+decompression, in particular within the same file, as for 
example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow 
can lead to deadlocks (process stalled forever) and/or data corruption 
(sometimes without errors at generation). Cf 
https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is 
in https://github.com/OSGeo/gdal/pull/8561


The issue is particularly visible on Windows, or more generally any 
operating system (or file system where the output file is located) which 
has no VSIVirtualHandle::PRead() implementation, but it can also be 
occasionally reproduced on Linux (at least as a deadlock).


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Reading a shapefile containing multi-parts Polyline from OGR (C++ API)

2023-10-16 Thread Even Rouault via gdal-dev



Le 16/10/2023 à 15:15, Javier Jimenez Shaw via gdal-dev a écrit :

Do you mean a MultiLineString?

This piece of code is working for me (I hope without any bug).

if (wkbFlatten(poGeometry->getGeometryType()) == wkbMultiLineString)
multiLineStringGeometry(poGeometry);
...
multiLineStringGeometry(OGRGeometry* poGeometry)
            {
                OGRMultiLineString* multiLine = 
dynamic_cast(poGeometry);

                int numGeometries = multiLine->getNumGeometries();
                for (int i = 0; i < numGeometries; i++)
                {
                    OGRGeometry* eachGeometry = 
multiLine->getGeometryRef(i);
                    OGRLineString* line = 
dynamic_cast(eachGeometry);

                    // do whatever with that LineString
                }
            }


Can further be simplified to something like:

multiLineStringGeometry(const OGRGeometry* poGeometry)
            {
                for (const OGRLineString* line: 
poGeometry->toMultiLineString())

                {
                    // do whatever with that LineString
                }
            }



On Mon, 16 Oct 2023 at 12:14, Roland Baviere via gdal-dev 
 wrote:


Hi all,

 I hope this e-mel finds you well.

I am trying to read a shapefile layer from OGR from a C++
application following the tutorial found here:
https://gdal.org/tutorials/vector_api_tut.html#reading-from-ogr

We have difficulties with a layer containing PolyLines with
several parts.

The pre-existing code works fine for polylines having a unique part.

Do you have any example or advice to help me implement a version
that works for polylines containing several parts. My intention is
to display each shape on a map, so I need to retrieve the
coordinates of points and draw segment between points when relevant.

Thanks a lot for your help.

Kind regards,

Roland



 CODE THAT WORKS for "simple" polylines but fails when
dealiing with a polyline containing several parts
int o = 0;
                        for(auto y : poGeometry->toLineString()) {
                            if (o == 0) {
                                line.mInCoord =
QGeoCoordinate(y.getX(), y.getY());
                            } else if (o ==
(poGeometry->toLineString()->getNumPoints() - 1)) {
                                line.mOutCoord =
QGeoCoordinate(y.getX(), y.getY());
                            } else {
line.mPath.addCoordinate(QGeoCoordinate(y.getX(), y.getY()));
                            }
                            qDebug() << o;
                            o++;
                        }

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Performance regression testing/benchmarking for CI

2023-10-15 Thread Even Rouault via gdal-dev

Le 15/10/2023 à 13:34, Javier Jimenez Shaw via gdal-dev a écrit :

Hi Even. Thanks, it sounds good.
However I see a potential problem. I see that you use once 
"SetCacheMax". We should not forget about that in the future for 
sensible tests. The cache of gdal is usually a percentage of the total 
memory, that may change among the environments and time.

Javier,

What is sure is that the timings got in one session of the perf tests in 
CI are comparable to nothing else that timings done in the same session 
(and that's already challenging!). So the effect of the RAM available in 
the CI worker might affect the speed of the tests, but it will affect it 
in the same way for both the reference run and the tested run (while the 
GDAL_CACHEMAX=5% setting remains the same and the general working of the 
block cache remains similar). I anticipate that at some points changes 
in GDAL might make the perf test suite no longer comparable to the 
current reference version and that we will have to upgrade the commit of 
the reference version while that happens. Actually if the perf test 
suite is extended, it might be useful to upgrade the commit of the 
reference version at release time of feature releases. For example, when 
GDAL 3.8.0 is released, it will become the reference point for 3.9.0 
development, and so on (otherwise we wouldn't get perf regression 
testing of added tests). The downside of this is that this wouldn't 
catch progressive slowdowns over several release cycles. But given that 
I had to raise the threshold for failure to > 30% regression to avoid 
false positives, the perf test suite (at least when run in CI with all 
its unpredictability) can only catch major "instant" regressions.

Even

On Wed, 11 Oct 2023, 07:53 Laurențiu Nicola via gdal-dev, 
 wrote:

Hi,

No experience with pytest-benchmark, but I maintain an unrelated
project that runs some benchmarks on CI, and here are some things
worth mentioning:

 - we store the results as a newline-delimited JSON file in a
different GitHub repository

(https://raw.githubusercontent.com/rust-analyzer/metrics/master/metrics.json,
warning, it's a 5.5 MB unformatted JSON)
 - we have an in-browser dashboard that retrieves the whole file
and displays them: https://rust-analyzer.github.io/metrics/
 - we do track build time and overall run time, but we're more
interested in correctness
 - the display is a bit of a mess (partly due to trying to keep
the setup as simple as possible), but you can look for the "total
time", "total memory" and "build" to get an idea
 - we store the runner CPU type and memory in that JSON; they're
almost all Intel, but they do upgrade from time to time
 - we even have two AMD EPYC runs, note that boost is disabled in
a different way there (we don't try to disable it, though)
 - we also try to measure the CPU instruction count (the perf
counter), but it doesn't work on GitHub and probably in most VMs
 - the runners have been very reliable, but not really consistent
in performance
 - a bigger problem for us was that somebody actually needs to
look at the dashboard to spot any regressions and investigate them
(some are caused by external changes)
 - in 3-5 years we'll probably have to trim down the JSON or
switch to a different storage

    Laurentiu

On Tue, Oct 10, 2023, at 21:08, Even Rouault via gdal-dev wrote:
> Hi,
>
> I'm experimenting with adding performance regression testing in
our CI.
> Currently our CI has quite extensive functional coverage, but
totally
> lacks performance testing. Given that we use pytest, I've spotted
> pytest-benchmark
(https://pytest-benchmark.readthedocs.io/en/latest/) as
> a likely good candidate framework.
>
> I've prototyped things in https://github.com/OSGeo/gdal/pull/8538
>
> Basically, we now have a autotest/benchmark directory where
performance
> tests can be written.
>
> Then in the CI, we checkout a reference commit, build it and run
the
> performance test suite in --benchmark-save mode
>
> And then we run the performance test suite on the PR in
> --benchmark-compare mode with a --benchmark-compare-fail="mean:5%"
> criterion (which means that a test fails if its mean runtime is 5%
> slower than the reference one)
>
>  From what I can see, pytest-benchmark behaves correctly if
tests are
> removed or added (that is not failing, just skipping them during
> comparison). The only thing one should not do is modify an
existing test
> w.r.t the reference branch.
>
> Does someone has practical experience of pytest-benchmark, in
particular
> in CI setups? With virtualization, it is hard to g

Re: [gdal-dev] including higher-level library in GPX creator string

2023-10-13 Thread Even Rouault via gdal-dev

Cf https://github.com/OSGeo/gdal/pull/8558: "GPX: add a CREATOR dataset 
creation option"


It will be of course up to the application / application user to set it.

I've considered a global GDALSetDefaultCreatorApplication() (could be 
similar to CPLHTTPSetDefaultUserAgent(): 
https://gdal.org/doxygen/cpl__http_8h.html#ae83c06d51089bb340ed0b98a6ed8d01d) 
so that an application could set a default for all formats that write a 
creator application, but this leads to introducing potential options to 
express "totally replace the GDAL version" vs "add my name in addition 
to the GDAL version", or "put application/version number only when 
required by the format" vs "put application/version wherever the format 
could allow it, but only if there's a standardized location to do so" vs 
"put application/version wherever the format could allow, potentially it 
in a custom metadata item if there's no standardized location". e.g in 
netCDF, we have a custom attribute GDAL = "GDAL x.y.z, released 
/MM/DD" (it would not be appropriate to put the identification of 
the larger application in it, given the name of the attribute and the 
fact that GDAL itself parses it on reading to determine the bottom-up vs 
top-up orientaiton).  So I've not pursued this given all those dilemmas :-)


Even

Le 13/10/2023 à 14:52, Greg Troxel via gdal-dev a écrit :

I am a member of the GPX standards list, and we are discussing a new
version and also dealing with broken data.

I wrote a gpx (from geojson), with just ogr2ogr, and the creator looks
like this:

   creator="GDAL 3.5.3"

that's arguably totally fine with ogr2ogr.

But it seems that if some random big program links with gdal and uses it
to produce a file, the creator is just GDAL, and not "qgis 3.28.5 via
GDAL 3.5.3" or something like that.  I groveled over all my gpx files
and found none that indicated anything like this.

It would be helpful to the GPX community if files were attributed to the
larger software as well, so that they can be chased down or at least
understood to be separately broken.

I don't see anything at
   https://gdal.org/drivers/vector/gpx.html#vector-gpx
that could be used to do this.

Does this make sense to people here?

Is there any facility that would do this, that people just aren't using?

Any other thoughts?

Greg
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Cannot choose netCDF driver instead of HDF5 for nc files

2023-10-12 Thread Even Rouault via gdal-dev


Works for me with GDAL 3.6, 3.7 and master on Ubuntu 20.04

$ gdalinfo 
/vsicurl/https://nbstds.met.no/thredds/fileServer/NBS/S2B/2022/07/03/S2B_MSIL1C_20220703T112119_N0400_R037_T32VLN_20220703T121203.nc

Driver: netCDF/Network Common Data Format
[...]

/vsicurl/ functionality for the netCDF driver requires using a specific 
system call of Linux. See 
https://gdal.org/drivers/raster/netcdf.html#vsi-virtual-file-system-api-support


Le 12/10/2023 à 14:01, Stefan Blumentrath via gdal-dev a écrit :

Hi,
I am wondering if the behavior of the NetCDF driver changed in recent 
GDAL version(s)?
I do have the netCDF driver installed and also HDF5, which has been 
shaddowing the netCDF driver. So I used to do the following:

export GDAL_SKIP=HDF5
gdalinfo 
/vsicurl/https://nbstds.met.no/thredds/fileServer/NBS/S2B/2022/07/03/S2B_MSIL1C_20220703T112119_N0400_R037_T32VLN_20220703T121203.nc

But this gives me now:
ERROR 4: 
`/vsicurl/https://nbstds.met.no/thredds/fileServer/NBS/S2B/2022/07/03/S2B_MSIL1C_20220703T112119_N0400_R037_T32VLN_20220703T121203.nc' 
not recognized as a supported file format.

(or in Python syntax:
from osgeo import gdal
gdal.GetDriverByName("HDF5").Deregister()
ds = 
gdal.Open("/vsicurl/https://nbstds.met.no/thredds/fileServer/NBS/S2B/2022/07/03/S2B_MSIL1C_20220703T112119_N0400_R037_T32VLN_20220703T121203.nc;)

)
If I do not deregister the HDF5 driver the dataset is opened, but with 
the HDF5 driver, and I want to use the netCDF driver for those 
CF-convention files...

I am on Ubuntu 22.04, GDAL 3.6.4, released 2023/04/17...
Any idea how that can be solved? Is it a bug or a feature?
Thanks for helping in advance!
Cheers
Stefan

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Performance regression testing/benchmarking for CI

2023-10-10 Thread Even Rouault via gdal-dev


Hi,

I'm experimenting with adding performance regression testing in our CI. 
Currently our CI has quite extensive functional coverage, but totally 
lacks performance testing. Given that we use pytest, I've spotted 
pytest-benchmark (https://pytest-benchmark.readthedocs.io/en/latest/) as 
a likely good candidate framework.


I've prototyped things in https://github.com/OSGeo/gdal/pull/8538

Basically, we now have a autotest/benchmark directory where performance 
tests can be written.


Then in the CI, we checkout a reference commit, build it and run the 
performance test suite in --benchmark-save mode


And then we run the performance test suite on the PR in 
--benchmark-compare mode with a --benchmark-compare-fail="mean:5%" 
criterion (which means that a test fails if its mean runtime is 5% 
slower than the reference one)


From what I can see, pytest-benchmark behaves correctly if tests are 
removed or added (that is not failing, just skipping them during 
comparison). The only thing one should not do is modify an existing test 
w.r.t the reference branch.


Does someone has practical experience of pytest-benchmark, in particular 
in CI setups? With virtualization, it is hard to guarantee that other 
things happening on the host running the VM might not interfer. Even 
locally on my own machine, I initially saw strong variations in timings, 
which can be reduced to acceptable deviation by disabling Intel 
Turboboost feature (echo 1 | sudo tee 
/sys/devices/system/cpu/intel_pstate/no_turbo)


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] OAPIF: how does the driver utilize schema?

2023-10-10 Thread Even Rouault via gdal-dev


Hi Jukka,


The documentation in 
https://gdal.org/drivers/vector/oapif.html#layer-schema is wrong when 
it claims this:


OGR needs a fixed schema per layer, but OGC API - Features Core 
doesn't impose fixed schema. So the driver will retrieve the first 
page of features (10 features) and establish a schema from this.


I can see from the debug info and from the source code that if GDAL 
finds a relation of type “describedBy” for a collection then it reads 
and interprets the schema from the link. However, I am not sure if 
 GDAL in this case trusts the schema and skips scanning of the first 
ten features for harvesting the attributes.


It doesn't fully trust the schema provided by describedBy and still 
scans the first page of feature to complement it. Doc adjusted


BTW, the name of the relation type has been changed into “describedby” 
due to https://github.com/opengeospatial/ogcapi-features/issues/338. 
It is better to support both describedBy and describedby.



Thanks. Fixed too (GDAL made case insensitive comparisons)

Even


-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] format for topology?

2023-10-09 Thread Even Rouault via gdal-dev


Michael,

you won't find much support for that in GDAL itself. You'd better have a 
look at Spatialite topology, PostGIS topology or GRASS topology. The 
process is basically: import your geometries as a topology,  do 
reprojection or other work in the topology domain, and go back to the 
geometry domain.


QGIS has a plugin to work with PostGIS topology: 
https://plugins.qgis.org/plugins/pgtopoeditor/


(I was involved in discussions yeaaars ago about having GDAL offering a 
generic API and translation between all those topology formats but that 
never materialized)


Even

Le 09/10/2023 à 21:55, Michael Sumner via gdal-dev a écrit :
Are there any formats that record "coverage" topology? What I'm 
worried about is when shapes are encoded as blob geometry with 
initially identical  coordinates at shared vertices, is there any 
process that can validate or record that particular coords should have 
the same values even after reprojection or decomposition/rebuild?


I know how to do this in custom ways, I'm hoping to find some 
off-the-shelf process to at least refer to (I'm a bit pressed for time 
so I'm asking without doing the extra homework that's needed).  I 
would build an indexer of the unique coordinates and how they relate 
to the blob geometries before and after and do checks there, but I 
don't know of an existing process that guarantees this.


I understand that precision-setting is used to prevent numeric loss, 
but is there any formalization about that loss? (i.e.  different 
instances of a vertex really do stay unique). I guess the result is 
going to be pretty good if you choose the precision appropriately and 
modifications aren't extreme?


Thanks for consideration!

Cheers, Mike






--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] The relationship between overview levels and zoom levels?

2023-10-08 Thread Even Rouault via gdal-dev


James,

It all depends on how software will deal with overviews. Some might 
explicitly have logic where it inspects which overviews are present and 
decide which one to use. If we are talking about a client like QGIS that 
uses the RasterIO() method on the full resolution band with a window of 
interest (nXOff, nYOff, nXSize, nYSize) and a target buffer size 
(nBufXSize, nBufYSize),  GDAL computes a target downscaling factor for 
that which is the min(nXSize/nBufXSize, nYSize/nBufYSize).  It then 
walks through overviews and will select the frist one whose downscaling 
factor is greater than target_dowscanling_factor / 1.2


If you have overviews at [2,4,8,16]

target_downscaling_factor    selected_overview
-    --
]0, 2. /1.2[     full resolution band
[2. / 1.2, 4. / 1.2[     2x downsampled
[4. / 1.2, 8. / 1.2[     4x downsampled
[8. / 1.2, 16. / 1.2[    8x downsampled
[16. / 1.2, inf[ 16x downsampled

If you just have overview [16]

target_downscaling_factor    selected_overview
-    --
]0, 16. /1.2[    full resolution band
[16. / 1.2, inf[ 16x downsampled

I've submitted in https://github.com/OSGeo/gdal/pull/8526 a doc 
improvement capturing above explanations


Even


Le 08/10/2023 à 00:01, James Sammone via gdal-dev a écrit :
I'm not sure if this is the best channel to ask this question as it 
might be beyond the scope, but I've asked it in a few others and have 
had no responses aside from others also being curious.


I am trying to understand the relationship between overviews and zoom 
levels so I know how to make more efficient Cloud Optimized GeoTIFFs. 
Using gdaladdo or gdal.BuildOverviews(), I can create overviews at 
[2,4,8,16] or at just [16]. From my understanding, this means the size 
is being divided by those values to provide downsample arrays of the 
original source. In the first example [2,4,8,16], I've created 4 
separate overview arrays into the GeoTIFF that are 2x, 4x, 8x, and 16x 
downsampled. And in the second example using only [16], I've built one 
overview array into the GeoTIFF that is 16x downsampled.


How can I understand how these overviews are applied when it comes to 
zoom levels? Does the 16x downsample appear sooner in the second 
example when zooming out than for the first example due to being first 
in order? Or do the 16x downsamples appear at the same zoom level for 
both cases but the second example has additional 2x, 4x, and 8x 
downsamples that also appear before getting there?




Thanks for any insight into this anyone can provide. Despite using 
overviews all the time, I've struggled with this for a while and had 
largely consigned to not understanding it.


Best,

James


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Questions regarding temporary database file (ogr2ogr with MVT driver)

2023-10-06 Thread Even Rouault via gdal-dev


You could possibly paralle

Your text cut off at the interesting bit, could you elaborate on this @Even?


ah I meant you could parallelize generation on several machines, each 
one with a different extent and/or zoom level


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Default page size of the OAPIF driver

2023-10-06 Thread Even Rouault via gdal-dev


Hi Jukka,

yes it would make sense for the driver to use a larger page size. You 
can file an issue about that


Actually we could reuse the same logic as the QGIS OAPIF provider that 
determines the page size from the limit.schema.maximum and 
limit.schema.default values of the /api response:


    if ( apiRequest.defaultLimit() > 0 && apiRequest.maxLimit() > 0 )
    {
  // Use the default, but if it is below 1000, aim for 1000
  // but clamp to the maximum limit
  mShared->mPageSize = std::min( std::max( 1000, 
apiRequest.defaultLimit() ), apiRequest.maxLimit() );

    }
    else if ( apiRequest.defaultLimit() > 0 )
  mShared->mPageSize = std::max( 1000, apiRequest.defaultLimit() );
    else if ( apiRequest.maxLimit() > 0 )
  mShared->mPageSize = apiRequest.maxLimit();
    else
  mShared->mPageSize = 100; // fallback to arbitrary page size


Even

Le 06/10/2023 à 16:39, Rahkonen Jukka via gdal-dev a écrit :


Hi,

The OAPIF driver is using page size of 10 features by default 
https://gdal.org/drivers/vector/oapif.html#open-options.


“PAGE_SIZE=: Defaults to 10. Number of features to retrieve 
per request. Minimum is 1, maximum 1.”


I know that 10 is the default page size (“limit”) also in the standard 
but I think that this is too small page size, both for the users and 
especially for the service providers. Most users run GDAL utilities 
with the defaults. The buildings collection in our OGC API Features 
service has about 5 million features, which means that GDAL users are 
fetching the data by sending half a million request. That is much 
slower for the users than making 500 request, 1 features each. And 
think about our log files.


The better default page size would be the biggest that the OAPIF 
server supports. By the standard the maximum is 1 but service 
providers may use bigger or smaller maximum page sizes. By the 
standard the server can always return less features than client is 
asking with “limit=” and the client must be prepared to follow the 
next links.


My suggestion for the new page size default is 1. If that feels 
too big for some reason, then would 1000 feel better?


-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Questions regarding temporary database file (ogr2ogr with MVT driver)

2023-10-06 Thread Even Rouault via gdal-dev


Hi,,


I’m creating vector tiles from a PostGIS database using ogr2ogr with 
the MVT driver. The region is quite large and I’m creating tiles for 
levels 0-15, so the process takes quite some time (hours). The tiles 
are written directly to a S3 storage using the /vsis3/ virtual file 
system (see: 
https://gdal.org/user/virtual_file_systems.html#vsis3-aws-s3-files).


Looking at the logs, I can see that, firstly, a “temporary database” 
file (*.temp.db) is created. Some things are notable:


- this step seems to be CPU intensive compared to the creation of the 
actual pbf files
- the temporary database file is not being written to the output file 
directory (S3 storage in this case) as stated in the documentation


I don't observe this:

$ ogr2ogr /vsis3/X/foo.mvt poly.shp
ERROR 1: wb+ not supported for /vsis3/, unless 
CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE is set to YES


vs

$ ogr2ogr /vsis3/spatialys/foo.mvt poly.shp -dsco 
TEMPORARY_DB=/tmp/my.temp.db


- the dataset creation option `TEMPORARY_DB` (see: 
https://gdal.org/drivers/vector/mvt.html#dataset-creation-options) 
seems to be ignored (I tried writing the file to a local directory but 
no such file shows up)
On Unix, the file is immediately deleted after being opened, so that if 
you kill the process there's no remaining temporary file. You can set 
the OGR_MVT_REMOVE_TEMP_FILE env variable to NO to disable this
- my guess is that the file is only kept in memory (which would be 
/vsimem/ if I’m correct)


Is it possible to skip the step of first creating the temp db and 
directly create tiles from PostGIS or any other data source?

No
Where/how exactly is the database file stored? Any ideas on optimizing 
the process?


You could possibly paralle

Also, the MVT generator is intended to be basic. You might perhaps have 
more functionality with https://github.com/felt/tippecanoe




I’m looking forward to some discussion on this topic. I will happily 
try to contribute where I can.


Regards,

Leo

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSIGetMemFileBuffer Returns NULL When Copying Vector Dataset to Vsimem

2023-10-05 Thread Even Rouault via gdal-dev



Le 06/10/2023 à 01:11, Soren Saville Scott a écrit :

Hi Even,

Suggests something wrong with my build. Is there possibly something 
strange happening across DLL boundaries? Do you know of any build 
flags that could change the VSIMEM behaviour?
I'm not seeing anything suspicious here, especially because 
VSIGetMemFileBuffer() is in the C API.


As a secondary question, for multi-file datasets (like shapefiles) 
does VSIGetMemFileBuffer() encapsulate all those files - i.e. I should 
be able to call VSIFileFromMemBuffer() on that buffer and GDALOpen() 
and get the same information back?


You need to read each file separately. VSIGetMemFileBuffer() works at 
the file level. It has no idea of what a GDAL dataset is.


You can use GDALGetFileList() to get the list of files of a dataset. You 
might want to do that on a dataset after re-opening, since there has 
been a recent bug fix w.r.t GetFileList() on shapefiles just created 
where one of the side car file wasn't listed




Thanks for your help,
Soren

On Wed, 4 Oct 2023 at 21:09, Even Rouault  
wrote:


Hi,

size_t is an incorrect size for DataLength (if you run on 32 bit
architecture). You should use vsi_l_offset.

Besides that the following standalone inspired from your code
works fine for me:

#include 
#include "gdal_priv.h"

int main()
{
    GDALAllRegister();
    GDALDataset* Dataset
=GDALDataset::FromHandle(GDALOpenEx("poly.shp", GDAL_OF_VECTOR |
GDAL_OF_VERBOSE_ERROR, nullptr, nullptr, nullptr));
    GDALDataset* Copy =
Dataset->GetDriver()->CreateCopy("/vsimem/foo.shp", Dataset,
false, nullptr, nullptr, nullptr); //appears to produce a valid
Dataset containing the same layers & features as its source.
    GDALClose(Dataset);
    GDALClose(Copy);
    vsi_l_offset DataLength = 0;
    GByte* Buffer = VSIGetMemFileBuffer("/vsimem/foo.shp",
, true);
    printf("%p %d\n", Buffer, int(DataLength)); // buffer not null
and DataLength > 0
    CPLFree(Buffer);
    VSIUnlink("/vsimem/foo.dbf");
    VSIUnlink("/vsimem/foo.shx");
    return 0;
}

Even

Le 04/10/2023 à 06:26, Soren Saville Scott via gdal-dev a écrit :

Hi,

I'm integrating GDAL in C++ with a system that defines its own
asset types stored on disk, and am writing a wrapper for datasets
using this asset system. To 'import' a dataset I call GDALOpenEx
on it, CreateCopy() it to some "/vsimem/foo.shp" path,
GDALClose() the datasets to flush it, and call
VSIGetMemFileBuffer with bUnlinkAndSeize=true to take ownership
of the underlying memory buffer.

GDALDataset* Dataset = GDALOpenEx("foo.shp", GDAL_OF_VECTOR |
GDAL_OF_VERBOSE_ERROR, nullptr, nullptr, nullptr);
GDALDataset* Copy =
Dataset->GetDriver()->CreateCopy("/vsimem/foo.shp", Dataset,
false, nullptr, nullptr, nullptr); //appears to produce a valid
Dataset containing the same layers & features as its source.
GDALClose(Dataset);
GDALClose(Copy);
size_t DataLength;
GByte* Buffer = VSIGetMemFileBuffer("/vsimem/foo.shp",
, true); //<-- returns NULL

For some reason VSIGetMemFileBuffer always fails, returning a
nullptr. This is basically identical to a similar problem I had,
described
here: 
https://stackoverflow.com/questions/73644157/get-raw-buffer-for-in-memory-dataset-in-gdal-c-api

<https://stackoverflow.com/questions/73644157/get-raw-buffer-for-in-memory-dataset-in-gdal-c-api>
 which
I never really found a solution to. I have similar issues with
other VSI functions, e.g. calling GDALOpenEx on a path created by
VSIFileFromMemBuffer also fails.

Any idea as to what the problem is here?

Alternatively, maybe there's a better way to 'emplace' datasets
inside some other file?

Thanks,
Soren Saville Scott

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
http://www.spatialys.com

My software is free, but my time generally not.


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] VSIGetMemFileBuffer Returns NULL When Copying Vector Dataset to Vsimem

2023-10-04 Thread Even Rouault via gdal-dev


Hi,

size_t is an incorrect size for DataLength (if you run on 32 bit 
architecture). You should use vsi_l_offset.


Besides that the following standalone inspired from your code works fine 
for me:


#include 
#include "gdal_priv.h"

int main()
{
    GDALAllRegister();
    GDALDataset* Dataset 
=GDALDataset::FromHandle(GDALOpenEx("poly.shp", GDAL_OF_VECTOR | 
GDAL_OF_VERBOSE_ERROR, nullptr, nullptr, nullptr));
    GDALDataset* Copy = 
Dataset->GetDriver()->CreateCopy("/vsimem/foo.shp", Dataset, false, 
nullptr, nullptr, nullptr); //appears to produce a valid Dataset 
containing the same layers & features as its source.

    GDALClose(Dataset);
    GDALClose(Copy);
    vsi_l_offset DataLength = 0;
    GByte* Buffer = VSIGetMemFileBuffer("/vsimem/foo.shp", , 
true);
    printf("%p %d\n", Buffer, int(DataLength)); // buffer not null and 
DataLength > 0

    CPLFree(Buffer);
    VSIUnlink("/vsimem/foo.dbf");
    VSIUnlink("/vsimem/foo.shx");
    return 0;
}

Even

Le 04/10/2023 à 06:26, Soren Saville Scott via gdal-dev a écrit :

Hi,

I'm integrating GDAL in C++ with a system that defines its own asset 
types stored on disk, and am writing a wrapper for datasets using this 
asset system. To 'import' a dataset I call GDALOpenEx on it, 
CreateCopy() it to some "/vsimem/foo.shp" path, GDALClose() the 
datasets to flush it, and call VSIGetMemFileBuffer with 
bUnlinkAndSeize=true to take ownership of the underlying memory buffer.


GDALDataset* Dataset = GDALOpenEx("foo.shp", GDAL_OF_VECTOR | 
GDAL_OF_VERBOSE_ERROR, nullptr, nullptr, nullptr);
GDALDataset* Copy = 
Dataset->GetDriver()->CreateCopy("/vsimem/foo.shp", Dataset, false, 
nullptr, nullptr, nullptr); //appears to produce a valid Dataset 
containing the same layers & features as its source.

GDALClose(Dataset);
GDALClose(Copy);
size_t DataLength;
GByte* Buffer = VSIGetMemFileBuffer("/vsimem/foo.shp", , 
true); //<-- returns NULL


For some reason VSIGetMemFileBuffer always fails, returning a nullptr. 
This is basically identical to a similar problem I had, described 
here: https://stackoverflow.com/questions/73644157/get-raw-buffer-for-in-memory-dataset-in-gdal-c-api 
 which 
I never really found a solution to. I have similar issues with other 
VSI functions, e.g. calling GDALOpenEx on a path created by 
VSIFileFromMemBuffer also fails.


Any idea as to what the problem is here?

Alternatively, maybe there's a better way to 'emplace' datasets inside 
some other file?


Thanks,
Soren Saville Scott

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Number of fields when creating layer?

2023-10-02 Thread Even Rouault via gdal-dev



Le 02/10/2023 à 17:23, Abel Pau a écrit :


Thanks Even,

that it’s not so bad. I can “wait” until ICreateFeature and if it’s 
the first feature create the number of fields it has.


But, then, what happens with other fields that can appear after in 
next features? I might ignore them, but... it’s not a good thing.


Just look at existing drivers, like the JML one I pointed. They will 
error out when attempting to call CreateField() once a feature has been 
written, and they will stop advertizing the OLCCreateField capability as 
well.


Can they appear later or not? I haven’t understood it at complete.

Calling CreateField() at any time is a possibility of the OGR API. That 
doesn't mean that all drivers have to support it. Most writers will 
generally create all needed fields before writing features, if they can, 
to be compatible with as many output drivers as possible.



--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Number of fields when creating layer?

2023-10-02 Thread Even Rouault via gdal-dev


Abel Pau,

no you can't know the number of fields at the moment of creating the 
layer. A number of drivers only accept the CreateField() method to be 
called until the first call to ICreateFeature(). Cf the 
"bFeaturesWritten" variable in ogr/ogrsf_frmts/jml/ogrjmlwriterlayer.cpp


Typically ogr2ogr will never call CreateField() after ICreateFeature()

Even

Le 02/10/2023 à 15:54, Abel Pau via gdal-dev a écrit :


Hi everybody,

do anyone know if a driver can know the number of fields (and it’s 
names, etc) of a layer BEFORE starting fetch features? So in the 
moment of create the layer.


I mean, when a driver is creating the own layer it could probably be 
interested in creating all fields before starting and not in the 
middle of translation. It could be caused because the output format 
requires that in terms of efficiency.


For instance, first feature have 3 fields, second feature have four, 
etc... Until the last feature driver is not sure of all fields are 
definitive.


And every time a feature is added the driver have to verify in which 
destination field corresponds to the new one.


So, I accepted that it happens with the number of features (Even can 
remember my concerns that time) and I am wondering if I have to accept 
that for the number of fields.


Thanks a lot.

*Abel Pau Garcia*

*GIS developer*



https://www.creaf.cat/sites/default/files/creaf-signature.png

*a@creaf.uab.cat* 

*Let's chat on Teams!* 



*Tel. +34 934814277*



https://www.creaf.cat/sites/default/files/so-en-signature.png

https://www.creaf.cat/sites/default/files/twitter-icon-signature.png 
https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png 
https://www.creaf.cat/sites/default/files/youtube-icon-signature.png 
https://www.creaf.cat/sites/default/files/instagram-icon-signature.png 



*www.creaf.cat* *| **http://blog.creaf.cat* 





https://www.creaf.cat/sites/default/files/uab_logo_signatura.png

CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona)


Before printing this electronic message, think about the environment.

http://www.creaf.uab.cat/_signatura/line.gif


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GRIB2 file extent not matchi GRIB2 metadata (grib_dump)

2023-09-28 Thread Even Rouault via gdal-dev


Hi Louis-Philippe,

yes this is a matter of conventions. GRIB indeed uses a center-of-pixel 
registration whereas GDAL uses a top-left corner of pixel ones. So GDAL 
adds a half-pixel shift to expose GRIB georeferencing using its 
convention, which is a feature. What is annoying is that it leads to 
latitudes > 90 for such datasets, but there isn't much that can be done 
in the driver itself.


Except doing post-processing to subset the file to crop the top and 
bottom line (gdal_translate -srcwin 0 1 2400 1199), which will result in 
longitudes in [-90 + 0.15, 90 - 0.15].   That said that file is a bit 
inconsistent. It is quite expected that it uses an odd number of rows 
(1201) to encode a clean (90 - -90) / (1201 - 1) = 0.15 vertical 
resolution. But for the horizontal dimension, it uses an even number of 
columns (2400), which leads to a compute resolution of (360 - 0) / (2400 
- 1) = 0.15006252605252188 resolution.


Or if you're OK to deal with a slight alteration in the georeferencing 
(maximum relative error should be 0.5 / 1200 = 0.04 % at the equator), 
gdal_translate -a_ullr 0 90 360 -90


Another approach would be to crop using non-integer coordinates (to 
remove the top and bottom half-rows) + apply a non-nearest resampling like


gdal_translate -srcwin 0 0.5 2400 1200 -r cubic in.grib2 out.tif

You should get an extent of latitudes in [-90,90].

Even

Le 28/09/2023 à 21:31, Rousseau Lambert, Louis-Philippe (ECCC) via 
gdal-dev a écrit :


Hi,

I am facing an issue with GRIB2 data, and the extent reported by a 
gdal_info command. Here is a sample data to test: 
https://drive.google.com/file/d/1URvQs2qHXgRkq3YfC7ZR8VGYK3SKXB59/view?usp=drive_link


Here is what I am testing with:

  * Ubuntu 20.04.6 LTS
  * GDAL 3.5.2, released 2022/09/02
  * PROJ Rel. 8.2.0, November 1st, 2021

So, using grib_dump, I see the longitude of the file should be from 0 
to 360 degress:


/grib_dump out.grib2 | grep shapeOfTheEarth -A 12/

/  shapeOfTheEarth = 6;/

/  Ni = 2400;/

/  Nj = 1201;/

/  iScansNegatively = 0;/

/  jScansPositively = 1;/

/  jPointsAreConsecutive = 0;/

/  alternativeRowScanning = 0;/

/  latitudeOfFirstGridPointInDegrees = -90;/

/*longitudeOfFirstGridPointInDegrees = 0;*/

/  latitudeOfLastGridPointInDegrees = 90;/

/*longitudeOfLastGridPointInDegrees = 360;*/

/  iDirectionIncrementInDegrees = 0.15;/

/  jDirectionIncrementInDegrees = 0.15;/

But using gdal_info I see the extent as:

/gdal_info -proj4 out.grib2 | grep PROJ -A 9 /

/PROJ.4 string is:/

/'+proj=longlat +R=6371229 +no_defs'/

/Origin = (-0.075031263026261,90.0750003)/

/Pixel Size = (0.150062526052522,-0.150)/

/Corner Coordinates:/

/Upper Left  (  -0.0750313,  90.075) (  0d 4'30.11"W, 90d 4'30.00"N)/

/Lower Left  (  -0.0750313, -90.075) (  0d 4'30.11"W, 90d 4'30.00"S)/

/Upper Right ( 360.075,  90.075) (360d 4'30.11"E, 90d 4'30.00"N)/

/Lower Right ( 360.075, -90.075) (360d 4'30.11"E, 90d 4'30.00"S)/

/Center  ( 180.000,   0.000) (180d 0' 0.00"E,  0d 0' 0.01"N)/

I would expect the gdal_info extent to be also from 0 to 360 degrees 
(2400 pixels * 0.15 = 360). From reading various source, I see the 
issue is potentially that the GRIB2 files define the extent using the 
center of the pixel while gdal assumes it is the top left corner.


My question: Is there a way to change this behavior in gdal, or this 
expected, or a bug ?


Thanks

LP


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev



Le 28/09/2023 à 20:17, Scott a écrit :

Thanks for digging into that Even!

Can I create my new .fgb in sections?

If I limit the number of source rows with -sql, doing that multiple 
times with -update, will it still build the entire R-tree when writing 
to the destination?
That would actually be a bad idea. Appending in FlatGeobuf is not a 
native capability. It is emulated by storing into RAM the new features, 
and then writing at dataset closing the full .fgb file from the original 
one + the new features. You would actually need a lot more RAM than 
doing a single ogr2ogr run.


I'm looking for a way to get the desired results.


Rent a bigger VM instance... or contribute a RAM-friendly implementation 
of the RTree building :-)


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev

ok, that now makes sense. Writing a .fgb files comes into those 
exceptions where RAM consumption might be important, as it involves 
building a packed Hilbert R-Tree in memory. With the current 
implementation, you need at least the number of features times some 
constant amount of RAM, at least to store the list of each feature 
bounding box + their offset in a temporary file. From what I can see 
this constant is at least 40 bytes. So in your particular case this 
requires at least 145459485 * 40 = 5.5 GB of RAM. And probably (not 
totally sure) twice that to store this initial list and the tree itself. 
I guess the implementation could be made smarter and use on-disk 
temporary memory, but that would likely involve serious implementation 
complications. I let Björn comment more on this if he follows this 
discussion.


I've submitted a doc enhancement to mention this requirement: 
https://github.com/OSGeo/gdal/pull/8490


Le 28/09/2023 à 19:17, Scott a écrit :
USA.fgb is 36 GB. I've renamed it from its original source which can 
be found here:

https://beta.source.coop/vida/google-microsoft-open-buildings

ogr2ogr -sql "select area_in_meters from bfp_USA" -nln footprints 
footprints.fgb ~/Downloads/USA.fgb


GDAL 3.7.1
OS Debian Buster

Output from ogrinfo -ro -al USA.fgb

Layer name: bfp_USA
Geometry: Unknown (any)
Feature Count: 145459485
Extent: (-160.221701, 17.677691) - (-64.583428, 71.360579)
Layer SRS WKT:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
    ELLIPSOID["WGS 84",6378137,298.257223563,
    LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
    ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
    AXIS["geodetic latitude (Lat)",north,
    ORDER[1],
    ANGLEUNIT["degree",0.0174532925199433]],
    AXIS["geodetic longitude (Lon)",east,
    ORDER[2],
    ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
    SCOPE["unknown"],
    AREA["World"],
    BBOX[-90,-180,90,180]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
boundary_id: Integer64 (0.0)
bf_source: String (0.0)
confidence: Real (0.0)
area_in_meters: Real (0.0)
OGRFeature(bfp_USA):0
  boundary_id (Integer64) = 116
  bf_source (String) = google
  confidence (Real) = 0.906
  area_in_meters (Real) = 187.4652
  POLYGON ((-64.6399621676723 17.7225504518464,-64.6400377660957 
17.722583049763,-64.6400238635835 17.7226126625647,-64.6400901719124 
17.7226412545727,-64.640104074415
 17.722611641767,-64.6401239848718 17.7226202271066,-64.6401528522526 
17.7225587385527,-64.6400955687758 17.7225340380511,-64.640105121 
17.7225136746756,-64.640040
1136221 17.7224856402151,-64.640030553504 
17.7225060035881,-64.6399910351014 17.7224889633119,-64.6399621676723 
17.7225504518464))


OGRFeature(bfp_USA):1
  boundary_id (Integer64) = 116
  bf_source (String) = microsoft
  area_in_meters (Real) = 51.0777955237376
  POLYGON ((-64.6398677811851 17.7219759840792,-64.6397939789141 
17.7219853127982,-64.6398020235506 17.7220430591893,-64.6398758258215 
17.7220337304732,-64.63986778118

51 17.7219759840792))

OGRFeature(bfp_USA):2
  boundary_id (Integer64) = 116
  bf_source (String) = google
  confidence (Real) = 0.8323
  area_in_meters (Real) = 178.5448
  POLYGON ((-64.6397672401299 17.7220665249078,-64.6397654280552 
17.722041016034,-64.6395789582891 17.7220531822569,-64.6395832735872 
17.7221139302758,-64.639696737462
3 17.7221065273415,-64.639698399651 17.7221299263498,-64.6398064310524 
17.7221228777942,-64.6398022655579 17.7220642396531,-64.6397672401299 
17.7220665249078))



On 9/28/23 10:03, Even Rouault wrote:


Le 28/09/2023 à 18:47, Scott via gdal-dev a écrit :


I should have been more specific.

One particular machine has 8GB of memory. When I try to do the most 
simple ogr2ogr command on large files, the host runs out of memory 
(vmstat shows this) and ogr2ogr terminates with 'Killed', nothing more.


The data formats I have experienced this with are .fgb, .parquet and 
.gpkg. The data files are 10's of GB.


As input ? as output? Which operating system ? Which GDAL version ? 
The output of "ogrinfo -al -so the_input" might also be helpful. An 
exact ogr2ogr command line invocation that triggers the issue would 
certainly be useful.  In general, most GDAL drivers and ogr2ogr 
itself operate in streaming mode with low RAM requirements, but there 
might be exceptions (some configurations of GeoJSON file may require 
full ingestion on reading for example).  I'm also aware of issues 
with RAM fragmentation due to how some memory allocators work, but 
they seem to be restricted to multithreaded uses 
(https://gdal.org/user/multithreading.html#ram-fragmentation-and-multi-threading), 
which current ogr2ogr shouldn't trigger


Even



Thanks for the responses!

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev



Le 28/09/2023 à 18:47, Scott via gdal-dev a écrit :


I should have been more specific.

One particular machine has 8GB of memory. When I try to do the most 
simple ogr2ogr command on large files, the host runs out of memory 
(vmstat shows this) and ogr2ogr terminates with 'Killed', nothing more.


The data formats I have experienced this with are .fgb, .parquet and 
.gpkg. The data files are 10's of GB.


As input ? as output? Which operating system ? Which GDAL version ? The 
output of "ogrinfo -al -so the_input" might also be helpful. An exact 
ogr2ogr command line invocation that triggers the issue would certainly 
be useful.  In general, most GDAL drivers and ogr2ogr itself operate in 
streaming mode with low RAM requirements, but there might be exceptions 
(some configurations of GeoJSON file may require full ingestion on 
reading for example).  I'm also aware of issues with RAM fragmentation 
due to how some memory allocators work, but they seem to be restricted 
to multithreaded uses 
(https://gdal.org/user/multithreading.html#ram-fragmentation-and-multi-threading), 
which current ogr2ogr shouldn't trigger


Even



Thanks for the responses!
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL Interpolation Help

2023-09-27 Thread Even Rouault via gdal-dev


Bill,

here you upsample from a 3x3 to a 5x5 array. Consequently each target 
pixel averages 5/3 x 5/3 contributing source pixels , not 2 x 2.  This 
way the total sum of the values of the target array is preserved 
compared to to the sum of the values of the source array, if you 
multiply the sum by 3/5 * 3/5.


To get what you expect you should work with even-sized arrays

Even

Le 27/09/2023 à 19:44, Bill Myers via gdal-dev a écrit :

Hi,
I am new to working with GDAL. We are trying to spatially interpolate 
Numerical Weather Prediction (NWP) model data to an arbitrary 
location, i.e. lat/lon. We have been using gdal_translate but have 
been getting unexpected results.
We have used the resampling method called "average". In this toy 
example, we are trying to resample to a grid with twice the density of 
the input data set. As can be seen in the image in the link below, the 
values at the points that match the input points agree with the input 
values. That's good. However, the interpolated values do not match our 
expectations. We would expect the interpolated value at the midpoint 
(directly between two input points) to be the average of those two 
input points. As can be seen, this is not the case.
Can someone please help us understand what we might be doing 
incorrectly? Also, does GDAL have the functionality that we are 
looking for?

Thanks for your help.
-b
https://pasteboard.co/uumLD4JtNmp3.png

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

< 1 2 3 4 5 6 7 8 9 10 >

301 - 400 of 7550 matches

Mail list logo