Re: [gdal-dev] slow translate with OVERVIEW_LEVEL=NONE when no overviews exist

2023-11-24 Thread Even Rouault via gdal-dev
Hi Michael, Le 25/11/2023 à 00:44, Michael Sumner via gdal-dev a écrit : When I translate this GeoTIFF to 10% original size, it's very very slow if OVERVIEW_LEVEL=NONE is set. Fixed per https://github.com/OSGeo/gdal/pull/8819 Even -- http://www.spatialys.com My software is free, but my time

Re: [gdal-dev] GDAL 3.8.1 RC1 is available, and motion to approve it

2023-11-24 Thread Even Rouault via gdal-dev
Le 24/11/2023 à 14:34, Sebastiaan Couwenberg via gdal-dev a écrit : On 11/24/23 13:22, Hernán De Angelis via gdal-dev wrote: May be this is of help: I have seen the case of the .py utilities not been compiled and installed a couple of times when I mistakenly had two Python versions present

Re: [gdal-dev] Problems compiling in Mac

2023-11-24 Thread Even Rouault via gdal-dev
Hi Carsten, should be fixed per https://github.com/OSGeo/gdal/pull/8810 Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org

Re: [gdal-dev] GDAL 3.8.1 RC1 is available, and motion to approve it

2023-11-24 Thread Even Rouault via gdal-dev
Bas, The Python utilities are no longer installed with the .py extension, this broke their use in QGIS when we removed this in the Debian package some time ago. Don't know if that's still the case. Either way, this doesn't seem like an appropriate change for a .1 patch release. Are you sure

[gdal-dev] "RFC 98: Build requirements for GDAL 3.9" available for review

2023-11-24 Thread Even Rouault via gdal-dev
Hi, Please find "RFC 98: Build requirements for GDAL 3.9" in https://github.com/OSGeo/gdal/pull/8802 for review & comments Summary: The document updates RFC68 with the new build requirements for GDAL 3.9: * C++ >= 17 * CMake >= 3.16 * PROJ >= 6.3.1 The minimum version for the following

[gdal-dev] GDAL 3.8.1 RC2 is available (was Re: GDAL 3.8.1 RC1 is available, and motion to approve it)

2023-11-24 Thread Even Rouault via gdal-dev
Withdrawing previous motion and: Motion: adopt 3.8.1 RC2 as final 3.8.1 release Starting with my +1, Even Le 24/11/2023 à 06:34, Sebastiaan Couwenberg via gdal-dev a écrit : On 11/23/23 22:56, Even Rouault via gdal-dev wrote:  From a packaging point of view, the following change might have impacts

[gdal-dev] GDAL 3.8.1 RC1 is available, and motion to approve it

2023-11-23 Thread Even Rouault via gdal-dev
Hi, I have prepared a GDAL/OGR 3.8.1 release candidate. Pick up an archive among the following ones (by ascending size):   https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc1.tar.xz   https://download.osgeo.org/gdal/3.8.1/gdal-3.8.1rc1.tar.gz  

Re: [gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-23 Thread Even Rouault via gdal-dev
Hi, However, I doubt that these organisations use the SOSI driver from GDAL/OGR to produce these files. They can't: the driver is read-only Discussing that topic with other developers today during GDAL maintainer meeting, we could proceed with how we did in the past with drivers we

Re: [gdal-dev] Problems compiling in Mac

2023-11-23 Thread Even Rouault via gdal-dev
I thought that it was enough disabling GDAL_USE_EXTERNAL_LIBS (we are not enabling explicitly arrow or kml). Am I wrong? He might have to remove CMakeCache.txt because it could contain the GDAL_USE_LIBKML=ON declaration that was set before turning off

Re: [gdal-dev] Problems compiling in Mac

2023-11-23 Thread Even Rouault via gdal-dev
Le 23/11/2023 à 14:53, Javier Jimenez Shaw via gdal-dev a écrit : Hi A colleague of mine is compiling in Mac. I am aware of the problems with arrow and libkml: https://gdal.org/development/building_from_source.html#building-on-macos But we are compiling with GDAL_USE_EXTERNAL_LIBS=OFF

[gdal-dev] Motion: adopt RFC 97: OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn "sealing"

2023-11-22 Thread Even Rouault via gdal-dev
Hi, Motion: adopt RFC 97: OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn "sealing" https://github.com/OSGeo/gdal/pull/8734 Pre-rendered view: https://github.com/rouault/gdal/blob/rfc97_text/doc/source/development/rfc/rfc97_feature_and_fielddefn_sealing.rst Starting with my +1, Even

[gdal-dev] Advanced 3.8.1 release

2023-11-21 Thread Even Rouault via gdal-dev
Hi, yesterday was a very prolific day where several regressions of 3.8.0 were reported: - https://github.com/OSGeo/gdal/issues/8755: Python: Filtered parquet/arrow layers sometimes produce invalid Map arrays - https://github.com/OSGeo/gdal/issues/8757: Conversion from GPKG using ogr2ogr

Re: [gdal-dev] COPY statement failed with PG_USE_COPY=YES

2023-11-21 Thread Even Rouault via gdal-dev
Matteo, From a quick test, I can't reproduce. Please provide a minimum shapefile that reproduces the issue $ cat test.csv id,dt 1,"/00/00" $ cat test.csvt Integer,Date $ ogr2ogr date_null.shp test.csv Warning 1: Invalid value type found in record 1 for field dt. This warning will no

Re: [gdal-dev] Resampling a raster with gdalwarp

2023-11-19 Thread Even Rouault via gdal-dev
Le 19/11/2023 à 03:08, Giovanni Anconitano a écrit : Thank you very much. Just to be sure, gdalwarp takes the absolute value of the second tr value independently from the units of the pixel size. It can be either meters or degrees depending on the coordinate reference system. Is that

Re: [gdal-dev] Resampling a raster with gdalwarp

2023-11-18 Thread Even Rouault via gdal-dev
Giovanni, gdalwarp takes the absolute value of the second tr value, so -tr 10 -10 is understood as -tr 10 10 It is expected that gdalinfo reports a negative value for the pixel size factor for the vertical axis, as in GeoTIFF when the image is north-up, the first line of the image is

[gdal-dev] Patches to fix build errors against libxml2 2.12

2023-11-18 Thread Even Rouault via gdal-dev
Hi, Just to notify (mostly packagers) that libxml2 2.12 has been released 2 days ago, and GDAL needs the following 2 patches to build against it: - https://github.com/OSGeo/gdal/commit/cbed9fc91dffba30d0f9a6a06a412a04d9cd36fa -

Re: [gdal-dev] GeoPDF and neatline

2023-11-17 Thread Even Rouault via gdal-dev
Hi, _My question :_ Is there a way to gdal_translate / gdalwarp the PDF file without EXPLICITLY extracting the neatline ? No. Your above method is the nominal one. I tried to use the "*-oo NEATLINE=something*" option : gdal_translate -of GTiff -oo NEATLINE="POLYGON ((467455.095191925

Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

2023-11-17 Thread Even Rouault via gdal-dev
Hi, Motion passed with +1 from PSC members KurtS, HowardB, JukkaR, JavierJS and me Even Le 15/11/2023 à 10:51, Even Rouault via gdal-dev a écrit : Hi, Motion: adopt RFC 96: Deferred C++ plugin loading https://github.com/OSGeo/gdal/pull/8648 Pre-rendered view: https://github.com/rouault

[gdal-dev] RFC97 OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn "sealing" available for review

2023-11-16 Thread Even Rouault via gdal-dev
Hi, Please review RFC 97: OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn "sealing" at https://github.com/OSGeo/gdal/pull/8734 Summary: This RFC aims at avoiding common misuse of the setter methods of the OGRFeatureDefn, OGRFieldDefn and OGRGeomFieldDefn classes. Indeed, the setter

Re: [gdal-dev] Converting parquet to geopackage

2023-11-16 Thread Even Rouault via gdal-dev
Michael, this error comes from libarrow-cpp. Several potential causes: - one of the parquet file is corrupted - one of the parquet files is valid but uses "something" that libarrow-cpp can't understand. I mention this because we have seen an interoperability issues between files generated

Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

2023-11-16 Thread Even Rouault via gdal-dev
Hi Sean, I think this makes great sense for the project. I don't yet understand what it means for an enterprise like Rasterio's PyPI wheels. I'd say it probably changes nothing. The RFC just postpones the time where the plugins are loaded, but the fact that they are dlopen()'ed (early or

Re: [gdal-dev] Export image binary data from GDB dataset

2023-11-15 Thread Even Rouault via gdal-dev
Andrea, looking with an hexadecimal editor, one sees the strings "Fotografia di Photo Editor", "MSPhotoEd.3", so it seems to be Microsoft Photo Editor format. According to https://stackoverflow.com/questions/43902205/microsoft-photo-editor-conversion-file-signatures , it might be a OLE based

[gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading

2023-11-15 Thread Even Rouault via gdal-dev
Hi, Motion: adopt RFC 96: Deferred C++ plugin loading https://github.com/OSGeo/gdal/pull/8648 Pre-rendered view: https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst Starting with my +1, Even -- http://www.spatialys.com My software

Re: [gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-14 Thread Even Rouault via gdal-dev
ub4=10604_sub5=EmailSignature__Static_> On Tue, Nov 14, 2023 at 8:41 PM, Sebastiaan Couwenberg via gdal-dev wrote: On 11/14/23 20:05, Even Rouault via gdal-dev wrote: > Le 14/11/2023 à 19:48, Sebastiaan Couwenberg via gdal-dev a écrit : >> On 11/14/23 19:16, E

Re: [gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-14 Thread Even Rouault via gdal-dev
Le 14/11/2023 à 19:48, Sebastiaan Couwenberg via gdal-dev a écrit : On 11/14/23 19:16, Even Rouault via gdal-dev wrote: ==> Are there users still using the existing OGR SOSI driver ? We don't have metrics for the OGR driver, but libfyba0 still has 513 users in popcon.  ht

[gdal-dev] Is the SOSI Norwegian driver still useful?

2023-11-14 Thread Even Rouault via gdal-dev
Hi, During the OSGeo code sprint, it was mentioned to me that the OGR SOSI driver (https://gdal.org/drivers/vector/sosi.html) is probably no longer relevant. Kartverket, the Norwegian mapping agency, which is the entity delivering datasets in the SOSI standard and which contributed the

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-13 Thread Even Rouault via gdal-dev
le in your installation. You may install it with with 'conda install -c conda-forge libgdal-arrow-parquet' I'll submit soon the RFC to vote if there are no further remarks. Even Le 06/11/2023 à 14:50, Even Rouault via gdal-dev a écrit : Hi, I've revised a bit the RFC (https://github.com/OSGeo

[gdal-dev] GDAL 3.8.0 is released

2023-11-13 Thread Even Rouault via gdal-dev
Hi, On behalf of the GDAL/OGR development team and community, I am pleased to announce the release of GDAL/OGR 3.8.0. GDAL/OGR is a C++ geospatial data access library for raster and vector file formats, databases and web services. It includes bindings for several languages, and a variety of

Re: [gdal-dev] Motion: adopt GDAL 3.8.0RC2 as 3.8.0 release

2023-11-13 Thread Even Rouault via gdal-dev
I declare this motion passed with +1 from PSC members JukkaR, JavierJS, HowardB and me Even Le 09/11/2023 à 11:12, Even Rouault via gdal-dev a écrit : Retracting the motion to adopt 3.8.0RC1 and updating it to: Motion: Adopt GDAL 3.8.0RC2 as 3.8.0 release Starting with my +1 Even Le 08

[gdal-dev] Motion: adopt GDAL 3.8.0RC2 as 3.8.0 release

2023-11-09 Thread Even Rouault via gdal-dev
Retracting the motion to adopt 3.8.0RC1 and updating it to: Motion: Adopt GDAL 3.8.0RC2 as 3.8.0 release Starting with my +1 Even Le 08/11/2023 à 09:39, Even Rouault via gdal-dev a écrit : Hi, Motion: Adopt GDAL 3.8.0RC1 as 3.8.0 release Starting with my +1 Even -- http

[gdal-dev] GDAL 3.8.0 rc2 is available

2023-11-09 Thread Even Rouault via gdal-dev
, Even Rouault via gdal-dev a écrit : Hi, I have prepared a GDAL/OGR 3.8.0 release candidate. NEWS at:   https://github.com/OSGeo/gdal/blob/v3.8.0RC1/NEWS.md (small change in it since the initial version I mentioned last week: there is a new optional dependency: libaec, needed for some GRIB

[gdal-dev] Motion: Adopt GDAL 3.8.0RC1 as 3.8.0 release

2023-11-08 Thread Even Rouault via gdal-dev
Hi, Motion: Adopt GDAL 3.8.0RC1 as 3.8.0 release Starting with my +1 Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org

Re: [gdal-dev] tps - gdalwarp vs ogr2ogr

2023-11-07 Thread Even Rouault via gdal-dev
Hi, I've ticketed your issue as https://github.com/OSGeo/gdal/issues/8672 . What you could try to reduce the execution time is to enable multithreading by adding -wo NUM_THREADS=ALL_CPUS to your gdalwarp command line And yes the previous implementation of the raster TPS transformation was

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-06 Thread Even Rouault via gdal-dev
of omissions or inconsistencies between the proxy and actual drivers. Even Le 02/11/2023 à 12:59, Even Rouault via gdal-dev a écrit : Hi, I'm seeking for feedback and review on a new RFC (RFC 96: Deferred in-tree C++ plugin loading), detailed in https://github.com/OSGeo/gdal/pull/8648, whose

[gdal-dev] GDAL 3.8.0 rc1 is available

2023-11-06 Thread Even Rouault via gdal-dev
Hi, I have prepared a GDAL/OGR 3.8.0 release candidate. NEWS at:   https://github.com/OSGeo/gdal/blob/v3.8.0RC1/NEWS.md (small change in it since the initial version I mentioned last week: there is a new optional dependency: libaec, needed for some GRIB files) Pick up an archive among the

Re: [gdal-dev] Segfault with ogr2ogr and Parquet

2023-11-05 Thread Even Rouault via gdal-dev
Scott, I don't reproduce anymore on master, and I strongly suspect this is the same issue as https://lists.osgeo.org/pipermail/gdal-dev/2023-November/057856.html which I fixed post-beta1 Even Le 05/11/2023 à 19:57, Scott via gdal-dev a écrit : ogr2ogr tst.gpkg

Re: [gdal-dev] Advice on `GDALCreateAndReprojectImage` and destination no-data value.

2023-11-03 Thread Even Rouault via gdal-dev
Simeon,     Band 1 Block=210x39 Type=Byte, ColorInterp=Gray The only way I've been able to get the behavior I need is to open the written dataset from the filesystem and set the no-data value explicitly in the band, which is inefficient and awkward. You don't have much alternative than

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-11-03 Thread Even Rouault via gdal-dev
I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers. Sorry no updated NEWS.md file yet ==> now available at https://github.com/OSGeo/gdal/blob/master/NEWS.md -- http://www.spatialys.com My software is free, but my time generally not.

Re: [gdal-dev] Question: CPL minizip affected by CVE-2023-45853?

2023-11-03 Thread Even Rouault via gdal-dev
Hi James, thanks for the notice. GDAL copy has diverged a bit, but I've just managed to apply the upstream fix per https://github.com/OSGeo/gdal/pull/8658 Even Le 03/11/2023 à 16:17, James Addison via gdal-dev a écrit : Hi folks, I've arrived at the gdal mailing list after reading the

[gdal-dev] GDAL 3.7.3 is released

2023-11-03 Thread Even Rouault via gdal-dev
Hi, On behalf of the GDAL/OGR development team, I am pleased to announce the release of the GDAL/OGR 3.7.3 bug fix version. Consult the release notes for the list of issues addressed:     https://github.com/OSGeo/gdal/blob/v3.7.3/NEWS.md The sources are available at:    

Re: [gdal-dev] Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release

2023-11-03 Thread Even Rouault via gdal-dev
Motion passed with +1 from PSC members JukkaR, JavierJS, SeanG and me Le 01/11/2023 à 12:13, Even Rouault via gdal-dev a écrit : Hi, Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release Starting with my +1 Even -- http://www.spatialys.com My software is free, but my time generally

Re: [gdal-dev] Show isBigTIFF in image metadata

2023-11-02 Thread Even Rouault via gdal-dev
Jukka, Does it feel reasonable? I know that overviews may be standard TIFFs while the main image is BigTIFF but maybe the information from the header would be enough. Or have I missed some existing tool? I thought that tiffinfo at least would report TIFF/BigTIFF but it doesn’t. tiffdump

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-02 Thread Even Rouault via gdal-dev
Howard, I would like to see some language that describes the expectations for version compatibility of plugins (ie, what happens with a plugin built against x.0 is used against a main library of y.0). This RFC actually doesn't change anything regarding this. See the mention of the

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-11-02 Thread Even Rouault via gdal-dev
Hi Sean, Rasterio's CI picked up a change to the AAIGrid driver in 3.8. The 3.7 version driver used to have whitespace before a row and no whitespace after. It looks like this has flipped in 3.8. Is it intentional? Yes: https://github.com/OSGeo/gdal/pull/8362 Even --

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-02 Thread Even Rouault via gdal-dev
Hi Rob, This looks great from my perspective. Are there any downsides? - A bit of additional coding complexity for driver development, but not that much - As mentioned in the backwards compatibility paragraph, for people doing multi-step builds to build first libgdal and then plugins,

[gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

2023-11-02 Thread Even Rouault via gdal-dev
Hi, I'm seeking for feedback and review on a new RFC (RFC 96: Deferred in-tree C++ plugin loading), detailed in https://github.com/OSGeo/gdal/pull/8648, whose summary is: This RFC adds a mechanism to defer the loading of in-tree C++ plugin drivers to the point where their executable code is

Re: [gdal-dev] WFS: How to use feature ordering?

2023-11-02 Thread Even Rouault via gdal-dev
Craig, remove the leading end-of-line and spaces in your ExecuteSQL l = ds.ExecuteSQL( """SELECT * FROM "open-data-platform:v_s_parcel_proposed" ORDER BY "parcel_pfi" ASC """ ) Also fixed perhttps://github.com/OSGeo/gdal/commit/f6c7d95e2c66ba1f62f6ff17e31af37f7f8f6bc8 Even

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-11-01 Thread Even Rouault via gdal-dev
of Engineers *From: *gdal-dev on behalf of Even Rouault via gdal-dev *Reply-To: *Even Rouault *Date: *Tuesday, October 31, 2023 at 1:14 PM *To: *Rahkonen Jukka , "gdal-dev@lists.osgeo.org" *Subject: *Re: [gdal-dev] GDAL 3.8.0beta1 available for testing Le 31/10/2023 à 17:36, Rahk

[gdal-dev] Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release

2023-11-01 Thread Even Rouault via gdal-dev
Hi, Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release Starting with my +1 Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-10-31 Thread Even Rouault via gdal-dev
The ghcr.io/osgeo/gdal:ubuntu-small-latest, ghcr.io/osgeo/gdal:ubuntu-full-latest, ghcr.io/osgeo/gdal:alpine-normal-latest Docker images have been refreshed with 3.8.0beta1 (ghcr.io/osgeo/gdal:alpine-small-latest still building at time of writing). All -latest images are now at

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

2023-10-31 Thread Even Rouault via gdal-dev
Le 31/10/2023 à 17:36, Rahkonen Jukka a écrit : Hi, I made a simple test with ogr2ogr and geopackage to geopackage on Windows. With GDAL 3.8.0dev-3e4dc710a2 (no arrow, old R-Tree) the timing was 36 minutes, with GDAL 3.8.0dev-6bbd2c080a the same conversion took 21 minutes. The gpkg file is

Re: [gdal-dev] Motion: OSGeo Community Sprint Financial Support

2023-10-31 Thread Even Rouault via gdal-dev
+1 Even Le 31/10/2023 à 17:59, Howard Butler via gdal-dev a écrit : Dear PSC, Sorry for the short notice, but I would like to motion that the GDAL Sponsorship Program financially support GDAL PSC members who wish to attend the OSGeo Community Sprint in Vienna [1] next week. Support level is

[gdal-dev] GDAL 3.8.0beta1 available for testing

2023-10-31 Thread Even Rouault via gdal-dev
Hi, I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers. Sorry no updated NEWS.md file yet, but I'd in particular be interesting by testing of ogr2ogr workflows, since they have underwent significant changes in the underlying implementation: - when the source layer is a

Re: [gdal-dev] GDAL 3.7.3 RC1 available

2023-10-30 Thread Even Rouault via gdal-dev
Le 30/10/2023 à 18:11, Scott via gdal-dev a écrit : Looking at the release notes did #8262 make it in? https://github.com/OSGeo/gdal/pull/8262 no, this is 3.8.0 only material -- http://www.spatialys.com My software is free, but my time generally not.

[gdal-dev] GDAL 3.7.3 RC1 available

2023-10-30 Thread Even Rouault via gdal-dev
Hi, I have prepared a GDAL/OGR 3.7.3 release candidate. Pick up an archive among the following ones (by ascending size):   https://download.osgeo.org/gdal/3.7.3/gdal-3.7.3rc1.tar.xz   https://download.osgeo.org/gdal/3.7.3/gdal-3.7.3rc1.tar.gz  

Re: [gdal-dev] Tag for units in GeoTIFF

2023-10-26 Thread Even Rouault via gdal-dev
G:4269+5703 $ gdalinfo out.tif | grep Unit   Unit Type: foot $ gdal_translate byte.tif out.tif -a_srs EPSG:4269+8228 $ gdalinfo out.tif | grep Unit    Unit Type: foot -David On Oct 26, 2023, at 4:45 AM, Javier Jimenez Shaw via gdal-dev wrote: Thanks! On Thu, 26 Oct 2023 at 12:27, Even Roua

Re: [gdal-dev] Tag for units in GeoTIFF

2023-10-26 Thread Even Rouault via gdal-dev
Javier, Le 26/10/2023 à 11:59, Javier Jimenez Shaw via gdal-dev a écrit : Hi Using a GeoTIFF with a single band and float values can be very useful to show any distribution over the terrain. One typical example is DSM (digital surface model), where the value is the elevation on every point.

Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression

2023-10-16 Thread Even Rouault via gdal-dev
ng threads could see their file handle being seek() by another thread.  In short: multithreading is hard. On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev wrote: Hi, For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff compression+decompression, in particular within the

[gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression

2023-10-16 Thread Even Rouault via gdal-dev
Hi, For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff compression+decompression, in particular within the same file, as for example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow can lead to deadlocks (process stalled forever) and/or data corruption (sometimes without

Re: [gdal-dev] Reading a shapefile containing multi-parts Polyline from OGR (C++ API)

2023-10-16 Thread Even Rouault via gdal-dev
Le 16/10/2023 à 15:15, Javier Jimenez Shaw via gdal-dev a écrit : Do you mean a MultiLineString? This piece of code is working for me (I hope without any bug). if (wkbFlatten(poGeometry->getGeometryType()) == wkbMultiLineString) multiLineStringGeometry(poGeometry); ...

Re: [gdal-dev] Performance regression testing/benchmarking for CI

2023-10-15 Thread Even Rouault via gdal-dev
(some are caused by external changes)  - in 3-5 years we'll probably have to trim down the JSON or switch to a different storage Laurentiu On Tue, Oct 10, 2023, at 21:08, Even Rouault via gdal-dev wrote: > Hi, > > I'm experimenting with adding performanc

Re: [gdal-dev] including higher-level library in GPX creator string

2023-10-13 Thread Even Rouault via gdal-dev
Cf https://github.com/OSGeo/gdal/pull/8558: "GPX: add a CREATOR dataset creation option" It will be of course up to the application / application user to set it. I've considered a global GDALSetDefaultCreatorApplication() (could be similar to CPLHTTPSetDefaultUserAgent():

Re: [gdal-dev] Cannot choose netCDF driver instead of HDF5 for nc files

2023-10-12 Thread Even Rouault via gdal-dev
Works for me with GDAL 3.6, 3.7 and master on Ubuntu 20.04 $ gdalinfo /vsicurl/https://nbstds.met.no/thredds/fileServer/NBS/S2B/2022/07/03/S2B_MSIL1C_20220703T112119_N0400_R037_T32VLN_20220703T121203.nc Driver: netCDF/Network Common Data Format [...] /vsicurl/ functionality for the netCDF

[gdal-dev] Performance regression testing/benchmarking for CI

2023-10-10 Thread Even Rouault via gdal-dev
Hi, I'm experimenting with adding performance regression testing in our CI. Currently our CI has quite extensive functional coverage, but totally lacks performance testing. Given that we use pytest, I've spotted pytest-benchmark (https://pytest-benchmark.readthedocs.io/en/latest/) as a

Re: [gdal-dev] OAPIF: how does the driver utilize schema?

2023-10-10 Thread Even Rouault via gdal-dev
Hi Jukka, The documentation in https://gdal.org/drivers/vector/oapif.html#layer-schema is wrong when it claims this: OGR needs a fixed schema per layer, but OGC API - Features Core doesn't impose fixed schema. So the driver will retrieve the first page of features (10 features) and

Re: [gdal-dev] format for topology?

2023-10-09 Thread Even Rouault via gdal-dev
Michael, you won't find much support for that in GDAL itself. You'd better have a look at Spatialite topology, PostGIS topology or GRASS topology. The process is basically: import your geometries as a topology,  do reprojection or other work in the topology domain, and go back to the

Re: [gdal-dev] The relationship between overview levels and zoom levels?

2023-10-08 Thread Even Rouault via gdal-dev
James, It all depends on how software will deal with overviews. Some might explicitly have logic where it inspects which overviews are present and decide which one to use. If we are talking about a client like QGIS that uses the RasterIO() method on the full resolution band with a window of

Re: [gdal-dev] Questions regarding temporary database file (ogr2ogr with MVT driver)

2023-10-06 Thread Even Rouault via gdal-dev
You could possibly paralle Your text cut off at the interesting bit, could you elaborate on this @Even? ah I meant you could parallelize generation on several machines, each one with a different extent and/or zoom level -- http://www.spatialys.com My software is free, but my time generally

Re: [gdal-dev] Default page size of the OAPIF driver

2023-10-06 Thread Even Rouault via gdal-dev
Hi Jukka, yes it would make sense for the driver to use a larger page size. You can file an issue about that Actually we could reuse the same logic as the QGIS OAPIF provider that determines the page size from the limit.schema.maximum and limit.schema.default values of the /api response:  

Re: [gdal-dev] Questions regarding temporary database file (ogr2ogr with MVT driver)

2023-10-06 Thread Even Rouault via gdal-dev
Hi,, I’m creating vector tiles from a PostGIS database using ogr2ogr with the MVT driver. The region is quite large and I’m creating tiles for levels 0-15, so the process takes quite some time (hours). The tiles are written directly to a S3 storage using the /vsis3/ virtual file system

Re: [gdal-dev] VSIGetMemFileBuffer Returns NULL When Copying Vector Dataset to Vsimem

2023-10-05 Thread Even Rouault via gdal-dev
w.r.t GetFileList() on shapefiles just created where one of the side car file wasn't listed Thanks for your help, Soren On Wed, 4 Oct 2023 at 21:09, Even Rouault wrote: Hi, size_t is an incorrect size for DataLength (if you run on 32 bit architecture). You should use vsi_l_offset

Re: [gdal-dev] VSIGetMemFileBuffer Returns NULL When Copying Vector Dataset to Vsimem

2023-10-04 Thread Even Rouault via gdal-dev
Hi, size_t is an incorrect size for DataLength (if you run on 32 bit architecture). You should use vsi_l_offset. Besides that the following standalone inspired from your code works fine for me: #include #include "gdal_priv.h" int main() {     GDALAllRegister();     GDALDataset* Dataset

Re: [gdal-dev] Number of fields when creating layer?

2023-10-02 Thread Even Rouault via gdal-dev
Le 02/10/2023 à 17:23, Abel Pau a écrit : Thanks Even, that it’s not so bad. I can “wait” until ICreateFeature and if it’s the first feature create the number of fields it has. But, then, what happens with other fields that can appear after in next features? I might ignore them, but...

Re: [gdal-dev] Number of fields when creating layer?

2023-10-02 Thread Even Rouault via gdal-dev
Abel Pau, no you can't know the number of fields at the moment of creating the layer. A number of drivers only accept the CreateField() method to be called until the first call to ICreateFeature(). Cf the "bFeaturesWritten" variable in ogr/ogrsf_frmts/jml/ogrjmlwriterlayer.cpp Typically

Re: [gdal-dev] GRIB2 file extent not matchi GRIB2 metadata (grib_dump)

2023-09-28 Thread Even Rouault via gdal-dev
Hi Louis-Philippe, yes this is a matter of conventions. GRIB indeed uses a center-of-pixel registration whereas GDAL uses a top-left corner of pixel ones. So GDAL adds a half-pixel shift to expose GRIB georeferencing using its convention, which is a feature. What is annoying is that it leads

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev
Le 28/09/2023 à 20:17, Scott a écrit : Thanks for digging into that Even! Can I create my new .fgb in sections? If I limit the number of source rows with -sql, doing that multiple times with -update, will it still build the entire R-tree when writing to the destination? That would actually

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev
17.7221065273415,-64.639698399651 17.7221299263498,-64.6398064310524 17.7221228777942,-64.6398022655579 17.7220642396531,-64.6397672401299 17.7220665249078)) On 9/28/23 10:03, Even Rouault wrote: Le 28/09/2023 à 18:47, Scott via gdal-dev a écrit : I should have been more specific. One partic

Re: [gdal-dev] Using ogr2ogr with limited memory

2023-09-28 Thread Even Rouault via gdal-dev
Le 28/09/2023 à 18:47, Scott via gdal-dev a écrit : I should have been more specific. One particular machine has 8GB of memory. When I try to do the most simple ogr2ogr command on large files, the host runs out of memory (vmstat shows this) and ogr2ogr terminates with 'Killed', nothing

Re: [gdal-dev] GDAL Interpolation Help

2023-09-27 Thread Even Rouault via gdal-dev
Bill, here you upsample from a 3x3 to a 5x5 array. Consequently each target pixel averages 5/3 x 5/3 contributing source pixels , not 2 x 2.  This way the total sum of the values of the target array is preserved compared to to the sum of the values of the source array, if you multiply the

Re: [gdal-dev] About installing EXPAT

2023-09-25 Thread Even Rouault via gdal-dev
*Why after doing an gdal_master_env and installing everything and expat it doesn’t work?* Once again, I appreciate your help :) *De:*Even Rouault *Enviado el:* dilluns, 25 de setembre de 2023 14:30 *Para:* Abel Pau ; gdal dev *Asunto:* Re: [gdal-dev] About installing EXPAT Abel, In Cmakefile.txt of

Re: [gdal-dev] About installing EXPAT

2023-09-25 Thread Even Rouault via gdal-dev
Abel, In Cmakefile.txt of the project I’ve configured this (below # Developer may want to specify some variable to find proper version.): set(EXPAT_INCLUDE_DIR "D:/GitHub-repository/libexpat/expat/lib") set(EXPAT_LIBRARY "D:/GitHub-repository/libexpat/expat/build/Debug/libexpatd.lib")

Re: [gdal-dev] layer.GetSpatialRef() fails on linux for shapefiles

2023-09-24 Thread Even Rouault via gdal-dev
Le 24/09/2023 à 23:22, Jonathan Moules via gdal-dev a écrit : > Also check if the environment isn't messed up regarding PROJ and the PROJ_LIB/PROJ_DATA environment variable Thanks Even; sorry, what does this line mean? I'm guessing you're referring to:

Re: [gdal-dev] resampling algs warp vs translate

2023-09-21 Thread Even Rouault via gdal-dev
That should be relatively straightforward to adapt GDALResampleChunk_AverageOrRMS_T to do sum. Reuse the average code path, but just don't divide the sum by dfTotalWeight at lines 1418, 1473, 1688, 1700 and 1712. And error out if applied to a band with a color table (cf line 4361) For other

Re: [gdal-dev] Call for 4.0 ideas

2023-09-21 Thread Even Rouault via gdal-dev
| But I have more troubles to understand why breaking code that use spatial_ref for decades without any real need Have you ever looked at the mess in the netcdf driver? Aligning with recommendations from standards, especially on the write side, and helping to reduce a bit code bloat (although

Re: [gdal-dev] GeoTIFF and concurrent block reads

2023-09-21 Thread Even Rouault via gdal-dev
Of course, but I'm asking if it's worth calling ReadBlock on multiple threads (if it always takes a lock, it's not, and I should use RasterIO instead). Not on the same dataset object, otherwise you'll get crashes as no lock is taken (*) I think these can also be implemented at VSI level by

Re: [gdal-dev] GeoTIFF and concurrent block reads

2023-09-21 Thread Even Rouault via gdal-dev
Laurentiu, GDAL 3.6 added support for multi-threaded reading using PRead, but I couldn't spot ReadBlock using the same code path. If you read one single block at a time, the multi-threaded optimization cannot kick in, since the elementary decoding unit is a block. You must call RasterIO()

Re: [gdal-dev] layer.GetSpatialRef() fails on linux for shapefiles

2023-09-21 Thread Even Rouault via gdal-dev
Run ogrinfo on one shapefile to see if some more interesting error message is displayed Also check if the environment isn't messed up regarding PROJ and the PROJ_LIB/PROJ_DATA environment variable Le 21/09/2023 à 14:34, Jonathan Moules via gdal-dev a écrit : Yeah, I'm afraid the error

Re: [gdal-dev] adding metadata -mo in a domain?

2023-09-21 Thread Even Rouault
which don't seem to exist. Is that just an outdated comment? yes, as far as I can see from history those functions have never existed. This comment dates back to the initial commit and they didn't even exist at that time. -- http://www.spatialys.com My software is free, but my time

Re: [gdal-dev] Call for 4.0 ideas

2023-09-20 Thread Even Rouault
Le 21/09/2023 à 01:12, Joaquim Manuel Freire Luís a écrit : Remove spatial_ref WKT export in netCDF driver (GDAL 4) #4712 Would this mean all codes that use spatial_ref would be broken yes, spatial_ref is AFAIK a GDAL specific thing that predates the introduction of crs_wkt in the

[gdal-dev] Call for 4.0 ideas

2023-09-20 Thread Even Rouault
Hi, following latest discussions on RFC 95, I've drafted https://github.com/OSGeo/gdal/issues/8440 with a few ideas. Please add yours into it. We should focus on listing breaking changes, rather than new features that can be developed without involving breakage. Even --

Re: [gdal-dev] Call for discussion/review of RFC95: Use standard C/C++ integer types

2023-09-20 Thread Even Rouault
4.0 simultaneously if 4.0 made GDAL's C API tangibly better, like dataset unification in 2.0 or the PROJ switch in 3.0. Do you have something in mind about a tangibly better improvement to the API that would make it worth a 4.0? Even On Tue, Sep 19, 2023 at 10:30 AM Even Rouault wrote: No ot

Re: [gdal-dev] Motion: add Javier Jiménez Shaw to GDAL PSC

2023-09-20 Thread Even Rouault
Motion passed with +1 from PSC members MateuszL, HowardB, JukkaR, KurtS, FrankW, DanielM and me. Congrats Javier! You can file a PR to add yourself to https://gdal.org/community/index.html#psc Le 16/09/2023 à 13:31, Even Rouault a écrit : Hi, I would like to nominate Javier Jiménez Shaw

Re: [gdal-dev] adding metadata -mo in a domain?

2023-09-20 Thread Even Rouault
I'll try that, or is it too much over-reach? Ultimately it's for vrt:// con so could do vrt://{dsn}?dmo=DOMAIN:META-TAG=VALUE Cheers, Mike On Wed, Sep 20, 2023 at 10:19 AM Even Rouault wrote: Not from cli. From the API, it is dataset.SetMetadataItem(key, value, domain) Le 20/09/20

Re: [gdal-dev] adding metadata -mo in a domain?

2023-09-19 Thread Even Rouault
Not from cli. From the API, it is dataset.SetMetadataItem(key, value, domain) Le 20/09/2023 à 02:14, Michael Sumner a écrit : When we add metadata with -mo can we somehow specify the DOMAIN it would be added to, or be instantiated with? gdal_translate gcore/data/float32.tif /tmp/add_md.vrt

Re: [gdal-dev] Call for discussion/review of RFC95: Use standard C/C++ integer types

2023-09-19 Thread Even Rouault
No other reaction ? Are people comfortable with bumping to 4.0 ? If so, no opinion on what we should slip in while we are it (thinking more about breaking changes than new features, which generally can be added afterwards) ? Le 16/09/2023 à 14:53, Even Rouault a écrit : About GDAL 4.0 vs

Re: [gdal-dev] Call for discussion/review of RFC95: Use standard C/C++ integer types

2023-09-16 Thread Even Rouault
About GDAL 4.0 vs 3.8, I'm fine with 4.0. But I do not know if "users" will expect a bigger change in functionalities for a mayor release update. Yes, there are a few other tickets flagged as appropriate for 4.0: https://github.com/OSGeo/gdal/milestone/33 All of them could probably be

[gdal-dev] Motion: add Javier Jiménez Shaw to GDAL PSC

2023-09-16 Thread Even Rouault
Hi, I would like to nominate Javier Jiménez Shaw for GDAL PSC membership. Javier has been involved with GDAL for quite a time now, as a responsive user & ticket reporter, and has occasionally contributed fixes. Javier is well anchored in our ecosystem, with deep knowledge of CRS topics and

Re: [gdal-dev] Formalizing GDAL "file name" syntax in an RFC?

2023-09-15 Thread Even Rouault
I agree that there is an untidy legacy. And it's probably not worth formalizing the marginal filenames and connection strings. How about an RFC and formal syntax only for hierarchical datasets then? It seems like this is the direction the industry is growing. Hiearchical gridded datasets are

[gdal-dev] Call for discussion/review of RFC95: Use standard C/C++ integer types

2023-09-15 Thread Even Rouault
Hi, I've submitted an initial version of RFC95: Use standard C/C++ integer types     https://github.com/OSGeo/gdal/pull/8399 It has important implications on backward compatibility of C and C++ API. The "questions to be answer" paragraph need to be answered. Even --

Re: [gdal-dev] Formalizing GDAL "file name" syntax in an RFC?

2023-09-14 Thread Even Rouault
Sean, It is far from obvious to me to find a universal pattern that would fit existing and future use cases. If we would find a universal syntax, there would be the problem of deciding what to do with code that doesn't match it: support only new way (breaking external code),  or supporting

<    1   2   3   4   5   6   7   8   9   10   >