Re: [gdal-dev] Considering drivers removal ?

[Jed O. Kaplan]

I for one am using the GMT vector driver for GDAL on a regular basis, e.g., for 
integration between GRASS GIS and GMT for cartography. I am grateful to the 
developers for their continued support for this driver.

Thanks, Jed

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [EXTERNAL] Re: Considering drivers removal ?

[Hare, Trent M via gdal-dev]

Speaking on behave of folks who support archives, I highly recommend keeping 
old drivers, unless as Frank stated, they cause an issue (code or license). We 
are often faced with dealing with code rot, but in the same vein, we will also 
find ourselves dealing with more and more outdated file formats. I would much 
rather keep an old harmless driver around than trying to install crusty 
binaries (or compile old code). To clean-up the growing list of drivers, 
perhaps the code and driver listings can be grouped together (only shown with 
some flag .e.g. --show_crusty_drivers :-).

my 2 cents,
Trent




From: gdal-dev  on behalf of Frank Warmerdam 

Sent: Monday, January 11, 2021 3:54 PM
To: Even Rouault 
Cc: gdal dev 
Subject: [EXTERNAL] Re: [gdal-dev] Considering drivers removal ?


 Even,

My opinion is that old drivers which do not depend on external 
libraries/services and that we have tests for should be retained until they 
cause painful problems.  I would be supportive of dropping drivers for which 
there is no apparent interest, and that are not buildable and testable due to 
dependence on external libraries and services.  We can always reintroduce them 
if someone comes forward and wants them and is willing to help support them.

LAN, NTv1 and SDTS Raster are examples (IMHO) of relatively low value drivers 
that should be retained because they are buildable, testable and not 
problematic.

Best regards,
Frank


On Sun, Jan 10, 2021 at 6:02 PM Even Rouault 
mailto:even.roua...@spatialys.com>> wrote:
Hi,

It's not spring yet, but I'm in a mood lately of axing useless things, and we
probably have tons of candidate for that in GDAL, especially in drivers.
I was going to just axe the DB2 driver
(https://github.com/OSGeo/gdal/pull/3366)
 but the issue is more general.

Any idea how we can know what is used and what isn't ? A "call-home"
functionality where we would track driver usage would only be acceptable if
people enable it and have network connectivity, so we won't probably get lots
of feedback. Having a spreadsheet with the driver list and asking people to
fill it would probably also receive little feedback. So the idea I had was to
do something like the following in the Open() method of a candidate for
removal:

GDALDataset* FooDriver::Open(  )
{
   if( !Identify(poOpenInfo) )
  return nullptr;

   if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
   {
   CPLError(CE_Failure, CPLE_AppDefined,
"Driver FOO is considered for removal in GDAL 3.5. You are invited "
"to convert any dataset in that format to another more common one ."
"If you need this driver in future GDAL versions, create a ticket at "

"https://github.com/OSGeo/gdal
 (look first for an existing one first) to "
"explain how critical it is for you (but the GDAL project may still "
"remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
"configuration option / environment variable to YES");
   return nullptr;
}
...
}

That is, when we detect a file to be handled by the driver, emit the above
error message and do not open the dataset, unless the user defines the
environment variable.
Similarly in the Create()/CreateCopy() methods.
If we ship this in 3.3, with a 3.5 milestone for removal, this would offer a
feedback period of one year / 2 feature versions.

Here's my own list of candidates for retirement (probably over-conservative).
Mostly based on gut feeling. None of them are particularly bad citizens, but I
have no indication that they are still used, which doesn't mean they aren't.

* Raster side:
BPG
DB2Raster
DOQ1
DOQ2
E00GRID
Epsilon
FujiBAS
GS7BG
GSAG
IDA
JDEM
JPEG2000 (Jasper): JP2OpenJPEG is a better replacement
JPEGLS
LAN
MFF
MG4Lidar ?
NDF
NTv1
SDTS Raster
SGI
XPM
ZMap

* Vector side:
AERONAVFAA
ESRI ArcObjects
ARCGEN
BNA
Cloudant
CouchDB
DB2
DODS
FMEObjects Gateway
Geomedia MDB
GMT ASCII Vectors
GTM
HTF
INGRES
MongoDB (the old one, superseded by MongoDBv3)
OpenAIR
REC
SDTS
SUA
SVG
TIGER
WALK


Anything you'd add / remove ?

What is not obvious is what would be the criterion for keeping a driver: 1,
10, 100 users asking for the driver to be kept ?
If a GDAL developer contributing 

[gdal-dev] netCDF chunk fetch failed

[Kurt Schwehr]

Hi all,

In a build of gdal 3.2.0, I'm seeing this error: "netCDF chunk fetch
failed" for 2 out of 5 subdatasets.  Is there an easy way to handle this so
the bad data is returned as the nodata value?  And also, any idea what is
actually wrong with the netcdf files?

I am in contact with the NOAA folks who made this data.

Thanks!
-kurt



gsutil cp
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2017/263/21/OR_ABI-L2-FDCF-M3_G16_s20172632115407_e20172632126173_c20172632126283.nc
.
gsutil cp
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2018/123/12/OR_ABI-L2-FDCF-M3_G16_s20181231215382_e20181231226149_c20181231226258.nc
.
gsutil cp
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2018/324/18/OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc
.
gsutil cp
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2020/247/16/OR_ABI-L2-FDCF-M6_G16_s20202471650186_e20202471659494_c20202471700318.nc
.


for subdataset in Area Temp Mask Power DQF; do
echo $subdataset
(gdalinfo -stats
NETCDF:OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc:$subdataset
| grep 'Band 1');
done

Area
Warning 1: netCDFDataset::valid_range: min > max:
  min: 0.00
  max: -6.00

ERROR 1: netCDF chunk fetch failed: #-101 (NetCDF: HDF error)
ERROR 1:
NETCDF:OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc:Area,
band 1: IReadBlock failed at X offset 19, Y offset 10: netCDF chunk fetch
failed: #-101 (NetCDF: HDF error)
Band 1 Block=226x226 Type=Int16, ColorInterp=Undefined
Temp
Warning 1: netCDFDataset::valid_range: min > max:
  min: 0.00
  max: -6.00
Band 1 Block=226x226 Type=Int16, ColorInterp=Undefined
Mask
ERROR 1: netCDF chunk fetch failed: #-101 (NetCDF: HDF error)
ERROR 1:
NETCDF:OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc:Mask,
band 1: IReadBlock failed at X offset 4, Y offset 11: netCDF chunk fetch
failed: #-101 (NetCDF: HDF error)
Band 1 Block=226x226 Type=Int16, ColorInterp=Undefined
Power
Band 1 Block=226x226 Type=Float32, ColorInterp=Undefined
DQF
Band 1 Block=226x226 Type=Byte, ColorInterp=Undefined
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Joaquim Manuel Freire Luís]

> - GMT is an active project and some GMT developers appear on this list as 
> well. Maybe some of them happend to read this and say if GMT ASCII vectors 
> are still important for GMT. Or otherwise I can ask from the GMT forum.


Right.

The GMT raster driver is from the times GNT used a 1-d array to represent 2-D 
grids in netCDF. That was abandoned some ~15 years ago for COARS compliant nc 
grids and is a good candidate for deprecation.

The " GMT ASCII vectors" format (written under contract with NIWA) is still 
very much used (under the hood) by the GMT library and should be kept alive 
until we finish the integration with the GDAL vector side.

Joaquim




-Original Message-
From: gdal-dev  On Behalf Of jratike80
Sent: Monday, January 11, 2021 6:56 PM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Considering drivers removal ?

Hi,

The joy of being a Windows user is that it is so easy to use old GDAL versions 
if the binaries still happen to be on some dusty backup disk. Even the FWTools 
including GDAL 1.7.0 from 2010 seemed to work fine and include quite a many 
later removed drivers.

A few comments about the driver list:

- There are indeed questions about SVG in gis.stackexchange every now and then.
- I used LAN a lot when FWTools was young and ERDAS wrote bad GeoTIFFs.
Things are probably changed since that. 
- GMT is an active project and some GMT developers appear on this list as well. 
Maybe some of them happend to read this and say if GMT ASCII vectors are still 
important for GMT. Or otherwise I can ask from the GMT forum.
- GPS Track Maker, as far as I know, is quite popular in Brazil. However, when 
I used the software I just used GPX format for data transfer. Are there any 
Brazilian GDAL user here to comment?

When it comes to Windows binaries, there is a very valuable archive in 
https://gisinternals.com/archive.php. It would be pity if it would get lost 
some day.

-Jukka Rahkonen-



Even Rouault-2 wrote
> Hi,
> 
> It's not spring yet, but I'm in a mood lately of axing useless things, 
> and we probably have tons of candidate for that in GDAL, especially in 
> drivers.
> I was going to just axe the DB2 driver
> (https://github.com/OSGeo/gdal/pull/3366) but the issue is more general.
> 
> Any idea how we can know what is used and what isn't ? A "call-home" 
> functionality where we would track driver usage would only be 
> acceptable if people enable it and have network connectivity, so we 
> won't probably get lots of feedback. Having a spreadsheet with the 
> driver list and asking people to fill it would probably also receive 
> little feedback. So the idea I had was to do something like the 
> following in the Open() method of a candidate for
> removal:
> 
> GDALDataset* FooDriver::Open(  )
> {
>if( !Identify(poOpenInfo) )
>   return nullptr;
> 
>if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
>{
>CPLError(CE_Failure, CPLE_AppDefined,
> "Driver FOO is considered for removal in GDAL 3.5. You are invited "
> "to convert any dataset in that format to another more common one ."
> "If you need this driver in future GDAL versions, create a ticket at "
> "https://github.com/OSGeo/gdal (look first for an existing one 
> first) to "
> "explain how critical it is for you (but the GDAL project may still "
> "remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
> "configuration option / environment variable to YES");
>return nullptr;
> }
> ...
> }
> 
> That is, when we detect a file to be handled by the driver, emit the 
> above error message and do not open the dataset, unless the user 
> defines the environment variable.
> Similarly in the Create()/CreateCopy() methods.
> If we ship this in 3.3, with a 3.5 milestone for removal, this would 
> offer a feedback period of one year / 2 feature versions.
> 
> Here's my own list of candidates for retirement (probably 
> over-conservative).
> Mostly based on gut feeling. None of them are particularly bad 
> citizens, but I have no indication that they are still used, which 
> doesn't mean they aren't.
> 
> * Raster side:
> BPG
> DB2Raster
> DOQ1
> DOQ2
> E00GRID
> Epsilon
> FujiBAS
> GS7BG
> GSAG
> IDA
> JDEM
> JPEG2000 (Jasper): JP2OpenJPEG is a better replacement JPEGLS LAN MFF 
> MG4Lidar ?
> NDF
> NTv1
> SDTS Raster
> SGI
> XPM
> ZMap
> 
> * Vector side:
> AERONAVFAA
> ESRI ArcObjects
> ARCGEN
> BNA
> Cloudant
> CouchDB
> DB2
> DODS
> FMEObjects Gateway
> Geomedia MDB
> GMT ASCII Vectors
> GTM
> HTF
> INGRES
> MongoDB (the old one, superseded by MongoDBv3) OpenAIR REC SDTS SUA 
> SVG TIGER WALK
> 
> 
> Anything you'd add / remove ?
> 
> What is not obvious is what would be the criterion for keeping a driver:
> 1,
> 10, 100 users asking for the driver to be kept ?
> If a GDAL developer contributing to the overall good of the project 
> needs the preservation of a driver to be able to justify its continued 
> involvement, 

Re: [gdal-dev] Considering drivers removal ?

[Frank Warmerdam]

Even,

My opinion is that old drivers which do not depend on external
libraries/services and that we have tests for should be retained until they
cause painful problems.  I would be supportive of dropping drivers for
which there is no apparent interest, and that are not buildable and
testable due to dependence on external libraries and services.  We can
always reintroduce them if someone comes forward and wants them and is
willing to help support them.

LAN, NTv1 and SDTS Raster are examples (IMHO) of relatively low value
drivers that should be retained because they are buildable, testable and
not problematic.

Best regards,
Frank


On Sun, Jan 10, 2021 at 6:02 PM Even Rouault 
wrote:

> Hi,
>
> It's not spring yet, but I'm in a mood lately of axing useless things, and
> we
> probably have tons of candidate for that in GDAL, especially in drivers.
> I was going to just axe the DB2 driver
> (https://github.com/OSGeo/gdal/pull/3366) but the issue is more general.
>
> Any idea how we can know what is used and what isn't ? A "call-home"
> functionality where we would track driver usage would only be acceptable
> if
> people enable it and have network connectivity, so we won't probably get
> lots
> of feedback. Having a spreadsheet with the driver list and asking people
> to
> fill it would probably also receive little feedback. So the idea I had was
> to
> do something like the following in the Open() method of a candidate for
> removal:
>
> GDALDataset* FooDriver::Open(  )
> {
>if( !Identify(poOpenInfo) )
>   return nullptr;
>
>if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
>{
>CPLError(CE_Failure, CPLE_AppDefined,
> "Driver FOO is considered for removal in GDAL 3.5. You are invited "
> "to convert any dataset in that format to another more common one ."
> "If you need this driver in future GDAL versions, create a ticket at "
> "https://github.com/OSGeo/gdal (look first for an existing one first)
> to "
> "explain how critical it is for you (but the GDAL project may still "
> "remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
> "configuration option / environment variable to YES");
>return nullptr;
> }
> ...
> }
>
> That is, when we detect a file to be handled by the driver, emit the above
> error message and do not open the dataset, unless the user defines the
> environment variable.
> Similarly in the Create()/CreateCopy() methods.
> If we ship this in 3.3, with a 3.5 milestone for removal, this would offer
> a
> feedback period of one year / 2 feature versions.
>
> Here's my own list of candidates for retirement (probably
> over-conservative).
> Mostly based on gut feeling. None of them are particularly bad citizens,
> but I
> have no indication that they are still used, which doesn't mean they
> aren't.
>
> * Raster side:
> BPG
> DB2Raster
> DOQ1
> DOQ2
> E00GRID
> Epsilon
> FujiBAS
> GS7BG
> GSAG
> IDA
> JDEM
> JPEG2000 (Jasper): JP2OpenJPEG is a better replacement
> JPEGLS
> LAN
> MFF
> MG4Lidar ?
> NDF
> NTv1
> SDTS Raster
> SGI
> XPM
> ZMap
>
> * Vector side:
> AERONAVFAA
> ESRI ArcObjects
> ARCGEN
> BNA
> Cloudant
> CouchDB
> DB2
> DODS
> FMEObjects Gateway
> Geomedia MDB
> GMT ASCII Vectors
> GTM
> HTF
> INGRES
> MongoDB (the old one, superseded by MongoDBv3)
> OpenAIR
> REC
> SDTS
> SUA
> SVG
> TIGER
> WALK
>
>
> Anything you'd add / remove ?
>
> What is not obvious is what would be the criterion for keeping a driver:
> 1,
> 10, 100 users asking for the driver to be kept ?
> If a GDAL developer contributing to the overall good of the project needs
> the
> preservation of a driver to be able to justify its continued involvement,
> I'd
> tend to think it to be enough to keep it.
>
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam,
warmer...@pobox.com
light and sound - activate the windows | +1 650-701-7823
and watch the world go round - Rush| Geospatial Software Developer
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Sean Gillies via gdal-dev]

Hi Even,

On Mon, Jan 11, 2021 at 7:44 AM Even Rouault 
wrote:

> Hi,
>
> trying to answer the different points raised up to now:
>
> - SVG: let's keep it as it is used. This is exactly the feedback I'm
> seeking
> for. I had developed this as a toy, crazy me, I won't do it anymore. No
> idea
> anyone was using it actually.


> - making those driver plugins. This would require significant work, and
> the
> purpose is to save work. Our current build infrastructure is not ready for
> easy plugification. And announcing they will be unmaintained is probably
> not
> efficient to know in advance the impact of the planned breakage. People
> don't
> read documentation. The only way to force people to react is to break them
> in
> some way.
>
> - regarding Python, I'm not sure what the question is exactly. If it was
> how
> to still enable the drivers candidate for retirement to work, then it is
> just
> gdal.SetConfigOption('GDAL_ENABLE_DRIVER_FOO', 'YES')
>

Making users explicitly turn these formats back on feels good to me. I
don't see a downside to it. We might want to consider one round of warnings
before setting this option's default to NO? Doing so would take care of the
operators of deployed applications who *do* pay attention to warnings.

Would it make any sense to retire read and write of formats on a different
schedule? The fewer SDTS files written in the next 6-12 months, the less
impact there will be when we remove the driver.

When the change is made to GDAL, I might bikeshed about the config option
name a bit.


> - once we have decided which drivers should be retired, I don't find
> moving
> them to some cemetery repository very useful. Because that would lack part
> of
> the build scripts. What would be more useful is to add a paragraph in the
> documentation that drivers FOO, BAR, BAZ were retired in GDAL 3.5. That
> way
> people know they have to download a GDAL tarball or checkout a git tag
> before
> that release, or download a past OSGeo-Live VM, or Conda package, etc...
> And
> they will get (hopefully) functional code. Much better than the cemetery
> approach.
>
> - regarding schedule:
>* GDAL 3.3, May 2021: version with drivers semi-retired
>* GDAL 3.4, November 2021: still with drivers semi-retired
>* GDAL 3.5, May 2022: retired drivers are gone
>  So we won't get much feedback from Ubuntu LTS april 2022, as at that
> date,
> the drivers will have to be retired for the 3.5 release.
>
> - Where are the costs in keeping these drivers ?
>* monetary: there is one, but I'm unable to quantify it
>* environmental: burn CPU cycles each time someone does a GDAL build
>* psychological: prevent developers from doing modernization in GDAL
> internals. When you know you have to change 250 drivers, you think twice
> before doing the change, and generally you conclude it is not worth it, or
> fallback to hacks to limit the amount of code change. We should probably
> trim
> down the list to 20 raster and vector drivers to have a real effect
> regarding
> that. For a next time :-)
>
> Even
>

Would we disable default compilation of these drivers in 3.3 as well? I'd
be in favor of that.

Thanks for writing that third bullet point. I've thought twice about
changes for exactly that reason.

Formats come and go. We can't expect the GDAL project and its maintainers
(mostly you!) to be the curators forever of all data.

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] vector layer with Z coordinate - report extent in 3D

[Ricardo Filipe Soares Garcia da]

Hi list

Is there some brief way to get the Z extent of a vector layer? I mean
something analogous to the Layer.GetExtent() method, but which would
retrieve the extent of the Z coordinate?

OTherwise, it seems like I need to iterate over all features of the layer
and keep the min/max Z values. Shall this be a good tactic for retrieving
the Z extent of a vector?


Thanks in advance

-- 
___ ___ __
Ricardo Garcia Silva
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [Non-DoD Source] Re: Considering drivers removal ?

[TUELLER, SHAYNE R CIV USAF AFMC 519 SWES/MXDEC via gdal-dev]

I love FWTools. Still use it quite often.

I wish it was still actively maintained.

Shayne

From: gdal-dev  on behalf of jratike80 

Sent: Monday, January 11, 2021 11:55 AM
To: gdal-dev@lists.osgeo.org 
Subject: [Non-DoD Source] Re: [gdal-dev] Considering drivers removal ?

Hi,

The joy of being a Windows user is that it is so easy to use old GDAL
versions if the binaries still happen to be on some dusty backup disk. Even
the FWTools including GDAL 1.7.0 from 2010 seemed to work fine and include
quite a many later removed drivers.

A few comments about the driver list:

- There are indeed questions about SVG in gis.stackexchange every now and
then.
- I used LAN a lot when FWTools was young and ERDAS wrote bad GeoTIFFs.
Things are probably changed since that.
- GMT is an active project and some GMT developers appear on this list as
well. Maybe some of them happend to read this and say if GMT ASCII vectors
are still important for GMT. Or otherwise I can ask from the GMT forum.
- GPS Track Maker, as far as I know, is quite popular in Brazil. However,
when I used the software I just used GPX format for data transfer. Are there
any Brazilian GDAL user here to comment?

When it comes to Windows binaries, there is a very valuable archive in
https://gisinternals.com/archive.php. It would be pity if it would get lost
some day.

-Jukka Rahkonen-



Even Rouault-2 wrote
> Hi,
>
> It's not spring yet, but I'm in a mood lately of axing useless things, and
> we
> probably have tons of candidate for that in GDAL, especially in drivers.
> I was going to just axe the DB2 driver
> (https://github.com/OSGeo/gdal/pull/3366) but the issue is more general.
>
> Any idea how we can know what is used and what isn't ? A "call-home"
> functionality where we would track driver usage would only be acceptable
> if
> people enable it and have network connectivity, so we won't probably get
> lots
> of feedback. Having a spreadsheet with the driver list and asking people
> to
> fill it would probably also receive little feedback. So the idea I had was
> to
> do something like the following in the Open() method of a candidate for
> removal:
>
> GDALDataset* FooDriver::Open(  )
> {
>if( !Identify(poOpenInfo) )
>   return nullptr;
>
>if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
>{
>CPLError(CE_Failure, CPLE_AppDefined,
> "Driver FOO is considered for removal in GDAL 3.5. You are invited "
> "to convert any dataset in that format to another more common one ."
> "If you need this driver in future GDAL versions, create a ticket at "
> "https://github.com/OSGeo/gdal (look first for an existing one first)
> to "
> "explain how critical it is for you (but the GDAL project may still "
> "remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
> "configuration option / environment variable to YES");
>return nullptr;
> }
> ...
> }
>
> That is, when we detect a file to be handled by the driver, emit the above
> error message and do not open the dataset, unless the user defines the
> environment variable.
> Similarly in the Create()/CreateCopy() methods.
> If we ship this in 3.3, with a 3.5 milestone for removal, this would offer
> a
> feedback period of one year / 2 feature versions.
>
> Here's my own list of candidates for retirement (probably
> over-conservative).
> Mostly based on gut feeling. None of them are particularly bad citizens,
> but I
> have no indication that they are still used, which doesn't mean they
> aren't.
>
> * Raster side:
> BPG
> DB2Raster
> DOQ1
> DOQ2
> E00GRID
> Epsilon
> FujiBAS
> GS7BG
> GSAG
> IDA
> JDEM
> JPEG2000 (Jasper): JP2OpenJPEG is a better replacement
> JPEGLS
> LAN
> MFF
> MG4Lidar ?
> NDF
> NTv1
> SDTS Raster
> SGI
> XPM
> ZMap
>
> * Vector side:
> AERONAVFAA
> ESRI ArcObjects
> ARCGEN
> BNA
> Cloudant
> CouchDB
> DB2
> DODS
> FMEObjects Gateway
> Geomedia MDB
> GMT ASCII Vectors
> GTM
> HTF
> INGRES
> MongoDB (the old one, superseded by MongoDBv3)
> OpenAIR
> REC
> SDTS
> SUA
> SVG
> TIGER
> WALK
>
>
> Anything you'd add / remove ?
>
> What is not obvious is what would be the criterion for keeping a driver:
> 1,
> 10, 100 users asking for the driver to be kept ?
> If a GDAL developer contributing to the overall good of the project needs
> the
> preservation of a driver to be able to justify its continued involvement,
> I'd
> tend to think it to be enough to keep it.
>
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> ___
> gdal-dev mailing list

> gdal-dev@.osgeo

> https://lists.osgeo.org/mailman/listinfo/gdal-dev





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
___

[gdal-dev] Gdalwarp fails with Stereographic_North_Pole

[Rahkonen Jukka (MML)]

Hi,

Have a look at this question 
https://gis.stackexchange.com/questions/383825/difference-between-qgis-export-and-gdalwarp?

The original poster managed to get good output with gdalwarp by warping first 
into EPSG:9040 and then into EPSG:4326 but that feels like a workaround and not 
a solution. I could reproduce the issue and I wonder what QGIS is doing 
differently.

-Jukka Rahkonen-
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Unable to install GDAL on macOS with pip

[Chris Marsh]

Glad to hear it.
I'm not sure what Even et al's stance on using homebrew as an official
recommendation is. I am just a user who has gone down this route myself

On Mon, Jan 11, 2021 at 1:37 PM Marius Kreß  wrote:

> CAUTION: External to USask. Verify sender and use caution with links and
> attachments. Forward suspicious emails to phish...@usask.ca
>
> Thank you, this worked! Maybe this should be mentioned on the Download
> page? I think it is a bit incomplete in general: Nothing on macOS and the
> Windows section just mentions Conda and no alternatives (like
> gisinternals.com and OSGeo4W). Maybe it could be expanded?
>
> Marius
>
> Am 11.01.2021 um 19:18 schrieb Chris Marsh :
>
> Hi,
> You need to have the gdal binaries installed before pip install gdal will
> work. Specifically, pip install gdal installs the python bindings for gdal.
>
> In my opinion on mac, the easiest way is to install gdal via homebrew.
> Then the configuration utility gdal-info should be on the path and pip
> install GDAL should work
> https://pypi.org/project/GDAL/
>
> Going the homebrew route for gdal will ensure you have everything needed
> for gdal to work and avoids the complication of having to compile gdal
> yourself.
>
> Personally, I use gdal from base homebrew and then use this repackaging of
> the python gdal bindings. I've found it to be more reliable with venvs.
> https://github.com/nextgis/pygdal
>
> Both methods should work though,
>
> Cheers
> Chris
>
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Unable to install GDAL on macOS with pip

[Marius Kreß via gdal-dev]

Thank you, this worked! Maybe this should be mentioned on the Download page? I 
think it is a bit incomplete in general: Nothing on macOS and the Windows 
section just mentions Conda and no alternatives (like gisinternals.com and 
OSGeo4W). Maybe it could be expanded?

Marius

> Am 11.01.2021 um 19:18 schrieb Chris Marsh :
> 
> Hi,
> You need to have the gdal binaries installed before pip install gdal will 
> work. Specifically, pip install gdal installs the python bindings for gdal.
> 
> In my opinion on mac, the easiest way is to install gdal via homebrew. Then 
> the configuration utility gdal-info should be on the path and pip install 
> GDAL should work
> https://pypi.org/project/GDAL/ 
> 
> Going the homebrew route for gdal will ensure you have everything needed for 
> gdal to work and avoids the complication of having to compile gdal yourself. 
> 
> Personally, I use gdal from base homebrew and then use this repackaging of 
> the python gdal bindings. I've found it to be more reliable with venvs.
> https://github.com/nextgis/pygdal 
> 
> Both methods should work though,
> 
> Cheers
> Chris

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[jratike80]

Hi,

The joy of being a Windows user is that it is so easy to use old GDAL
versions if the binaries still happen to be on some dusty backup disk. Even
the FWTools including GDAL 1.7.0 from 2010 seemed to work fine and include
quite a many later removed drivers.

A few comments about the driver list:

- There are indeed questions about SVG in gis.stackexchange every now and
then.
- I used LAN a lot when FWTools was young and ERDAS wrote bad GeoTIFFs.
Things are probably changed since that. 
- GMT is an active project and some GMT developers appear on this list as
well. Maybe some of them happend to read this and say if GMT ASCII vectors
are still important for GMT. Or otherwise I can ask from the GMT forum.
- GPS Track Maker, as far as I know, is quite popular in Brazil. However,
when I used the software I just used GPX format for data transfer. Are there
any Brazilian GDAL user here to comment?

When it comes to Windows binaries, there is a very valuable archive in
https://gisinternals.com/archive.php. It would be pity if it would get lost
some day.

-Jukka Rahkonen-



Even Rouault-2 wrote
> Hi,
> 
> It's not spring yet, but I'm in a mood lately of axing useless things, and
> we 
> probably have tons of candidate for that in GDAL, especially in drivers.
> I was going to just axe the DB2 driver
> (https://github.com/OSGeo/gdal/pull/3366) but the issue is more general.
> 
> Any idea how we can know what is used and what isn't ? A "call-home" 
> functionality where we would track driver usage would only be acceptable
> if 
> people enable it and have network connectivity, so we won't probably get
> lots 
> of feedback. Having a spreadsheet with the driver list and asking people
> to 
> fill it would probably also receive little feedback. So the idea I had was
> to 
> do something like the following in the Open() method of a candidate for 
> removal:
> 
> GDALDataset* FooDriver::Open(  )
> {
>if( !Identify(poOpenInfo) )
>   return nullptr;
> 
>if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
>{
>CPLError(CE_Failure, CPLE_AppDefined,
> "Driver FOO is considered for removal in GDAL 3.5. You are invited "
> "to convert any dataset in that format to another more common one ."
> "If you need this driver in future GDAL versions, create a ticket at "
> "https://github.com/OSGeo/gdal (look first for an existing one first)
> to "
> "explain how critical it is for you (but the GDAL project may still "
> "remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
> "configuration option / environment variable to YES");
>return nullptr;
> }
> ...
> }
> 
> That is, when we detect a file to be handled by the driver, emit the above 
> error message and do not open the dataset, unless the user defines the 
> environment variable.
> Similarly in the Create()/CreateCopy() methods.
> If we ship this in 3.3, with a 3.5 milestone for removal, this would offer
> a 
> feedback period of one year / 2 feature versions.
> 
> Here's my own list of candidates for retirement (probably
> over-conservative). 
> Mostly based on gut feeling. None of them are particularly bad citizens,
> but I 
> have no indication that they are still used, which doesn't mean they
> aren't.
> 
> * Raster side:
> BPG
> DB2Raster
> DOQ1
> DOQ2
> E00GRID
> Epsilon
> FujiBAS
> GS7BG
> GSAG
> IDA
> JDEM
> JPEG2000 (Jasper): JP2OpenJPEG is a better replacement
> JPEGLS
> LAN
> MFF
> MG4Lidar ?
> NDF
> NTv1
> SDTS Raster
> SGI
> XPM
> ZMap
> 
> * Vector side:
> AERONAVFAA
> ESRI ArcObjects
> ARCGEN
> BNA
> Cloudant
> CouchDB
> DB2
> DODS
> FMEObjects Gateway
> Geomedia MDB
> GMT ASCII Vectors
> GTM
> HTF
> INGRES
> MongoDB (the old one, superseded by MongoDBv3)
> OpenAIR
> REC
> SDTS
> SUA
> SVG
> TIGER
> WALK
> 
> 
> Anything you'd add / remove ?
> 
> What is not obvious is what would be the criterion for keeping a driver:
> 1, 
> 10, 100 users asking for the driver to be kept ?
> If a GDAL developer contributing to the overall good of the project needs
> the 
> preservation of a driver to be able to justify its continued involvement,
> I'd 
> tend to think it to be enough to keep it.
> 
> 
> Even
> 
> -- 
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> ___
> gdal-dev mailing list

> gdal-dev@.osgeo

> https://lists.osgeo.org/mailman/listinfo/gdal-dev





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Unable to install GDAL on macOS with pip

[Chris Marsh]

Hi,
You need to have the gdal binaries installed before pip install gdal will
work. Specifically, pip install gdal installs the python bindings for gdal.

In my opinion on mac, the easiest way is to install gdal via homebrew. Then
the configuration utility gdal-info should be on the path and pip install
GDAL should work
https://pypi.org/project/GDAL/

Going the homebrew route for gdal will ensure you have everything needed
for gdal to work and avoids the complication of having to compile gdal
yourself.

Personally, I use gdal from base homebrew and then use this repackaging of
the python gdal bindings. I've found it to be more reliable with venvs.
https://github.com/nextgis/pygdal

Both methods should work though,

Cheers
Chris
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Unable to install GDAL on macOS with pip

[Marius Kreß via gdal-dev]

Dear GDAL developers and users,

I am unable to install GDAL on my Mac. I have set up Python with pyenv. When I 
enter pip install gdal I get a lot of exceptions that always end with 
FileNotFoundError: [Errno 2] No such file or directory: 
'../../apps/gdal-config‘ or  FileNotFoundError: [Errno 2] No such file or 
directory: 'gdal-config‘ The very last line says Could not find gdal-config. 
Make sure you have installed the GDAL native library and development headers. 
But that’s what I try to do, right?

When I download the package from http://download.osgeo.org/gdal/ and try to 
install it with pip install gdal-3.2.1.tar.gz I get the error message:

ERROR: Command errored out with exit status 1:
 command: /Users/marius/.pyenv/versions/3.9.0/bin/python3.9 -c 'import sys, 
setuptools, tokenize; sys.argv[0] = 
'"'"'/private/var/folders/99/kr62vcdn1nq7b376wln_0r6mgn/T/pip-req-build-hmpof14i/setup.py'"'"';
 
__file__='"'"'/private/var/folders/99/kr62vcdn1nq7b376wln_0r6mgn/T/pip-req-build-hmpof14i/setup.py'"'"';f=getattr(tokenize,
 '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', 
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info 
--egg-base 
/private/var/folders/99/kr62vcdn1nq7b376wln_0r6mgn/T/pip-pip-egg-info-3inooqvl
 cwd: 
/private/var/folders/99/kr62vcdn1nq7b376wln_0r6mgn/T/pip-req-build-hmpof14i/
Complete output (5 lines):
Traceback (most recent call last):
  File "", line 1, in 
  File "/Users/marius/.pyenv/versions/3.9.0/lib/python3.9/tokenize.py", 
line 392, in open
buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 
'/private/var/folders/99/kr62vcdn1nq7b376wln_0r6mgn/T/pip-req-build-hmpof14i/setup.py'

Do you have any idea what could be the problem? There are no installation 
instructions on the website, so I assumed it can be installed with pip just 
like any other package.

Best regards,

Marius___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Greg Troxel]

Sorry,  I didn't read the first message in it entirety before digging in
to the thread.

Reading over the proposed list of removals, nothing jumps out at me as
problematic.



signature.asc
Description: PGP signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[David Strip]

  
  
Bearing in mind that I use none of the drivers on Even's list, I
find his suggestion and reasoning compelling. I especially agree
with his comment that the only way to get anyone's attention is to
break their workflow, if only temporarily. The main risk here is
that a program that uses gdal (eg, QGIS) might hide this from it's
users by setting the options in code. Of course, when it truly
breaks then this program will have deal with unhappy users, so the
burden will be on them, not on gdal devs (we can only hope). 

As to the "cemetery" as Even calls it, this is in line with my
thinking before I saw Even's message. GIT maintains history, so
anyone wanting an old driver can always revert back to older
versions (at the cost of losing new capabilities.) I would consider
that instead of just noting in this in the docs somewhere, an
attempt to load a removed driver will result in a message that says
"This driver was removed in GIT update " to make it easier to
track down. A  variation on this that would require a little more
work is to replace each removed driver with a stub that prints an
appropriate failure message. In most cases, this would be the same
"this was removed" message, but in the rare case that someone else
picks up maintenance of the driver, the message could be something
like  "This driver is independently maintained at ". 
  

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] gdal_cachemax configuration

[Stephane Poissant]

Good day to all,

I am fairly new to goal / mapserver and I have to upgrade the servers to latest 
Linux available on Amazon. (CentOS 2 kinda)
I managed to get mapserver compiled and going properly. I’d like to configure 
the goal_cachemax to 512 but I cannot find
Where and how to apply this systemwide instead of per connection. 

[root@mapserver-0290 bin]# ./gdalinfo --version
GDAL 3.2.0, released 2020/10/26

[root@mapserver-0290 bin]# ./gdalinfo --config GDAL_CACHEMAX 512
Usage: gdalinfo [--help-general] [-json] [-mm] [-stats] [-hist] [-nogcp] [-nomd]
[-norat] [-noct] [-nofl] [-checksum] [-proj4]
[-listmdd] [-mdd domain|`all`] [-wkt_format WKT1|WKT2|...]*
[-sd subdataset] [-oo NAME=VALUE]* [-if format]* datasetname

Any suggestions? Unless you have a better idea of what could cause this error 
type?
For the records, it works fine on Amazon Linux 1...


I get errors in the apache log like the following:

(IP and URL were replaced for security reasons)
…
[Mon Jan 11 14:39:49.128142 2021] [cgi:error] [pid 2498] [client x.x.x.x:65072] 
AH01215: GDAL: GDALOpen(PG:host=floodsource2.tk.com port=5432 
dbname='floodsource' user='monkeysk' password=XX 
schema='public' table='flood_jba_cn_river_1500_2017_v2' mode='2', 
this=0x1d2bf80) succeeds as PostGISRaster.: /var/www/html/map/mapserv.cgi, 
referer: http://utility.something.com/
[Mon Jan 11 14:39:49.143728 2021] [cgi:error] [pid 2794] [client x.x.x.x:65076] 
AH01215: GDAL: GDALOpen(PG:host=floodsource2.tk.com port=5432 
dbname='floodsource' user='monkeysk' password=XX 
schema='public' table='flood_jba_cn_river_1500_2017_v2' mode='2', 
this=0x1273f80) succeeds as PostGISRaster.: /var/www/html/map/mapserv.cgi, 
referer: http://utility.something.com/
[Mon Jan 11 14:39:49.144051 2021] [cgi:error] [pid 2792] [client x.x.x.x:65074] 
AH01215: GDAL: GDALOpen(PG:host=floodsource2.tk.com port=5432 
dbname='floodsource' user='monkeysk' password=XX 
schema='public' table='flood_jba_cn_river_1500_2017_v2' mode='2', 
this=0x12baf80) succeeds as PostGISRaster.: /var/www/html/map/mapserv.cgi, 
referer: http://utility.something.com/
[Mon Jan 11 14:39:49.146908 2021] [cgi:error] [pid 2495] [client x.x.x.x:65077] 
AH01215: GDAL: GDALOpen(PG:host=floodsource2.tk.com port=5432 
dbname='floodsource' user='monkeysk' password=XX 
schema='public' table='flood_jba_cn_river_1500_2017_v2' mode='2', 
this=0x21abf80) succeeds as PostGISRaster.: /var/www/html/map/mapserv.cgi, 
referer: http://utility.something.com/
[Mon Jan 11 14:39:49.156075 2021] [cgi:error] [pid 2800] [client x.x.x.x:65073] 
AH01215: GDAL: GDALOpen(PG:host=floodsource2.tk.com port=5432 
dbname='floodsource' user='monkeysk' password=XX 
schema='public' table='flood_jba_cn_river_1500_2017_v2' mode='2', 
this=0x1debf80) succeeds as PostGISRaster.: /var/www/html/map/mapserv.cgi, 
referer: http://utility.something.com/
[Mon Jan 11 14:39:49.156355 2021] [cgi:error] [pid 2793] [client x.x.x.x:65075] 
AH01215: GDAL: GDALOpen(PG:host=floodsource2.tk.com port=5432 
dbname='floodsource' user='monkeysk' password=XX 
schema='public' table='flood_jba_cn_river_1500_2017_v2' mode='2', 
this=0x102af80) succeeds as PostGISRaster.: /var/www/html/map/mapserv.cgi, 
referer: http://utility.something.com/
[Mon Jan 11 14:39:49.382585 2021] [cgi:error] [pid 2498] [client x.x.x.x:65072] 
AH01215: GDAL: GDAL_CACHEMAX = 388 MB: /var/www/html/map/mapserv.cgi, referer: 
http://utility.something.com/
[Mon Jan 11 14:39:49.443576 2021] [cgi:error] [pid 2800] [client x.x.x.x:65073] 
AH01215: GDAL: GDAL_CACHEMAX = 388 MB: /var/www/html/map/mapserv.cgi, referer: 
http://utility.something.com/
[Mon Jan 11 14:39:49.477139 2021] [cgi:error] [pid 2495] [client x.x.x.x:65077] 
AH01215: GDAL: GDAL_CACHEMAX = 388 MB: /var/www/html/map/mapserv.cgi, referer: 
http://utility.something.com/
[Mon Jan 11 14:39:49.479542 2021] [cgi:error] [pid 2794] [client x.x.x.x:65076] 
AH01215: GDAL: GDAL_CACHEMAX = 388 MB: /var/www/html/map/mapserv.cgi, referer: 
http://utility.something.com/
[Mon Jan 11 14:39:49.505870 2021] [cgi:error] [pid 2793] [client x.x.x.x:65075] 
AH01215: GDAL: GDAL_CACHEMAX = 388 MB: /var/www/html/map/mapserv.cgi, referer: 
http://utility.something.com/
[Mon Jan 11 14:39:49.527354 2021] [cgi:error] [pid 2792] [client x.x.x.x:65074] 
AH01215: GDAL: GDAL_CACHEMAX = 388 MB: /var/www/html/map/mapserv.cgi, referer: 
http://utility.something.com/

Stéphane Poissant  
Portable: 514-793-3506
spoissan...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Even Rouault]

Hi,

trying to answer the different points raised up to now:

- SVG: let's keep it as it is used. This is exactly the feedback I'm seeking 
for. I had developed this as a toy, crazy me, I won't do it anymore. No idea 
anyone was using it actually.

- making those driver plugins. This would require significant work, and the 
purpose is to save work. Our current build infrastructure is not ready for 
easy plugification. And announcing they will be unmaintained is probably not 
efficient to know in advance the impact of the planned breakage. People don't 
read documentation. The only way to force people to react is to break them in 
some way.

- regarding Python, I'm not sure what the question is exactly. If it was how 
to still enable the drivers candidate for retirement to work, then it is just 
gdal.SetConfigOption('GDAL_ENABLE_DRIVER_FOO', 'YES')

- once we have decided which drivers should be retired, I don't find moving 
them to some cemetery repository very useful. Because that would lack part of 
the build scripts. What would be more useful is to add a paragraph in the 
documentation that drivers FOO, BAR, BAZ were retired in GDAL 3.5. That way 
people know they have to download a GDAL tarball or checkout a git tag before 
that release, or download a past OSGeo-Live VM, or Conda package, etc... And 
they will get (hopefully) functional code. Much better than the cemetery 
approach.

- regarding schedule:
   * GDAL 3.3, May 2021: version with drivers semi-retired
   * GDAL 3.4, November 2021: still with drivers semi-retired
   * GDAL 3.5, May 2022: retired drivers are gone
 So we won't get much feedback from Ubuntu LTS april 2022, as at that date, 
the drivers will have to be retired for the 3.5 release.

- Where are the costs in keeping these drivers ?
   * monetary: there is one, but I'm unable to quantify it
   * environmental: burn CPU cycles each time someone does a GDAL build
   * psychological: prevent developers from doing modernization in GDAL 
internals. When you know you have to change 250 drivers, you think twice 
before doing the change, and generally you conclude it is not worth it, or 
fallback to hacks to limit the amount of code change. We should probably trim 
down the list to 20 raster and vector drivers to have a real effect regarding 
that. For a next time :-)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Andrew C Aitchison]

On Mon, 11 Jan 2021, Even Rouault wrote:


It's not spring yet, but I'm in a mood lately of axing useless things, and
we probably have tons of candidate for that in GDAL, especially in drivers.
I was going to just axe the DB2 driver
(https://github.com/OSGeo/gdal/pull/3366) but the issue is more general.
   ...... 

That is, when we detect a file to be handled by the driver, emit the above
error message and do not open the dataset, unless the user defines the
environment variable.
Similarly in the Create()/CreateCopy() methods.
If we ship this in 3.3, with a 3.5 milestone for removal, this would offer a
feedback period of one year / 2 feature versions.


That sounds a reasonable timetable.

The next Ubuntu LTS is due for release April 2022, so is likely to have
GDAL 3.4 and the avoidable warning, which is good. (If it had GDAL 3.5
people moving to it from a previous LTS would not see the warning ...)

Where are the costs in keeping these drivers ?
If as Kurt describes, the costs are for the user, we already have plugins
and options to not include drivers in the main binary. We could add a 3 way
switch like the kernel: builtin/plugin/not built which would make things
simple for packagers and users.

I am assuming that these are drivers that don't get much maintenance
at the moment, so the gain for GDAL developers is not having to add
new GDAL features to so many drivers ?

--
Andrew C. Aitchison Kendal, UK
and...@aitchison.me.uk
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev