yes, GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR is the trick to both disable
directory listing and preventing individual file probing.
Javier, do you have GDAL_DISABLE_READDIR_ON_OPEN=YES also set? Otherwise
if GDAL_DISABLE_READDIR_ON_OPEN is not set, GDAL will normally issue a
S3 directory listing call, which should prevent later individual file
probing (if we have the list of files in the directory, we just need to
check the list, and don't need to do individual open()/GET attempts).
But that director listing is sometimes still undesirable if doing it in
a folder with a large number of files hence
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR. Or maybe you don't have
directory listing permission in that folder/bucket?
Related to that, Howard filed https://github.com/OSGeo/gdal/issues/9443
recently. but it is not that simple to find a different default behavior
that doesn't result in functional regressions.
I was wondering if a solution, restricted to COG, couldn't be to put a
hint in the so-called "header ghost area"
(https://gdal.org/drivers/raster/cog.html#header-ghost-area), like
SIDECAR_FILES=NO, that could be used by the GTiff driver. Not sure if it
is the right way to address the issue however (and would work only for
new files of course, and probably only on explicit user choice at file
creation, because there could be scenarios where a user would want to
provide side car files)
Le 27/04/2024 à 13:12, Michael Smith via gdal-dev a écrit :
Javier,
You can control with the GDAL_DISABLE_READDIR_ON_OPEN configuration
parameter (https://gdal.org/user/configoptions.html). Typically, I set
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR which will disable this.
Mike
--
Michael Smith
Remote Sensing/GIS Center
US Army Corps of Engineers
*From: *gdal-dev <gdal-dev-boun...@lists.osgeo.org> on behalf of
Javier Jimenez Shaw via gdal-dev <gdal-dev@lists.osgeo.org>
*Reply-To: *Javier Jimenez Shaw <j...@jimenezshaw.com>
*Date: *Saturday, April 27, 2024 at 7:01 AM
*To: *gdal dev <gdal-dev@lists.osgeo.org>
*Subject: *[gdal-dev] don't search side car files on s3 with gdalinfo
Hi
I am accessing a big file in S3. All works fine (when I learned how to
set the authentication environment variables), but gdalinfo is
checking for a lot of files that do not exist. I know there is only a
GeoTIFF - COG (no side car files)
However gdalinfo is trying to find several side car files, that are
not there. S3 returns a 403. It takes its time to search for all those
files.
Is there a way to tell to gdalinfo that it should not check for those
files?
Bonus: QGIS is also taking time to open it (just using 'qgis' instead
of 'gdalinfo' in the command line). I think it is for the same reason.
It there a hack there as well?
Thanks.
$ AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
AWS_SESSION_TOKEN=... gdalinfo /vsis3/blah/folder/ortho.tiff
Driver: GTiff/GeoTIFF
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.tiff.aux.xml: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.aux: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.AUX: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.tiff.aux: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.tiff.AUX: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.tiff.ovr: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.tiff.OVR: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.tiff.msk: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.tiff.MSK: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.XML: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.xml: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.IMD: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.imd: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.RPB: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.rpb: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.PVL: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.pvl: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_rpc.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_RPC.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_metadata.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_METADATA.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_MTL.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_MTL.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/METADATA.DIM: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/metadata.dim: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_metadata.xml: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho_METADATA.XML: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.RPC: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.rpc: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.pass: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/ortho.PASS: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/summary.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/SUMMARY.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/HDRdisplay.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/HDRdisplay.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/HDRho_display.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/HDRho_display.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/RPCdisplay.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/RPCdisplay.TXT: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/RPCho_display.txt: 403
Warning 1: HTTP response code on
https://blah.s3.amazonaws.com/folder/RPCho_display.TXT: 403
Files: /vsis3/blah/folder/ortho.tiff
...
.___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... .... ._ .__
_______________________________________________ gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
