Hi, I'm in contact with google about the problem. It might be a problem on their side.
But during testing I discovered something else which I found unusual. Maybe I understand the meaning of the GDAL_DISABLE_READDIR_ON_OPEN variable incorrectly but I still wanted to double check. Using: #!/bin/bash export GS_ACCESS_KEY_ID=<OUR_KEY> export GS_SECRET_ACCESS_KEY=<OUR_KEY> export GDAL_DISABLE_READDIR_ON_OPEN=TRUE export CPL_DEBUG=ON gdalinfo /vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2 Tries to open tons of files like e.g. https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/HDR_20170719T112119_B01.txt The other to settings FALSE and EMPTY_DIR still try to open some auxiliary files. But not as many. In this case it seems that setting GDAL_DISABLE_READDIR_ON_OPEN=TRUE is by far the slowest option. Is this expected? Thanks, Christoph On Wed, 16 Oct 2019 at 17:38, Christoph Paulik <[email protected]> wrote: > Hi, > > The thing is that one request works fine in any case. > But google seems to rate limit on the public bucket in the following cases: > > - no authentication (/vsicurl/ driver) > - authentication with automatic detection by gdal that it is running on a > GCE VM. > > but not when using the explicit authentication with > GS_ACCESS_KEY_ID/GS_SECRET_ACCESS_KEY > > Christoph > > > On Wed, 16 Oct 2019 at 17:31, Even Rouault <[email protected]> > wrote: > >> Hi, >> >> did you try with Google gsutil tool ? >> >> Anyway as it is a public resource that doesn't require any signing, you >> could >> just use >> >> /vsicurl/ >> https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/ >> >> NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/ >> L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2 >> <https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2> >> >> Even >> >> > Hi, >> > >> > When doing a few requests to the Sentinel 2 bucket of Google ( >> > https://cloud.google.com/storage/docs/public-datasets/sentinel-2) we >> have >> > been getting 503 Responses when using the automatic authentication. See >> the >> > end of this email for the full output. >> > >> > This behavior goes away when using e.g. the >> > GS_ACCESS_KEY_ID/GS_SECRET_ACCESS_KEY authentication method. >> > >> > The Service account this machine is running with has "Full Access to All >> > Cloud APIs" >> > >> > Not sure if this is a gdal problem or actually a google internal issue >> > since accessing our own buckets works fine. >> > >> > Best, >> > Christoph >> > >> > Output: >> > export CPL_DEBUG=ON >> > gdalinfo >> > >> /vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N >> > >> 0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T11 >> > 2511/IMG_DATA/T29TNE_20170719T112119_B01.jp2 >> > >> > HTTP: Fetch( >> > >> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts >> > /default/token ) >> > HTTP: libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.5 libssh2/1.8.2 >> > HTTP: GDAL was built against curl 7.64.1, but is running against 7.65.3. >> > HTTP: These HTTP headers were set: Metadata-Flavor: Google >> > GOA2: Refresh Token Response: >> > {"access_token":"REMOVED IN >> EMAIL","expires_in":2877,"token_type":"Bearer"} >> > GOA2: Access Token : 'REMOVED IN EMAIL' >> > GS: Using GCE inherited permissions >> > GS: <html><head><meta http-equiv="content-type" content="text/html; >> > charset=utf-8"/><title>Sorry...</title><style> body { font-family: >> verdana, >> > arial, sans-serif; background-color: #fff; color: #000; >> > }</style></head><body><div><table><tr><td><b><font face=sans-serif >> > size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font >> > color=#fbbc05>o</font><font color=#4285f4>g</font><font >> > color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td >> > style="text-align: left; vertical-align: bottom; padding-bottom: 15px; >> > width: 50%"><div style="border-bottom: 1px solid >> > #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: >> > 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be >> > sending automated queries. To protect our users, we can't process your >> > request right now.</p></div><div style="margin-left: 4em;">See <a href=" >> > https://support.google.com/websearch/answer/86640";>Google Help</a> for >> more >> > information.<br/><br/></div><div style="text-align: center; border-top: >> 1px >> > solid #dfdfdf;"><a href="https://www.google.com";>Google >> > Home</a></div></body></html> >> > VSICURL: GetFileSize( >> > >> https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_ >> > >> MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T2 >> > 9TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2)=0 >> > response_code=503 >> > GS: <html><head><meta http-equiv="content-type" content="text/html; >> > charset=utf-8"/><title>Sorry...</title><style> body { font-family: >> verdana, >> > arial, sans-serif; background-color: #fff; color: #000; >> > }</style></head><body><div><table><tr><td><b><font face=sans-serif >> > size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font >> > color=#fbbc05>o</font><font color=#4285f4>g</font><font >> > color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td >> > style="text-align: left; vertical-align: bottom; padding-bottom: 15px; >> > width: 50%"><div style="border-bottom: 1px solid >> > #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: >> > 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be >> > sending automated queries. To protect our users, we can't process your >> > request right now.</p></div><div style="margin-left: 4em;">See <a href=" >> > https://support.google.com/websearch/answer/86640";>Google Help</a> for >> more >> > information.<br/><br/></div><div style="text-align: center; border-top: >> 1px >> > solid #dfdfdf;"><a href="https://www.google.com";>Google >> > Home</a></div></body></html> >> > ERROR 11: HTTP response code: 503 >> > gdalinfo failed - unable to open >> > >> '/vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_ >> > >> N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T1 >> > 12511/IMG_DATA/T29TNE_20170719T112119_B01.jp2'. >> >> >> -- >> Spatialys - Geospatial professional services >> http://www.spatialys.com >> > > > -- > *Christoph Paulik* // Head of Infrastructure > VanderSat // Satellite observed water data. Globally. Daily. > Wilhelminastraat 43a, 2011 VK, Haarlem (NL), The Netherlands > <https://maps.google.com/?q=Wilhelminastraat+43a,+2011+VK,+Haarlem+(NL),+The+Netherlands&entry=gmail&source=g> > *M* +31 6 18271928 *W* www.vandersat.com > > > -------------------------------------------------------------------------------------- > -- *Christoph Paulik* // Head of Infrastructure VanderSat // Satellite observed water data. Globally. Daily. Wilhelminastraat 43a, 2011 VK, Haarlem (NL), The Netherlands <https://maps.google.com/?q=Wilhelminastraat+43a,+2011+VK,+Haarlem+(NL),+The+Netherlands&entry=gmail&source=g> *M* +31 6 18271928 *W* www.vandersat.com --------------------------------------------------------------------------------------
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
