Hi again, To loop back in the mailing list on the conversation Even and I had on the PR, the answer to my original question (at least in the /vsis3/ use case) was to enable CPL_DEBUG, which outputs the raw response in these sorts of error cases, e.g.:
"CPLE_None in S3: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>a-very-very-fake-s3-bucket</BucketName><RequestId>...</RequestId><HostId>...</HostId></Error>" I'd been thrown off by the fact that rasterio captures GDAL's CPL_DEBUG logs and re-emits them in Python, but CPL_CURL_VERBOSE messages aren't captured in the same way. Some quirks in the configuration of the logging on my end meant the DEBUG log messages were being discarded when I thought they weren't. Regards, Daniel On Wed, 6 Jul 2022 at 18:55, Even Rouault <even.roua...@spatialys.com> wrote: > Daniel, > > https://github.com/OSGeo/gdal/pull/6028 should help, but with the > restriction indicated in the commit message. > > Even > Le 06/07/2022 à 13:13, Daniel Evans a écrit : > > Hi, > > I'm trying to diagnose some issues with accessing remote datasets in GDAL > - I believe the problem is unrelated to GDAL itself, but it's proving hard > to debug. > > By enabling CPL_CURL_VERBOSE, I can see that the server is kicking back a > 400 Bad Request status, but the debug output doesn't include the actual > response content, which would allow me to understand the problem in more > detail. Is there a way to get GDAL/cURL to output the response content for > failed requests? > > > By way of example, if I attempt to access a nonexistent file on S3 with > cURL directly, I get the 404 error code and the descriptive XML error > message at the end: > > $ curl -v https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif > [...] > < HTTP/1.1 404 Not Found > < Content-Type: application/xml > < Transfer-Encoding: chunked > < Date: Wed, 06 Jul 2022 11:09:09 GMT > < Server: AmazonS3 > --- these lines --- > < > <?xml version="1.0" encoding="UTF-8"?> > <Error><Code>NoSuchBucket</Code><Message>The specified bucket does not > exist</Message><BucketName>a-very-very-fake-s3-bucket</BucketName>[...]</Error> > --- these lines --- > > > In contrast, the output from GDAL with CPL_CURL_VERBOSE excludes the XML > response: > > < HTTP/1.1 404 Not Found > < Content-Type: application/xml > < Transfer-Encoding: chunked > < Date: Wed, 06 Jul 2022 11:04:16 GMT > < Server: AmazonS3 > > --- no XML output here --- > > * Connection #1 to host a-very-very-fake-s3-bucket.s3.amazonaws.com left > intact > * Couldn't find host a-very-very-fake-s3-bucket.s3.amazonaws.com in the > .netrc file; using defaults > * Found bundle for host a-very-very-fake-s3-bucket.s3.amazonaws.com: > 0x25b5080 [can pipeline] > * Re-using existing connection! (#1) with host > a-very-very-fake-s3-bucket.s3.amazonaws.com > * Connected to a-very-very-fake-s3-bucket.s3.amazonaws.com > (52.216.207.115) port 443 (#1) > > GET /?delimiter=%2F&max-keys=100&prefix=example.tif%2F HTTP/1.1 > [...] > > Regards, > Daniel > > _______________________________________________ > gdal-dev mailing > listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- http://www.spatialys.com > My software is free, but my time generally not. > >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev