Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-20 Thread Michael Smith
: Even Rouault Date: Sunday, November 20, 2022 at 12:52 PM To: Michael Smith Cc: gdal-dev Subject: Re: [gdal-dev] errors using IAM instance profile auth in s3 Mike, I've ended up firing a EC2 instance and I did replicate with my private bucket too. With a EC2 Ubuntu 22.04 instance, on a mi

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-20 Thread Even Rouault
pszFilename, gdal.GA_ReadOnly) In [6]: hDataset Out[6]: 'GDALDatasetShadow *' at 0x7f827217c450> > In [7]: hDataset.GetGeoTransform() Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0) Mike *From: *Even Rouault *Date: *Saturday, November 19, 2022 at 10:08 AM *To: * *Cc: *gdal-dev *Subject:

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-20 Thread Michael Smith
["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3301"]]' > > On Sun, Nov 20, 2022 at 8:47 AM Michael Smith < > michael.smith.e...@gmail.com> wrote: > >> Is there a reason why OpenEx woul

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-20 Thread Michael Smith
l.OpenEx(pszFilename, gdal.GA_ReadOnly) > > In [6]: hDataset > > Out[6]: 'GDALDatasetShadow *' at 0x7f827217c450> > > > In [7]: hDataset.GetGeoTransform() > > Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0) > > > > Mike > > > >

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-20 Thread Michael Smith
dal.OpenEx(pszFilename, gdal.GA_ReadOnly) In [6]: hDataset Out[6]: > In [7]: hDataset.GetGeoTransform() Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0) Mike From: Even Rouault Date: Saturday, November 19, 2022 at 10:08 AM To: Cc: gdal-dev Subject: Re: [gdal-dev] errors using IAM ins

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread Michael Smith
debug. Mike From: Even Rouault Date: Saturday, November 19, 2022 at 10:08 AM To: Cc: gdal-dev Subject: Re: [gdal-dev] errors using IAM instance profile auth in s3 Le 19/11/2022 à 16:00, michael.smith.e...@gmail.com a écrit : Correct, not a public bucket, which is why the IAM

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread Even Rouault
Le 19/11/2022 à 16:00, michael.smith.e...@gmail.com a écrit : Correct, not a public bucket, which is why the IAM credentials are needed. If I set them manually, it all works fine. That's super weird if the result of a range request changes depending on how credentials have been set... Perhaps

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread michael . smith . erdc
Correct, not a public bucket, which is why the IAM credentials are needed. If I set them manually, it all works fine.[ u02]$ export AWS_ACCESS_KEY_ID=x[ u02]$ gdalinfo /vsis3/grid-dev-publiclidar/estoni

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread Even Rouault
Yes, a 206 response code means success here as we are requesting only bytes 0-16383. So maybe the file is not a valid TIFF ? ( "grid-dev-publiclidar" must not be so public I guess, because when trying with my credentials, I get a Access Denied) Le 19/11/2022 à 15:40, michael.smith.e...@gmail.

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread michael . smith . erdc
I’m seeing that it’s getting a 206 response code, so wouldn’t that indicate auth is working? gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif HTTP: Fetch(http://169.254.169.254/latest/api/token) HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0 HTTP:

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread michael . smith . erdc
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3 { "Code" : "Success", "LastUpdated" : "2022-11-19T14:30:52Z", "Type" : "AWS-HMAC", "AccessKeyId" : "xxx", "SecretAccessKey" : "x/1ssuwVLuUevA", "Token" : "xz6ZANv

Re: [gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread Even Rouault
Hi Mike, could you send the output of curl http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3 Slightly redacted of course, but with the exact formatting. This part of thee code currently uses a "simple JSON parser" (https://github.com/OSGeo/gdal/blob/c61d116a469821

[gdal-dev] errors using IAM instance profile auth in s3

2022-11-19 Thread michael . smith . erdc
Just recently I’m seeing errors using iam profile authentication using /vsis3 drivers in gdal (over multiple versions). gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia-dtm-5m.tif HTTP: Fetch(http://169.254.169.254/latest/api/token) HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2