Hi Ankinet, Thanks for reporting this. The proposal looks solid to me. To fix it, we should ensure that the system returns the region when vending AWS credentials. Specifically, the LoadTableResult should include the field client.region[1].
[1] https://github.com/apache/iceberg/blob/5439cbdb278232779fdd9a392bbf57f007f9bda0/open-api/rest-catalog-open-api.yaml#L3127-L3127 Yufei On Tue, Sep 10, 2024 at 8:04 PM Aniket Kulkarni <aniket.kulka...@dremio.com.invalid> wrote: > For iceberg tables stored in AWS S3 buckets, knowing the region of the > bucket is critical for engines using vended credentials (when configured) > to access a table. > > E.g - the vended credentials for AWS look like this > > { "s3.access-key-id": "ASI....”, > "s3.secret-access-key": "gbVT9PpFBY...”, > "s3.session-token": "IQoJb3JpZ2luX2VjEN3//////////...”, > "expiration-time": “1725572949000” } > > An engine consuming this, would need to either infer (s3api > get-bucket-location) the region or ask the end user to provide the region > separately which misses the point of vended credentials. > > A engine engine cannot use get-bucket-location, because the credential > generation explicitly allows only s3:GetObject, s3:GetObjectVersion, > s3:PutObject, s3:DeletObject, s3:ListBucket for the table location prefix. > Refer - > org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration#policyString > > I propose that > > - the storage setup for S3 should have parameter for the bucket region > (org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo) > - if the parameter is not specified, then Polaris attempts to look up > (get-bucket-location) the region. > - the information is returned in vended credentials (if enabled) as > "s3.region”:… > > Note - another option could be to allow ’s3:GetBucketLocation’ in the > policyString when generating vended credentials’ iam role, but that is sub > optimal and therefore I am not proposing it. It would engines to make > multiple get-bucket-location calls - one per table being looked up. > > -- > aniket > >
