rohangoli opened a new issue, #2743: URL: https://github.com/apache/polaris/issues/2743
### Is your feature request related to a problem? Please describe. Using Polaris with On-Premise S3 Storage like Dell ECS (in-built S3, IAM & STS services), code built on [1.2.0-incubating-SNAPSHOT](https://github.com/apache/polaris/tree/main) fails at following: - Create Table fails with #2207 error ``` org.apache.iceberg.exceptions.RESTException: Unable to process: Failed to get subscoped credentials: (Service: Sts, Status Code: 400, Request ID: null) (SDK Attempt Count: 1) ``` - Fails to handle custom STS endpoints with suffix (ex: https://ecs-prod.example.com:4443/sts) [NO references with #1146 & #2672] - Fails to parse ECS-Style STS XML response as Fallback when non-AWS Style response comes - [ECS Management API - IAM - STS Role](https://portal.ecstestdrive.com/Document/ECS/3.8/apidocs/StsService_AssumeRole_d41289fb2f39151a98d43ffd16dabaab_cfc300b9ce827245532e8894bd9ddbbb_detail.html) [NO references with #1146 & #2672] Or Instead use standardized JSON implementation ``` REQUEST: POST https://ecs-prod.example.com:4443/sts?Action=AssumeRole&RoleSessionName=temp&RoleArn=urn:ecs:iam::s3:role/s3assumeRole&DurationSeconds=7200 Accept: */* X-SDS-AUTH-TOKEN: <AUTH_TOKEN> RESPONSE: HTTP/1.1 200 OK Content-Type: application/xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ns2:AssumeRoleResponse xmlns:ns2="none"> <ResponseMetadata> <RequestId>0af7b369:171a31ad5b8:12e3a:b</RequestId> </ResponseMetadata> <AssumeRoleResult> <AssumedRoleUser> <Arn>urn:ecs:sts::s3:assumed-role/s3assumeRole/temp</Arn> <AssumedRoleId>AROA48AEC87ADCFF133F:temp</AssumedRoleId> </AssumedRoleUser> <Credentials> <AccessKeyId>ASIA48DB3A69C19CFE56</AccessKeyId> <Expiration>2020-04-23T19:45:46Z</Expiration> <SecretAccessKey>wDvqneW8lyJbu0VRAMU8xQHPKH1hIsYfTyozlVg5znM</SecretAccessKey> <SessionToken>CgJzMxIFdXNlcjEaFE0EyQ . . . mi5SBHRlbXBo2sSH9QU</SessionToken> </Credentials> </AssumeRoleResult> </ns2:AssumeRoleResponse> ``` ``` REQUEST: POST https://192.168.0.0:4443/sts?Action=AssumeRole&RoleSessionName=temp&RoleArn=urn:ecs:iam::s3:role/s3assumeRole&DurationSeconds=7200 Accept: application/json X-SDS-AUTH-TOKEN: <AUTH_TOKEN> RESPONSE: HTTP/1.1 200 OK Content-Type: application/json { "ResponseMetadata": { "RequestId": "0af9f5b8:171648dacb1:1bb9a:27" }, "AssumedRoleUser": { "AssumedRoleId": "AROAB7DED71BF3317149:user1-105-temp", "Arn": "urn:ecs:sts::s3:assumed-role/s3assumeRole/user1-105-temp" }, "Credentials": { "SecretAccessKey": "5KddtnuIyrfaCw7Yt1-s7_r24FU1VHl4Odyj7az_zzg", "SessionToken": "CgJzMxIFdXNlcjEaFEFS . . . T0EyQTdCRES10ZW1waKPGh_UF", "Expiration": "2020-04-24T02:49:07Z", "AccessKeyId": "ASIA57751957F480BE8E" } } ``` - Failes to handle custom STS ARN (ex: urn:ecs:iam::test-namespace:role/s3assumeRole vs. default: arn:aws:iam::123456789012:role/my-role) - Ignore SSL Verification Flag for development/test purposes #2705 ### Describe the solution you'd like Working Code Changes are available here: https://github.com/rohangoli/polaris/tree/ecs-onpremise - AWS/ECS STS handling: - Add or extend parsing for ECS-style STS credential XML responses: `org.apache.polaris.core.secrets.EcsXmlParser` plus unit tests. - Add an interceptor to transform ECS namespace values: `org.apache.polaris.core.storage.aws.EcsNamespaceTransformInterceptor`. - Introduce/update STS client provider abstractions: `org.apache.polaris.core.storage.aws.StsClientProvider` and a runtime pool implementation `org.apache.polaris.service.storage.aws.StsClientsPool`. - AwsCredentialsStorageIntegration update to use the new parsing/fallbacks and namespace transform behavior: `org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration`. - Configuration / model extensions: - `AwsStorageConfigurationInfo` gains fields/behaviors (endpointInternal, ignoreSSLVerification, pathStyleAccess, etc.) and serialization test coverage update: `org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo` and tests. - `CatalogEntity.Builder.setStorageConfigurationInfo(...)` mapping update to carry these new fields into internal configuration — see `org.apache.polaris.core.entity.CatalogEntity.Builder`. - Management API serialization tests update to expect the new JSON structure in catalog serialization: CatalogSerializationTest. - Runtime wiring: - New CDI/Jakarta producer or service registrations to expose `StsClientsPool`/`StsClientProvider` instances: `org.apache.polaris.service.config.ServiceProducers`. - `DefaultFileIOFactory` update to consider the Polaris storage access properties (endpoint / path-style / key/secret) and to merge properties and subscoped creds appropriately: `org.apache.polaris.service.catalog.io.DefaultFileIOFactory`. - Tests & tooling: - Tests add to validate ECS parsing fallback behavior and role ARN patterns. - STS raw request/response logging/capture tests add: see the two RawSts* tests. - MinIO testcontainer tooling update/add. ### Describe alternatives you've considered _No response_ ### Additional context I've modified the code to fit the above purposes as describe in the solution - https://github.com/rohangoli/polaris/tree/ecs-onpremise -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
