sathesuraj commented on issue #2207:
URL: https://github.com/apache/polaris/issues/2207#issuecomment-3486495341
@dimas-b / @adutra : I attempted to create a catalog and tables using
Polaris Catalog service version 1.2.0, but was unable to create a table when
using a non-AWS S3-compatible object storage service without STS. Below, I have
summarized the testing steps and observations. Please let me know if I have
overlooked any configuration or setup requirements.
Aim - To create a catalog and table in Polaris Catalog using a non-AWS
S3-compatible storage endpoint.
Setup Details
1. Polaris Catalog Service
- Version: 1.2.0 (Docker image)
- Application property file added before service start.
- Environment variable AWS_SHARED_CREDENTIALS_FILE set to a credentials
file containing non-AWS S3 access key and secret.
- Expectation: The default credential provider chain in Polaris should
use these credentials.
2. Python Application / REST API
- Used to call Polaris REST API for catalog creation and to set up
roles, permissions, and related configurations.
3. Spark Client
- Used to create a table in the catalog.
- X-Iceberg-Access-Delegation property is not set in spark client
Polaris Log Snippet (Catalog Creation)
Created new catalog class PolarisCatalog {
class Catalog {
type: INTERNAL
name: demo_catalog
properties: class CatalogProperties {
{default-base-location=s3://non-aws-bucket/demo_catalog/warehouse/}
defaultBaseLocation: s3://non-aws-bucket/demo_catalog/warehouse/
}
createTimestamp: 1234567891011
lastUpdateTimestamp: 0
entityVersion: 1
storageConfigInfo: class AwsStorageConfigInfo {
class StorageConfigInfo {
storageType: S3
allowedLocations:
[s3://non-aws-bucket/demo_catalog/warehouse/*,
s3://non-aws-bucket/demo_catalog/warehouse/]
}
roleArn: null
externalId: null
userArn: null
region: null
endpoint: https://s3.object.net:port
stsEndpoint: null
stsUnavailable: true
endpointInternal: https://s3.object.net:port
pathStyleAccess: true
}
}
}
Observed Error -
Caused by: java.net.UnknownHostException:
non-aws-bucket-name.s3.region.amazonaws.com: Name or service not known
Questions & Clarifications Needed
1. Unexpected AWS Hostname Resolution
- Why is Polaris still attempting to connect to
non-aws-bucket-name.s3.region.amazonaws.com (AWS domain host) instead of the
specified non-AWS S3 endpoint (https://s3.object.net:port)?
- Is there a configuration or property that needs to be set to ensure
Polaris uses the custom endpoint for S3 operations?
2. Credential Provider Chain Behavior
- When stsUnavailable is set to true, does Polaris use the default
credential provider chain to retrieve access keys and secrets from the
environment variable (AWS_SHARED_CREDENTIALS_FILE)?
- Are there additional steps required to ensure Polaris picks up
credentials for non-AWS S3 endpoints?
Due to time constraints, I have not been able to review the Polaris source
code in detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
