aiwhj opened a new issue, #8347: URL: https://github.com/apache/iceberg/issues/8347
### Query engine Flink + Minio ### Question I have been reading the official documentation [AWS (apache.org)](https://iceberg.apache.org/docs/latest/aws/#glue-catalog), which suggests using org.apache.iceberg.aws.glue.GlueCatalog when using S3. However, when I use GlueCatalog, I always encounter an error stating that AWS_ACCESS_KEY_ID cannot be found. I have followed the documentation and tried various different formats for setting it, but the error persists. ``` CREATE TABLE `icebergresult` ( `id` BIGINT NOT NULL, `name` STRING NOT NULL, `age` BIGINT NOT NULL, PRIMARY KEY (`id`) NOT ENFORCED ) WITH ( 'connector'='iceberg', 'catalog-name'='test_log', 'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog', 'warehouse'='s3://iceberg/my/key/prefix', 'fs.s3.access.key' = 'DNxkAw**********', 'fs.s3.secret.key' = 'YVMYE78PgQ8xBizDPm*************', 'fs.s3.access-key' = 'DNxkAw**********', 'fs.s3.secret-key' = 'YVMYE78PgQ8xBizDPm*************', 'fs.s3.awsAccessKeyId' = 'DNxkAw**********', 'fs.s3.awsSecretAccessKey' = 'YVMYE78PgQ8xBizDPm*************', 'fs.s3.access-key-id' = 'DNxkAw**********', 'fs.s3.secret-access-key' = 'YVMYE78PgQ8xBizDPm*************', 'fs.s3.endpoint' = 'http://127.0.0.1:30619', 's3.endpoint' = 'http://127.0.0.1:30619', 's3.access-key' = 'DNxkAw**********', 's3.secret-key' = 'YVMYE78PgQ8xBizDPm*************', 's3.awsAccessKeyId' = 'DNxkAw**********', 's3.awsSecretAccessKey' = 'YVMYE78PgQ8xBizDPm*************', 's3.access-key-id' = 'DNxkAw**********', 's3.secret-access-key' = 'YVMYE78PgQ8xBizDPm*************', 's3.access.key' = 'DNxkAw**********', 's3.secret.key' = 'YVMYE78PgQ8xBizDPm*************', 'aws.accessKeyId' = 'DNxkAw**********', 'aws.SecretAccessKey' = 'YVMYE78PgQ8xBizDPm*************', 's3.path.style.access'= 'true', 's3.path-style'= 'true', 'client.region'='us-east-1', 'io-impl'='org.apache.iceberg.aws.s3.S3FileIO' ); ``` ``` Flink SQL> select * from icebergresult; SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [ERROR] Could not execute SQL statement. Reason: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javap roperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])): Profile file contained no credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.] ``` It seems like the error is returned by AWS rather than Minio, so I checked the source code of iceberg_runtime. When initializing the Client in GlueCatalog, awsClientFactory.glue() is used, but it doesn't seem to utilize applyS3EndpointConfigurations. This seems strange to me. Does GlueCatalog not support connecting directly to Minio's S3? ``` // aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java initialize( name, properties.get(CatalogProperties.WAREHOUSE_LOCATION), new AwsProperties(properties), awsClientFactory.glue(), initializeLockManager(properties)); ``` ``` // aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java @Override public S3Client s3() { return S3Client.builder() .applyMutation(awsProperties::applyClientRegionConfiguration) .applyMutation(awsProperties::applyHttpClientConfigurations) .applyMutation(awsProperties::applyS3EndpointConfigurations) .applyMutation(awsProperties::applyS3ServiceConfigurations) .applyMutation(awsProperties::applyS3CredentialConfigurations) .applyMutation(awsProperties::applyS3SignerConfiguration) .build(); } @Override public GlueClient glue() { return GlueClient.builder() .applyMutation(awsProperties::applyClientRegionConfiguration) .applyMutation(awsProperties::applyHttpClientConfigurations) .applyMutation(awsProperties::applyGlueEndpointConfigurations) .applyMutation(awsProperties::applyClientCredentialConfigurations) .build(); } ``` How should I configure these properties when using Minio? Can you provide an example? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
