ja-michel opened a new issue, #5567: URL: https://github.com/apache/iceberg/issues/5567
### Query engine Spark 3.3.1 stand-alone Iceberg 0.14.0 iceberg-spark-runtime-3.3_2.12 0.14.0 ``` org.apache.iceberg:iceberg-aws:0.14.0 org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:0.14.0 org.apache.hadoop:hadoop-aws:3.3.3 ``` ### Question I'm running a simple spark application in a standalone pod inside and EKS cluster. The pods have a service account associated and the environment variables set: ``` AWS_DEFAULT_REGION: eu-west-2 AWS_REGION: eu-west-2 AWS_ROLE_ARN: arn:aws:iam::0000000000:role/MyServiceAccountRole AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token ``` This is my Spark configuration (relevant to Iceberg): ``` spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions iceberg.engine.hive.enabled=true spark.sql.catalog.ice=org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.ice.warehouse=s3a://slc-data-user-live/spark-jobs/warehouse spark.sql.catalog.ice.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog spark.sql.catalog.ice.io-impl=org.apache.iceberg.aws.s3.S3FileIO ``` When the spark context starts and the `DefaultAwsClientFactory` is created, I don't think it is using the `DefaultCredentialsProvider` because the clients ultimately use the node's role and not the role in the env variables and web identity token. **Q:** How do I configure the client factory to ensure it is using the `DefaultCredentialsProvider` and the `AWS_WEB_IDENTITY_TOKEN_FILE` env variable https://github.com/apache/iceberg/blob/6d2edd6284ebc5301dbe45376a31ca8316852a77/aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java#L181-L194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
