[
https://issues.apache.org/jira/browse/IMPALA-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13156 started by Laszlo Gaal.
--------------------------------------------
> S3 tests flaky: the Hadoop-AWS credential provider chain occasionally fails
> to provide credentials for S3 access
> ----------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-13156
> URL: https://issues.apache.org/jira/browse/IMPALA-13156
> Project: IMPALA
> Issue Type: Bug
> Reporter: Laszlo Gaal
> Assignee: Laszlo Gaal
> Priority: Major
>
> During S3-based test runs executed on private infrastructure the default
> Hadoop-AWS credential provider throws this error occasionally:
> {code}
> 2024-02-17T18:02:10,175 WARN [TThreadPoolServer WorkerProcess-22]
> fs.FileSystem: Failed to initialize fileystem
> s3a://****redacted****/test-warehouse/test_num_values_def_levels_mismatch_15b31ddb.db/too_many_def_levels:
> java.nio.file.AccessDeniedException: ****redacted****:
> org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
> provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
> software.amazon.awssdk.core.exception.SdkClientException: Unable to load
> credentials from system settings. Access key must be specified either via
> environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> 2024-02-17T18:02:10,175 ERROR [TThreadPoolServer WorkerProcess-22]
> utils.MetaStoreUtils: Got exception: java.nio.file.AccessDeniedException
> ****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No
> AWS Credentials provided by TemporaryAWSCredentialsProvider
> SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider
> IAMInstanceCredentialsProvider :
> software.amazon.awssdk.core.exception.SdkClientException: Unable to load
> credentials from system settings. Access key must be specified either via
> environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> java.nio.file.AccessDeniedException: ****redacted****:
> org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
> provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
> software.amazon.awssdk.core.exception.SdkClientException: Unable to load
> credentials from system settings. Access key must be specified either via
> environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> at
> org.apache.hadoop.fs.s3a.AWSCredentialProviderList.maybeTranslateCredentialException(AWSCredentialProviderList.java:351)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$2(S3AFileSystem.java:972)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2748)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:970)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing(S3AFileSystem.java:859)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:715)
> ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:124)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:132)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:860)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2573)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2807)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_261]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at com.sun.proxy.$Proxy30.create_table_req(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19574)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19553)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> ~[libthrift-0.16.0.jar:0.16.0]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_261]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_261]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
> ~[libthrift-0.16.0.jar:0.16.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_261]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_261]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]
> {code}
> Occasionally means once or twice during a run taking about 8 hours. Also,
> this does not happen during each run, but it is seen roughly once in every 3
> or four attempts.
> These tests are run on EC2 VM instances, which have a specific IAM instance
> role associated with them. The role allows them access to the designated test
> buckets.
> For S3-based test runs, Impala is configured with the entry
> {{s.s3a.aws.credentials.provider}} in {{core-site.xml}} left empty. This
> leaves Hadoop-AWS to try all its credential providers in sequence, as
> described in the Hadoop-AWS docs at
> https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Simple_name.2Fsecret_credentials_with_SimpleAWSCredentialsProvider.2A
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
