Laszlo Gaal created IMPALA-13156:
------------------------------------
Summary: S3 tests flaky: the Hadoop-AWS credential provider chain
occasionally fails to provide credentials for S3 access
Key: IMPALA-13156
URL: https://issues.apache.org/jira/browse/IMPALA-13156
Project: IMPALA
Issue Type: Bug
Reporter: Laszlo Gaal
During S3-based test runs executed on private infrastructure the default
Hadoop-AWS credential provider throws this error occasionally:
{code}
2024-02-17T18:02:10,175 WARN [TThreadPoolServer WorkerProcess-22]
fs.FileSystem: Failed to initialize fileystem
s3a://****redacted****/test-warehouse/test_num_values_def_levels_mismatch_15b31ddb.db/too_many_def_levels:
java.nio.file.AccessDeniedException: ****redacted****:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from system settings. Access key must be specified either via
environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
2024-02-17T18:02:10,175 ERROR [TThreadPoolServer WorkerProcess-22]
utils.MetaStoreUtils: Got exception: java.nio.file.AccessDeniedException
****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS
Credentials provided by TemporaryAWSCredentialsProvider
SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider
IAMInstanceCredentialsProvider :
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from system settings. Access key must be specified either via
environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
java.nio.file.AccessDeniedException: ****redacted****:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from system settings. Access key must be specified either via
environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
at
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.maybeTranslateCredentialException(AWSCredentialProviderList.java:351)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$2(S3AFileSystem.java:972)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2748)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:970)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing(S3AFileSystem.java:859)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:715)
~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:124)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:132)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:860)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2573)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2807)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_261]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at com.sun.proxy.$Proxy30.create_table_req(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19574)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19553)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
~[libthrift-0.16.0.jar:0.16.0]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_261]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_261]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
~[libthrift-0.16.0.jar:0.16.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_261]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]
{code}
Occasionally means once or twice during a run taking about 8 hours. Also, this
does not happen during each run, but it is seen roughly once in every 3 or four
attempts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
