Steve Loughran created HADOOP-18481:
---------------------------------------
Summary: AWS v2 SDK warning to skip warning of
EnvironmentVariableCredentialsProvider
Key: HADOOP-18481
URL: https://issues.apache.org/jira/browse/HADOOP-18481
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Ahmar Suhail
looking at test output with the sdk warnings enabled, it is now always warning
of a v1 provider reference, even if the user hasn't set any
fs.s3a.credential.provider option
{code}
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade
(LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential
provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK V1
credential providers will be removed once S3A is upgraded to SDK V2
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
com.amazonaws.auth.EnvironmentVariableCredentialsProvider
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
{code}
This is because the EnvironmentVariableCredentialsProvider provider is on the
default list of providers.
Everybody who is using the S3 a connector and who has not explicitly declared a
new set of providers excluding this one will be seeing the error message.
Proposed:
Don't warn on this provider. Instead with the v2 move the classname can be
patched to switch to a modified one.
The alternative would be to provide an s3a specific env var provider subclass
of this; and while that is potentially good in future it is a bit more effort
for the forthcoming 3.3.5 release.
And especially because and it will not be in previous versions people cannot
explicitly switch to it in their configs and be confident it will always be
there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
