[
https://issues.apache.org/jira/browse/HADOOP-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424988#comment-15424988
]
Chris Nauroth commented on HADOOP-13252:
----------------------------------------
Patch 005 for branch-2 looks good after addressing the Checkstyle nitpicks.
However, I am seeing test failures when I apply this patch to trunk. (Full
stack trace below.) It seems like getting the classes out of {{Configuration}}
is behaving differently on trunk vs. branch-2.
{code}
awsClasses = conf.getClasses(AWS_CREDENTIALS_PROVIDER,
AWSCredentialsProvider.class);
{code}
The difference in behavior might be due to HADOOP-7851, which I notice is in
trunk but not branch-2.
Steve, would you please take a look? {{AWSCredentialsProvider}} can't be used
as the default, because it can't be instantiated.
{code}
testOverwriteNonEmptyDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate)
Time elapsed: 1.362 sec <<< ERROR!
java.io.IOException: com.amazonaws.auth.AWSCredentialsProvider constructor
exception. A class specified in fs.s3a.aws.credentials.provider must provide
an accessible constructor accepting URI and Configuration, or an accessible
default constructor.
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:319)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:286)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2805)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2842)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2824)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:388)
at
org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.lang.NoSuchMethodException:
com.amazonaws.auth.AWSCredentialsProvider.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:315)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:286)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2805)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2842)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2824)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:388)
at
org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> Tune S3A provider plugin mechanism
> ----------------------------------
>
> Key: HADOOP-13252
> URL: https://issues.apache.org/jira/browse/HADOOP-13252
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-13252-branch-2-001.patch,
> HADOOP-13252-branch-2-003.patch, HADOOP-13252-branch-2-004.patch,
> HADOOP-13252-branch-2-005.patch
>
>
> We've now got some fairly complex auth mechanisms going on: -hadoop config,
> KMS, env vars, "none". IF something isn't working, it's going to be a lot
> harder to debug.
> Review and tune the S3A provider point
> * add logging of what's going on in s3 auth to help debug problems
> * make a whole chain of logins expressible
> * allow the anonymous credentials to be included in the list
> * review and updated documents.
> I propose *carefully* adding some debug messages to identify which auth
> provider is doing the auth, so we can see if the env vars were kicking in,
> sysprops, etc.
> What we mustn't do is leak any secrets: this should be identifying whether
> properties and env vars are set, not what their values are. I don't believe
> that this will generate a security risk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
