[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701734#comment-16701734
]
Steve Loughran commented on HADOOP-15808:
-----------------------------------------
Revisited this to see about changing message for token loading.
I'm actually reluctant to do this because of the following classpath isn't as
rare as you think:
* hadoop-aws on CP, aws-sdk not.
We saw that with the FileSystem service loader, logging warnings created too
many stack traces, issues related to "why am I being told about s3 when I'm
working with HDFS, etc".
I would like to keep this noise out. Now, it will hide the issue "there's an
implementation declared which isn't going to load", but if you are really
looking for a token, the fact it won't load will soon surface. The problem is
that we've lost some of the root cause. Really you'd want to cache the stack
and only print it in a specific failure, e.g when a token can't be decoded
because its kind is unrecognised.
But I can't see an easy way to do that with the current code.
the one thing we could do is cache the fact that 1+ warning was found and then
in {{Token.decodeIdentifier()}}, when there's no match for a token kind, log
that fact.
The problem I have is
* this stuff gets used in a lot of places
* its currently fairly brittle code: one bad class and *nothing* will load.
* A patch to really harden this stuff with null checks everywhere didn't take
I'm confident that the current design is going to fail badly once HADOOP-14556
goes in. I've looked to see if I can get away with making sure the token
identifier has no indirect refs to any AWS libraries, but it's a doomed
exercise: no way to test, and it'll inevitably regress.
> Harden Token service loader use
> -------------------------------
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
> Affects Versions: 2.9.1, 3.1.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch,
> HADOOP-15808-003.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
