[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754144#comment-16754144
]
Steve Loughran commented on HADOOP-15808:
-
Followup, this isn't enough: token.toString() will raise an exception if
there's any token which doesn't load
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/example/auth/TokenIdentifier
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:158)
at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:185)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.printTokensToString(DelegationTokenFetcher.java:224)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.printTokens(DelegationTokenFetcher.java:244)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher$1.run(DelegationTokenFetcher.java:139)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.main(DelegationTokenFetcher.java:135)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.main(DelegationTokenFetcher.java:72)
Caused by: java.lang.ClassNotFoundException:
com.google.cloud.hadoop.fs.gcs.auth.AbstractGCPTokenIdentifier
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 27 more
{code}
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.1.2, 3.2.1
>
> Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch,
> HADOOP-15808-003.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717660#comment-16717660 ] Hudson commented on HADOOP-15808: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15591 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15591/]) HADOOP-15808. Harden Token service loader use. (stevel: rev 202926ac3301298753abd0e6e1f324caf0202ec6) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/DtFileOperations.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.2, 3.2.1 > > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717616#comment-16717616 ] Larry McCay commented on HADOOP-15808: -- Okay, very good, [~ste...@apache.org]. +1 on the patch as it LGTM. > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717595#comment-16717595
]
Steve Loughran commented on HADOOP-15808:
-
bq. I assume once this gets in a patch for all of the consumers that will need
to accommodate null will created?
they are already broken that way, I'm just making sure that a failure to load
one specific token isn't immediately fatal
BTW: stack trace surfaces in YARN RM renewer thread as fatal for the RM
{code}
Exception: java.lang.NoClassDefFoundError thrown from the
UncaughtExceptionHandler in thread "Timer-5"
{code}
I am going to review the javadocs. Do you have any other issues or any
alternative strategy towards improving Hadoop token resilience to service
loader problems
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch,
> HADOOP-15808-003.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717604#comment-16717604 ] Steve Loughran commented on HADOOP-15808: - javadocs are correct. ReflectionUtils.newInstance() raises an RTE The javadocs are simply declaring the current behavior of the system; I am not attempting to change it, which is a more significant issue and one I am not attempting. > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715232#comment-16715232 ] Larry McCay commented on HADOOP-15808: -- [~ste...@apache.org] - what is the difference between no class found and not being able to be instantiated - they are separate exceptions but wondering whether the javadoc is correct in... + * @return the token identifier, or null if there was no class found for it + * @throws IOException failure to unmarshall the data + * @throws RuntimeException if the token class could not be instantiated. is it literally distinguishing between the two - so that a class that is found but cannot be loaded gets a runtime exception and a class not found gets a null returned? I assume once this gets in a patch for all of the consumers that will need to accommodate null will created? > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713171#comment-16713171 ] Steve Loughran commented on HADOOP-15808: - I've seen the exact issue I was predicting surface in the field. We don't need to care about unloadable ID tokens except in the special case "someone is loading a token", where the fact no token is decodable > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701734#comment-16701734
]
Steve Loughran commented on HADOOP-15808:
-
Revisited this to see about changing message for token loading.
I'm actually reluctant to do this because of the following classpath isn't as
rare as you think:
* hadoop-aws on CP, aws-sdk not.
We saw that with the FileSystem service loader, logging warnings created too
many stack traces, issues related to "why am I being told about s3 when I'm
working with HDFS, etc".
I would like to keep this noise out. Now, it will hide the issue "there's an
implementation declared which isn't going to load", but if you are really
looking for a token, the fact it won't load will soon surface. The problem is
that we've lost some of the root cause. Really you'd want to cache the stack
and only print it in a specific failure, e.g when a token can't be decoded
because its kind is unrecognised.
But I can't see an easy way to do that with the current code.
the one thing we could do is cache the fact that 1+ warning was found and then
in {{Token.decodeIdentifier()}}, when there's no match for a token kind, log
that fact.
The problem I have is
* this stuff gets used in a lot of places
* its currently fairly brittle code: one bad class and *nothing* will load.
* A patch to really harden this stuff with null checks everywhere didn't take
I'm confident that the current design is going to fail badly once HADOOP-14556
goes in. I've looked to see if I can get away with making sure the token
identifier has no indirect refs to any AWS libraries, but it's a doomed
exercise: no way to test, and it'll inevitably regress.
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch,
> HADOOP-15808-003.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683781#comment-16683781 ] Steve Loughran commented on HADOOP-15808: - bq. I don't fully understand why the Iterator#next() triggers the exception but service loading has lazy behaviors so I guess that must be it. Exactly. They build a list of classes to load, but the load is only in the next() call. And ~everything which uses the service loader API doesn't expect failures here. w.r.t those test options # Create a fake type that triggers this by not having the dependencies. # Spy some of the types to trigger the exception always. I thought of #1 but felt it was too risky; imagine if it was in hadoop-common-test and somehow that JAR ended up on the CP of a hadoop version which didn't have the hardening. You'd not be able to load any tokens. The other option would be factor out the service loading into some templated class for use everywhere: you pass in the details needed for the load, get back the list of successfully loaded classes. We could also take an arg about log level of errors (debug, info, error). For FS loading, debug is all we need. With this factored out, the test would be to load some explicitly failing service purely for hadoop-common-test, so it wouldn't interfere with token loading -and we'd have something we knew worked everywhere. > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669031#comment-16669031
]
Íñigo Goiri commented on HADOOP-15808:
--
I have to say that the pattern of iterating and then catching exceptions is
much uglier than just using the {{for(:)}} approach.
I've been trying to see if there's any way to catch those in the iterator but
there's nothing really clean.
I guess we'll have to live with this.
I don't fully understand why the {{Iterator#next()}} triggers the exception but
service loading has lazy behaviors so I guess that must be it.
Regarding the exception output, given that right now, it just crashes the whole
thing, I think that going to debug (which is pretty much swallowing it), might
be a little too much.
I would log errors for those cases.
In any case, I think we can add some coverage to these cases, not sure what's
the cleanest way to trigger {{ServiceConfigurationError}}; a few options I can
think:
* Create a fake type that triggers this by not having the dependencies.
* Spy some of the types to trigger the exception always.
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch,
> HADOOP-15808-003.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638392#comment-16638392
]
Hadoop QA commented on HADOOP-15808:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 107 unchanged - 3 fixed = 107 total (was 110) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
1s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HADOOP-15808 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942375/HADOOP-15808-003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 013e363ef647 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 81f635f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15289/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15289/testReport/ |
| Max. process+thread count | 1765 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638122#comment-16638122 ] Steve Loughran commented on HADOOP-15808: - patch 003; removed that now unused import > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638112#comment-16638112
]
Steve Loughran commented on HADOOP-15808:
-
checkstyle
{code}
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java:21:import
com.google.common.base.Preconditions;:8: Unused import -
com.google.common.base.Preconditions. [UnusedImports]
{code}
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637263#comment-16637263
]
Hadoop QA commented on HADOOP-15808:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 107 unchanged - 3 fixed = 108 total (was 110) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
48s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HADOOP-15808 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942276/HADOOP-15808-002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c2df4ab07b5e 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7051bd7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15284/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15284/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15284/testReport/ |
| Max. process+thread count | 1747 (vs. ulimit of 1) |
| modules
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637017#comment-16637017 ] Steve Loughran commented on HADOOP-15808: - HADOOP-15808: patch 002: remove state checks for no class found on decode; leave callers to deal with it. That's not ideal, as most of the code doesn't bother to handle decoded token == null, but its clear that some bits of code does, and is happy for things to return null. Someone would need to go through every single use of decodeIdentifier and review/patch it, which is beyond the scope of this patch. * Improve error text on TestSaslRPC failures (used in tracking down problem) * fix checkstyle. > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636928#comment-16636928
]
Steve Loughran commented on HADOOP-15808:
-
Checkstyle is minor, test failure not.
{code}
[ERROR] TestSaslRPC.testKerberosServer:692->assertAuthEquals:929
[ERROR] TestSaslRPC.testKerberosServer:692->assertAuthEquals:929
[ERROR] TestSaslRPC.testKerberosServer:692->assertAuthEquals:929
[ERROR] TestSaslRPC.testKerberosServer:692->assertAuthEquals:929
[ERROR] TestSaslRPC.testKerberosServer:692->assertAuthEquals:929
[ERROR] TestSaslRPC.testNoClientFallbackToSimple:575->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testNoClientFallbackToSimple:575->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testNoClientFallbackToSimple:575->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testNoClientFallbackToSimple:575->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testNoClientFallbackToSimple:575->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServerWithTokens:622->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServerWithTokens:622->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServerWithTokens:622->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServerWithTokens:622->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServerWithTokens:622->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServer:563->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServer:563->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServer:563->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServer:563->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testSimpleServer:563->assertAuthEquals:923
expected:<[SIMPLE]> but was:<[java.lang.IllegalStateException: Unknown/Unloaded
token identifier for token kind ]>
[ERROR] TestSaslRPC.testTokenOnlyServer:661->assertAuthEquals:929
[ERROR] TestSaslRPC.testTokenOnlyServer:661->assertAuthEquals:929
[ERROR] TestSaslRPC.testTokenOnlyServer:661->assertAuthEquals:929
[ERROR] TestSaslRPC.testTokenOnlyServer:661->assertAuthEquals:929
[ERROR] TestSaslRPC.testTokenOnlyServer:661->assertAuthEquals:929
{code}
I'm going to conclude that making the existing token check code check for
null-nes breaks those few bits of code which actually expect it, and that a
stricter method is going to have to go in, even if it's just some utility
wrapper we can put around 95% of uses of decodeIdentifier in the hadoop code
itself
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15808-001.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634669#comment-16634669
]
Hadoop QA commented on HADOOP-15808:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 9 unchanged - 3 fixed = 10 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 58s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestSaslRPC |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HADOOP-15808 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942024/HADOOP-15808-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c25450a2cc29 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cc80ac2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15272/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15272/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15272/testReport/ |
| Max. process+thread count | 1470 (vs. ulimit of
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634539#comment-16634539
]
Steve Loughran commented on HADOOP-15808:
-
Patch 001
* harden service loading in Token clases & tools.
* Also tweak logging and "modernize" the tests.
No extra tests of service loading as the only way to do it would be to register
faulty service implementations in the test JAR, and that could unintentionally
break other things. Not this hardened code, obviously, but any third party code
or a situation where the hadoop-common-test JAR from this version of hadoop
ended up on the same classpath as hadoop-common.
Please review carefully, looking at {{FileSystem.loadFileSystems()}} as the
source of this design
+[~daryn] [~raviprak]
BTW, noticed there's no way in the existing dtutail code to retrieve DTs for
arbitrary filesystems, though hdfs fetchdt can do this. Unless a generic FS
loader can be added, the command is going to need a registered service for all
the filesystems which export DTs (abfs, wasb, now s3a). They can share the same
reference impl, just need to return a service name which matches the URI schema
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15808-001.patch
>
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[
https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634435#comment-16634435
]
Steve Loughran commented on HADOOP-15808:
-
There's a related issue, which is most (all?) uses of
{{Token.decodeIdentifier()}} assume that the return value is never null, but it
can be if the Kind of the token isn't known. This will trigger an NPE.
Proposed:
* have decodeIdentifier fail with a useful error, including the token kind,
raise an IllegalStateException
* review uses of the code to see if there is anything which does actually look
out for null, tune it for the new world
It'd be nice if we correlate a failure to load a class with an unknown token
identifier impl later, but we can't because you can't determine the kind of an
implementation if that load doesn't work. Pity.
> Harden Token service loader use
> ---
>
> Key: HADOOP-15808
> URL: https://issues.apache.org/jira/browse/HADOOP-15808
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.1, 3.1.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> The Hadoop token service loading (identifiers, renewers...) works provided
> there's no problems loading any registered implementation. If there's a
> classloading or classcasting problem, the exception raised will stop all
> token support working; possibly the application not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on
> the classpath. It probably lurks in the wasb/abfs support too, but things
> have worked there because the installations with DT support there have always
> had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate
> a service provider impl and skip it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
