[
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271067#comment-16271067
]
Allen Wittenauer commented on HADOOP-15059:
-------------------------------------------
bq. Yes, and writing that version does enable supporting multiple versions, but
only for software that understands what those version numbers mean.
... and can actually read the file. This makes it more than a cosmetic change.
It's a pretty big deal if one is processing token files with anything other
than Java or both endians. Theoretically, switching from Java serialization to
protobuf should greatly lessen the impact if additions to the token file are
needed. The file format is now something easily portable across a wide variety
of scenarios.
It's probably worth pointing out how well this change goes hand-in-hand with
things such as slider/yarn-native-services. It should enable jobs to directly
interact with the system without needing a 'helper' running next to it.
Let's be clear: this is only a problem if one has a bundled hadoop-common.jar.
Is the end result of this JIRA going to be that all file formats are locked
forever, regardless of where they come from? This seems like a very dark
precedent for the project to take if it ever wants to move forward. [It's
looking more and more like Hadoop will be required to bite the bullet on
log4j2, likely sooner rather than later.]
Also, Hadoop releases have broken rolling upgrade (and non-rolling upgrades,
for that matter) in the middle of the 2.x stream before by removing things such
as container execution types. Where is the actual line? It's clear the
compatibility guidelines aren't really followed/enforced.
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Reporter: Junping Du
> Assignee: Jason Lowe
> Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch,
> HADOOP-15059.003.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_000001
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_000001/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
