[
https://issues.apache.org/jira/browse/OOZIE-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126996#comment-16126996
]
Andras Piros commented on OOZIE-3035:
-------------------------------------
[~gezapeti] I cannot see any {{Master.java}} appearing outside of package
{{org.apache.hadoop.mapred}}.
[~rkanter] you're right, but {{Master.java}} delegates to {{NetUtils.java}}
that is also [*marked as {{@LimitedPrivate
@Unstable}}*|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L62-L63],
so moving {{Master.java}} contents to Oozie might not solve all our problems.
The idea to use {{Master#getMasterPrincipal}} came to me while having a look at
[*Spark's
{{HadoopFSDelegationTokenProvider#getDelegationTokenRenewer}}*|https://github.com/apache/spark/blob/cde64add18dac712c48de0637f1979f1043e333e/core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala#L78]
that does essentially the same. Moreover, I cannot find
{{HadoopAccessorService#getMRDelegationTokenRenewer}}, can you please point to
the source?
[~pbacsko] addressing your review comments as well:
* I will organize the code in a way in one method there is only one {{if}} level
* we can throw an {{ActionExecutorException}} in cases where:
** {{Credentials}} is present but YARN renewer is not found
** {{Credentials}} is present and YARN renewer is also found, but HDFS
delegation token is not found
* when on an insecure cluster, {{Credentials}} won't be set, right? Any other
way to tell 100 % it's an insecure setup?
Waiting for [~rkanter]'s answers to provide a new patch.
> HDFS HA and log aggregation: getting HDFS delegation token from YARN renewer
> within JavaActionExecutor
> ------------------------------------------------------------------------------------------------------
>
> Key: OOZIE-3035
> URL: https://issues.apache.org/jira/browse/OOZIE-3035
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 4.3.0
> Environment: * [*Hadoop 3 alpha
> 4*|https://github.com/apache/hadoop/tree/branch-3.0.0-alpha4]
> * [*HDFS
> HA*|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html]
> * log aggregation turned on
> Reporter: Andras Piros
> Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-3035.001.patch
>
>
> In a secure environment, when both HDFS HA and log aggregation are turned on,
> {{JavaActionExecutor}} is not able to call {{YarnClient#submitApplication}}
> since {{HDFS_DELEGATION_TOKEN}} is missing.
> In those cases we need to get {{HDFS_DELEGATION_TOKEN}} from YARN:
> * get YARN renewer via {{Master#getMasterPrincipal}}
> * get {{HDFS_DELEGATION_TOKEN}} via {{DFSClient#getDelegationToken}}
> * add {{HDFS_DELEGATION_TOKEN}} to {{Credentials}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
