[
https://issues.apache.org/jira/browse/SPARK-37205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-37205:
-----------------------------
Description: {{mapreduce.job.send-token-conf}} is a useful feature in
Hadoop (see [YARN-5910|https://issues.apache.org/jira/browse/YARN-5910] with
which RM is not required to statically have config for all the secure HDFS
clusters. Currently it only works for MRv2 but it'd be nice if Spark can also
use this feature. I think we only need to pass the config to
{{LaunchContainerContext}} in {{Client.createContainerLaunchContext}}. (was:
{{mapreduce.job.send-token-conf}} is a useful feature in Hadoop (see
[YARN-5910|https://issues.apache.org/jira/browse/YARN-5910] with which RM is
not required to statically have config for all the secure HDFS clusters.
Currently it only works for MRv2 but it'd be nice if Spark can also use this
feature. I think we only need to pass the config to {{LaunchContainerContext}}
before invoking {{NMClient.startContainer}}.)
> Support mapreduce.job.send-token-conf when starting containers in YARN
> ----------------------------------------------------------------------
>
> Key: SPARK-37205
> URL: https://issues.apache.org/jira/browse/SPARK-37205
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Affects Versions: 3.3.0
> Reporter: Chao Sun
> Priority: Major
>
> {{mapreduce.job.send-token-conf}} is a useful feature in Hadoop (see
> [YARN-5910|https://issues.apache.org/jira/browse/YARN-5910] with which RM is
> not required to statically have config for all the secure HDFS clusters.
> Currently it only works for MRv2 but it'd be nice if Spark can also use this
> feature. I think we only need to pass the config to
> {{LaunchContainerContext}} in {{Client.createContainerLaunchContext}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
