[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605097#comment-17605097
]
ASF GitHub Bot commented on YARN-11255:
---------------------------------------
PrabhuJoseph commented on PR #4884:
URL: https://github.com/apache/hadoop/pull/4884#issuecomment-1247563160
Thanks @ashutoshcipher for the patch. Overall looks good to me except a
minor one.
Could you also document the steps to run an application like Spark using
this new environment variable YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG in
https://hadoop.apache.org/docs/r3.0.2/hadoop-yarn/hadoop-yarn-site/DockerContainers.html
For Example:
```
DOCKER_IMAGE_NAME=hadoop-docker
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
sparkR.R
```
> Support loading alternative docker client config from system environment
> ------------------------------------------------------------------------
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Ashutosh Gupta
> Assignee: Ashutosh Gupta
> Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
