[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605301#comment-17605301
]
ASF GitHub Bot commented on YARN-11255:
---------------------------------------
ashutoshcipher commented on PR #4884:
URL: https://github.com/apache/hadoop/pull/4884#issuecomment-1248022535
> Thanks @ashutoshcipher for the patch. Overall looks good to me except a
minor one.
>
> Could you also document the steps to run an application like Spark using
this new environment variable YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG in
https://hadoop.apache.org/docs/r3.0.2/hadoop-yarn/hadoop-yarn-site/DockerContainers.html
>
> For Example:
>
> ```
> DOCKER_IMAGE_NAME=hadoop-docker
> DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
> spark-submit --master yarn \
> --deploy-mode cluster \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
> sparkR.R
> ```
Thanks @PrabhuJoseph . Added the documentation
> Support loading alternative docker client config from system environment
> ------------------------------------------------------------------------
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Ashutosh Gupta
> Assignee: Ashutosh Gupta
> Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
