[ https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Gupta updated YARN-11255: ---------------------------------- Description: When using YARN docker support, although the hadoop shell supported {code:java} -docker_client_config{code} to pass the client config file that contains security token to generate the docker config for each job as a temporary file. For other applications that submit jobs to YARN, e.g. Spark, which loads the docker setting via system environment e.g. {code:java} spark.executorEnv.* {code} will not be able to add those authorization token because this system environment isn't considered in YARN. Add genetic solution to handle these kind of cases without making changes in spark code or others Eg When using remote container registry, the {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json file containing the credentials used to authenticate. {code:java} DOCKER_IMAGE_NAME=hadoop-docker DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json spark-submit --master yarn \ --deploy-mode cluster \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \ sparkR.R{code} was: When using YARN docker support, although the hadoop shell supported {code:java} -docker_client_config{code} to pass the client config file that contains security token to generate the docker config for each job as a temporary file. For other applications that submit jobs to YARN, e.g. Spark, which loads the docker setting via system environment e.g. {code:java} spark.executorEnv.* {code} will not be able to add those authorization token because this system environment isn't considered in YARN. Add genetic solution to handle these kind of cases without making changes in spark code or others Eg When using remote container registry, the {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json file containing the credentials used to authenticate. DOCKER_IMAGE_NAME=hadoop-docker DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json spark-submit --master yarn \ --deploy-mode cluster \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \ sparkR.R > Support loading alternative docker client config from system environment > ------------------------------------------------------------------------ > > Key: YARN-11255 > URL: https://issues.apache.org/jira/browse/YARN-11255 > Project: Hadoop YARN > Issue Type: New Feature > Reporter: Ashutosh Gupta > Assignee: Ashutosh Gupta > Priority: Major > Labels: pull-request-available > > When using YARN docker support, although the hadoop shell supported > {code:java} > -docker_client_config{code} > to pass the client config file that contains security token to generate the > docker config for each job as a temporary file. > For other applications that submit jobs to YARN, e.g. Spark, which loads the > docker setting via system environment e.g. > {code:java} > spark.executorEnv.* {code} > will not be able to add those authorization token because this system > environment isn't considered in YARN. > Add genetic solution to handle these kind of cases without making changes in > spark code or others > Eg > When using remote container registry, the > {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json > file containing the credentials used to authenticate. > {code:java} > DOCKER_IMAGE_NAME=hadoop-docker > DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json > spark-submit --master yarn \ > --deploy-mode cluster \ > --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ > --conf > spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \ > --conf > spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG > \ > --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ > --conf > spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME > \ > --conf > spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG > \ > sparkR.R{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org