ScrapCodes commented on a change in pull request #25609: [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time URL: https://github.com/apache/spark/pull/25609#discussion_r387587996
########## File path: docs/security.md ########## @@ -845,8 +845,13 @@ When talking to Hadoop-based services behind Kerberos, it was noted that Spark n so that non-local processes can authenticate. These delegation tokens in Kubernetes are stored in Secrets that are shared by the Driver and its Executors. As such, there are three ways of submitting a Kerberos job: -In all cases you must define the environment variable: `HADOOP_CONF_DIR` or -`spark.kubernetes.hadoop.configMapName.` +In all cases you must define the environment variable: `HADOOP_CONF_DIR` or/and +`spark.kubernetes.hadoop.configMapName`. When only `HADOOP_CONF_DIR` is configured, the hadoop configurations in there +will be used by the client process, the Driver and its Executors. When only `spark.kubernetes.hadoop.configMapName` +is configured, the hadoop configurations will only be used by the Driver and its Executors. If your client process has +extra dependencies to upload to `spark.kubernetes.file.upload.path`, you may need to configure `HADOOP_CONF_DIR` too. +When these two variables are both set, Spark will prefer `spark.kubernetes.hadoop.configMapName` to be mounted on the +Driver/Executor pods. Review comment: I think it is still okay to warn the user about, the configuration is being picked from the configMap, inspite of the fact, HADOOP_CONF_DIR is defined. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
