Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20059#discussion_r158768975
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,54 @@ by their appropriate remote URIs. Also, application
dependencies can be pre-moun
Those dependencies can be added to the classpath by referencing them with
`local://` URIs and/or setting the
`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like
HDFS or HTTP servers, the driver and executor pods
+need a Kubernetes
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
for downloading
+the dependencies so the driver and executor containers can use them
locally. This requires users to specify the container
+image for the init-container using the configuration property
`spark.kubernetes.initContainer.image`. For example, users
+simply add the following option to the `spark-submit` command to specify
the init-container image:
+
+```
+--conf spark.kubernetes.initContainer.image=<init-container image>
+```
+
+The init-container handles remote dependencies specified in `spark.jars`
(or the `--jars` option of `spark-submit`) and
+`spark.files` (or the `--files` option of `spark-submit`). It also handles
remotely hosted main application resources, e.g.,
+the main application jar. The following shows an example of using remote
dependencies with the `spark-submit` command:
+
+```bash
+$ bin/spark-submit \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+ --deploy-mode cluster \
+ --name spark-pi \
+ --class org.apache.spark.examples.SparkPi \
+ --jars https://path/to/dependency1.jar,https://path/to/dependency2.jar
+ --files hdfs://host:port/path/to/file1,hdfs://host:port/path/to/file2
+ --conf spark.executor.instances=5 \
+ --conf spark.kubernetes.driver.docker.image=<driver-image> \
+ --conf spark.kubernetes.executor.docker.image=<executor-image> \
--- End diff --
Done.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
