Github user foxish commented on a diff in the pull request:
https://github.com/apache/spark/pull/19717#discussion_r154952559
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
---
@@ -119,5 +139,60 @@ private[spark] object Config extends Logging {
"must be a positive integer")
.createWithDefault(10)
+ val WAIT_FOR_APP_COMPLETION =
+ ConfigBuilder("spark.kubernetes.submission.waitAppCompletion")
+ .doc("In cluster mode, whether to wait for the application to finish
before exiting the " +
+ "launcher process.")
+ .booleanConf
+ .createWithDefault(true)
+
+ val REPORT_INTERVAL =
+ ConfigBuilder("spark.kubernetes.report.interval")
+ .doc("Interval between reports of the current app status in cluster
mode.")
+ .timeConf(TimeUnit.MILLISECONDS)
+ .createWithDefaultString("1s")
+
+ private[spark] val JARS_DOWNLOAD_LOCATION =
+ ConfigBuilder("spark.kubernetes.mountDependencies.jarsDownloadDir")
+ .doc("Location to download jars to in the driver and executors. When
using" +
+ " spark-submit, this directory must be empty and will be mounted
as an empty directory" +
+ " volume on the driver and executor pod.")
+ .stringConf
+ .createWithDefault("/var/spark-data/spark-jars")
--- End diff --
We do need this directory to be writable and we can definitely document it.
We are running things as root in the reference images within the container, and
we can instead create and use a more restricted user - but that would imply an
assumption on our end. @ash211, can you comment on your setup?
K8s also offers a way to change on the fly that using
[PodSecurityPolicy](https://kubernetes.io/docs/concepts/policy/pod-security-policy/#users-and-groups),
one can change the `runAsUser` config to run as a particular user. So, one can
create a user "spark" on their nodes and have pod security policy enforce that
all containers run as that user, i.e. our default to root doesn't prevent
customization in that regard.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
