Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/21092#discussion_r186244105
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala
---
@@ -63,10 +67,17 @@ private[spark] case class KubernetesConf[T <:
KubernetesRoleSpecificConf](
.map(str => str.split(",").toSeq)
.getOrElse(Seq.empty[String])
- def sparkFiles(): Seq[String] = sparkConf
- .getOption("spark.files")
- .map(str => str.split(",").toSeq)
- .getOrElse(Seq.empty[String])
+ def pyFiles(): Option[String] = sparkConf
+ .get(KUBERNETES_PYSPARK_PY_FILES)
+
+ def pySparkMainResource(): Option[String] = sparkConf
--- End diff --
This seems redundant with the driver specific spark conf's MainAppResource.
Perhaps remove the need to specify this thing twice?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
