Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21426#discussion_r191491127
--- Diff: core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala ---
@@ -153,4 +154,30 @@ object PythonRunner {
.map { p => formatPath(p, testWindows) }
}
+ /**
+ * Resolves the ".py" files. ".py" file should not be added as is
because PYTHONPATH does
+ * not expect a file. This method creates a temporary directory and puts
the ".py" files
+ * if exist in the given paths.
+ */
+ private def resolvePyFiles(pyFiles: Array[String]): Array[String] = {
+ val dest = Utils.createTempDir(namePrefix = "localPyFiles")
+ pyFiles.flatMap { pyFile =>
+ // In case of client with submit, the python paths should be set
before context
+ // initialization because the context initialization can be done
later.
+ // We will copy the local ".py" files because ".py" file shouldn't
be added
+ // alone but its parent directory in PYTHONPATH. See SPARK-24384.
+ if (pyFile.endsWith(".py")) {
+ val source = new File(pyFile)
+ if (source.exists() && source.canRead) {
--- End diff --
I think providing a non-existent file to spark-submit should result in an
error. Whether the error happens here or somewhere else it doesn't really
matter.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
