Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21267#discussion_r187274825
--- Diff: python/pyspark/context.py ---
@@ -211,9 +211,23 @@ def _do_init(self, master, appName, sparkHome,
pyFiles, environment, batchSize,
for path in self._conf.get("spark.submit.pyFiles", "").split(","):
if path != "":
(dirname, filename) = os.path.split(path)
- if filename[-4:].lower() in self.PACKAGE_EXTENSIONS:
- self._python_includes.append(filename)
- sys.path.insert(1,
os.path.join(SparkFiles.getRootDirectory(), filename))
+ try:
+ filepath = os.path.join(SparkFiles.getRootDirectory(),
filename)
+ if not os.path.exists(filepath):
+ # In case of YARN with shell mode,
'spark.submit.pyFiles' files are
+ # not added via SparkContext.addFile. Here we
check if the file exists,
+ # try to copy and then add it to the path. See
SPARK-21945.
+ shutil.copyfile(path, filepath)
--- End diff --
Yup, that's only missing on driver side in this mode specifically. Yarn
doesn't add it since `spark.files` is not set if I understood correctly. They
are specially handled in case of submit but shell case seems missing.
I described a bit in the PR description too.
> In case of Yarn client and cluster with submit, these are manually being
handled. In particular #6360 added most of the logics. In this case, the Python
path looks manually set via, for example, deploy.PythonRunner. We don't use
spark.files here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
