Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/6360#discussion_r32145368
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -490,9 +498,11 @@ private[spark] class ApplicationMaster(
new MutableURLClassLoader(urls, Utils.getContextOrSparkClassLoader)
}
+ var userArgs = args.userArgs
if (args.primaryPyFile != null && args.primaryPyFile.endsWith(".py")) {
- System.setProperty("spark.submit.pyFiles",
- PythonRunner.formatPaths(args.pyFiles).mkString(","))
+ // Second argument is the list of files to add to PYTHONPATH, which
Client.scala already
+ // handles, so it's empty.
--- End diff --
It took a while for me to get this. This is actually referring to the
arguments to `o.a.s.deploy.PythonRunner`'s main method. We need to document
this because others who are not familiar with Spark submit at all will most
certainly get lost around here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
