Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1845#discussion_r16496323
--- Diff: bin/spark-class ---
@@ -146,10 +155,28 @@ if $cygwin; then
fi
export CLASSPATH
-if [ "$SPARK_PRINT_LAUNCH_COMMAND" == "1" ]; then
- echo -n "Spark Command: " 1>&2
- echo "$RUNNER" -cp "$CLASSPATH" $JAVA_OPTS "$@" 1>&2
- echo -e "========================================\n" 1>&2
+# In Spark submit client mode, the driver is launched in the same JVM as
Spark submit itself.
+# Here we must parse the properties file for relevant "spark.driver.*"
configs before launching
+# the driver JVM itself. Instead of handling this complexity in Bash, we
launch a separate JVM
+# to prepare the launch environment of this driver JVM.
+
+if [ -n "$SPARK_SUBMIT_BOOTSTRAP_DRIVER" ]; then
+ # This is used only if the properties file actually contains these
special configs
+ # Export the environment variables needed by
SparkSubmitDriverBootstrapper
+ export RUNNER
+ export CLASSPATH
+ export JAVA_OPTS
+ export OUR_JAVA_MEM
+ export SPARK_CLASS=1
+ shift # Ignore main class and use our own
--- End diff --
Just so I understand - is this only ever used when the main class is
`SparkSubmit`? It might be good to say that in the comment. The way it is now
makes it sound like we could substitute a different class from the one that was
specified, but in fact, all this does is ignore the class here and we hard code
it in the bootstrapper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
