Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1770#discussion_r15907357
--- Diff: bin/spark-submit ---
@@ -36,9 +38,49 @@ while (($#)); do
done
DEPLOY_MODE=${DEPLOY_MODE:-"client"}
+PROPERTIES_FILE=${PROPERTIES_FILE:-"$SPARK_HOME/conf/spark-defaults.conf"}
-if [ -n "$DRIVER_MEMORY" ] && [ $DEPLOY_MODE == "client" ]; then
- export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
+# For client mode, the driver will be launched in the JVM that launches
+# SparkSubmit, so we need to handle the class paths, java options, and
+# memory pre-emptively in bash. Otherwise, it will be too late by the
+# time the JVM has started.
+
+if [ $DEPLOY_MODE == "client" ]; then
+ # We parse the default properties file here, assuming each line is
+ # a key value pair delimited either by white space or "=" sign. All
+ # spark.driver.* configs must be processed now before it's too late.
+ if [ -f "$PROPERTIES_FILE" ]; then
+ echo "Using properties file $PROPERTIES_FILE." 1>&2
+
+ # Parse the value of the given config
+ # This removes the "=" sign, whitespace, and double quotes around the
value (if any)
+ parse_config() {
+ result=$( \
+ sed "/^#/ d" "$PROPERTIES_FILE" | \
+ grep "$1" | \
+ sed "s/$1//g" | \
--- End diff --
I'm not sure you want `/g` here, in the unlikely case that `$1` is part of
the option's value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
