Github user koertkuipers commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-49559153
on the command line i can get this to work now, but its still way beyond my
bash skills to use "exec spark-submit" inside a script with multiple java
options. this is a fairly normal use case i imagine if you want to launch an
application using spark-submit.
for example my attempt to launch SparkPi with multiple java options is
below. when i run it i get:
$ ./pi
spark-submit --class org.apache.spark.examples.SparkPi
--driver-java-options "-Dfoo=a -Dbar=b"
/usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
Error: Could not find or load main class "-Dfoo=a
i have no doubt its me that is doing something wrong here, but it shouldnt
be this hard to use... perhaps a different syntax would make life easier?
best, koert
$ more pi
#!/bin/bash -e
DRIVER_JAVA_OPTS="-Dfoo=a -Dbar=b"
SPARK_SUBMIT="spark-submit"
OPTS=""
OPTS+=" --class org.apache.spark.examples.SparkPi"
OPTS+=" --driver-java-options \"${DRIVER_JAVA_OPTS}\""
echo $SPARK_SUBMIT $OPTS
/usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
exec $SPARK_SUBMIT $OPTS
/usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
