Can you check if this JIRA is relevant? https://issues.apache.org/jira/browse/SPARK-2608
If not, can you make a new one? On Thu, Oct 27, 2016 at 10:27 PM, Rodrick Brown <rodr...@orchard-app.com> wrote: > Try setting the values in $SPARK_HOME/conf/spark-defaults.conf > > i.e. > > $ egrep 'spark.(driver|executor).extra' /data/orchard/spark-2.0.1/ > conf/spark-defaults.conf > spark.executor.extraJavaOptions -Duser.timezone=UTC > -Xloggc:garbage-collector.log > spark.driver.extraJavaOptions -Duser.timezone=UTC > -Xloggc:garbage-collector.log > > -- > > [image: Orchard Platform] <http://www.orchardplatform.com/> > > Rodrick Brown / DevOPs Engineer > +1 917 445 6839 / rodr...@orchardplatform.com > <char...@orchardplatform.com> > > Orchard Platform > 101 5th Avenue, 4th Floor, New York, NY 10003 > http://www.orchardplatform.com > > Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending > Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/> > > On Oct 6, 2016, at 12:20 PM, vonnagy <i...@vadio.com> wrote: > > I am trying to submit a job to spark running in a Mesos cluster. We need to > pass custom java options to the driver and executor for configuration, but > the driver task never includes the options. Here is an example submit. > > GC_OPTS="-XX:+UseConcMarkSweepGC > -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:$appdir/gc.out > -XX:MaxPermSize=512m > -XX:+CMSClassUnloadingEnabled " > > EXEC_PARAMS="-Dloglevel=DEBUG -Dkafka.broker-address=${KAFKA_ADDRESS} > -Dredis.master=${REDIS_MASTER} -Dredis.port=${REDIS_PORT} > > spark-submit \ > --name client-events-intake \ > --class ClientEventsApp \ > --deploy-mode cluster \ > --driver-java-options "${EXEC_PARAMS} ${GC_OPTS}" \ > --conf "spark.ui.killEnabled=true" \ > --conf "spark.mesos.coarse=true" \ > --conf "spark.driver.extraJavaOptions=${EXEC_PARAMS}" \ > --conf "spark.executor.extraJavaOptions=${EXEC_PARAMS}" \ > --master mesos://someip:7077 \ > --verbose \ > some.jar > > When the driver task runs in Mesos it is creating the following command: > > sh -c 'cd spark-1*; bin/spark-submit --name client-events-intake --class > ClientEventsApp --master mesos://someip:5050 --driver-cores 1.0 > --driver-memory 512M ../some.jar ' > > There are no options for the driver here, thus the driver app blows up > because it can't find the java options. However, the environment variables > contain the executor options: > > SPARK_EXECUTOR_OPTS -> -Dspark.executor.extraJavaOptions=-Dloglevel=DEBUG > ... > > Any help would be great. I know that we can set some "spark.*" settings in > default configs, but these are not necessarily spark related. This is not > an > issue when running the same logic outside of a Mesos cluster in Spark > standalone mode. > > Thanks! > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Submit-job-with-driver-options-in- > Mesos-Cluster-mode-tp27853.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > > > *NOTICE TO RECIPIENTS*: This communication is confidential and intended > for the use of the addressee only. If you are not an intended recipient of > this communication, please delete it immediately and notify the sender by > return email. Unauthorized reading, dissemination, distribution or copying > of this communication is prohibited. This communication does not constitute > an offer to sell or a solicitation of an indication of interest to purchase > any loan, security or any other financial product or instrument, nor is it > an offer to sell or a solicitation of an indication of interest to purchase > any products or services to any persons who are prohibited from receiving > such information under applicable law. The contents of this communication > may not be accurate or complete and are subject to change without notice. > As such, Orchard App, Inc. (including its subsidiaries and affiliates, > "Orchard") makes no representation regarding the accuracy or completeness > of the information contained herein. The intended recipient is advised to > consult its own professional advisors, including those specializing in > legal, tax and accounting matters. Orchard does not provide legal, tax or > accounting advice. > -- Michael Gummelt Software Engineer Mesosphere
