hey patrick, i have a SparkConf i can add them too. i was looking for a way to do this where they are not hardwired within scala, which is what SPARK_JAVA_OPTS used to do. i guess if i just set -Dspark.akka.frameSize=10000 on my java app launch then it will get picked up by the SparkConf too right?
On Wed, May 14, 2014 at 2:54 PM, Patrick Wendell <pwend...@gmail.com> wrote: > Just wondering - how are you launching your application? If you want > to set values like this the right way is to add them to the SparkConf > when you create a SparkContext. > > val conf = new SparkConf().set("spark.akka.frameSize", > "10000").setAppName(...).setMaster(...) > val sc = new SparkContext(conf) > > - Patrick > > On Wed, May 14, 2014 at 9:09 AM, Koert Kuipers <ko...@tresata.com> wrote: > > i have some settings that i think are relevant for my application. they > are > > spark.akka settings so i assume they are relevant for both executors and > my > > driver program. > > > > i used to do: > > SPARK_JAVA_OPTS="-Dspark.akka.frameSize=10000" > > > > now this is deprecated. the alternatives mentioned are: > > * some spark-submit settings which are not relevant to me since i do not > use > > spark-submit (i launch spark jobs from an existing application) > > * spark.executor.extraJavaOptions to set -X options. i am not sure what > -X > > options are, but it doesnt sound like what i need, since its only for > > executors > > * SPARK_DAEMON_OPTS to set java options for standalone daemons (i.e. > master, > > worker), that sounds like i should not use it since i am trying to change > > settings for an app, not a daemon. > > > > am i missing the correct setting to use? > > should i do -Dspark.akka.frameSize=10000 on my application launch > directly, > > and then also set spark.executor.extraJavaOptions? so basically repeat > it? >