The -Xms512m -Xmx512m is coming from $SPARK_MEM: https://github.com/apache/incubator-spark/blob/v0.8.0-incubating/spark-class#L88
You could use SPARK_MEM to adjust the heap size, but that won't let you set separate minimum and maximum sizes. On Wed, Oct 23, 2013 at 2:15 PM, <[email protected]> wrote: > using spark 0.8.0**** > > ** ** > > I have this line in my spark-env.sh**** > > ** ** > > SPARK_JAVA_OPTS+=" -Dspark.local.dir=/tmp/spark -XX:+UseParallelGC > -XX:+UseParallelOldGC -XX:+DisableExplicitGC -Xms1024m -Xmx2048m > -XX:MaxPermSize=256m"**** > > export SPARK_JAVA_OPTS**** > > ** ** > > logs shows it used but also -Xms512m -Xmx512m, how I can remove any > reference to use -Xms512m -Xmx512m when I am setting explicitly -Xms1024m > -Xmx2048m in my spark-env.sh ?**** > > ** ** > > Spark Command: /opt/jdk1.7.0_40/bin/java -cp > :/opt/spark-0.8.0/conf:/opt/spark-0.8.0/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar > -Dspark.local.dir=/tmp/spark -XX:+UseParallelGC -XX:+UseParallelOldGC > -XX:+DisableExplicitGC -Xms1024m -Xmx2048m -XX:MaxPermSize=256m > -Dspark.local.dir=/tmp/spark -XX:+UseParallelGC -XX:+UseParallelOldGC > -XX:+DisableExplicitGC -Xms1024m -Xmx2048m -XX:MaxPermSize=256m > -Dspark.local.dir=/tmp/spark -XX:+UseParallelGC -XX:+UseParallelOldGC > -XX:+DisableExplicitGC -Xms1024m -Xmx2048m -XX:MaxPermSize=256m > -Djava.library.path= -Xms512m -Xmx512m > org.apache.spark.deploy.master.Master --ip poc1.stc1lab.local --port 7077 > --webui-port 8080**** > > ** ** > > ========================================**** > > ** ** > > if I set in my spark-env.sh**** > > export SPARK_JAVA_OPTS=" -Dspark.local.dir=/tmp/spark -XX:+UseParallelGC > -XX:+UseParallelOldGC -XX:+DisableExplicitGC -Xms1024m -Xmx2048m > -XX:MaxPermSize=256m"**** > > ** ** > > logs shows:**** > > ** ** > > Spark Command: /opt/jdk1.7.0_40/bin/java -cp > :/opt/spark-0.8.0/conf:/opt/spark-0.8.0/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar > -Djava.library.path= -Xms512m -Xmx512m > org.apache.spark.deploy.master.Master --ip poc1.stc1lab.local --port 7077 > --webui-port 8080**** > > ========================================**** > > ** ** > > can someone please update on the correct way of setting SPARK_JAVA_OPTS > in spark-env.sh?**** > > ** ** > > Thanks,**** > > ** ** > > ** ** >
