Mina, thanks for detailed explanation!
First I went with approach "without SPARK_HOME". I just keep getting OOM
from some threads.
But now I try to do it right way - with SPARK_HOME.
I haven't changed my conf/zeppelin-env.sh at all and my spark-defaults.conf
in SPARK_HOME is:
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.master yarn-client
spark.executor.memory 10g
spark.yarn.executor.memoryOverhead 3000
spark.executor.cores 2
spark.driver.memory 20g
spark.yarn.driver.memoryOverhead 3000
spark.sql.autoBroadcastJoinThreshold 500485760
spark.network.timeout 400s
spark.driver.maxResultSize 3g
spark.akka.frameSize 500
spark.akka.askTimeout 30
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true
spark.dynamicAllocation.minExecutors 5
spark.dynamicAllocation.executorIdleTimeout 100s
spark.dynamicAllocation.cachedExecutorIdleTimeout 300s
spark.scheduler.mode FAIR
I get: "Error occurred during initialization of VM
Incompatible minimum and maximum heap sizes specified" in my
logs/zeppelin-egor-hadoop2.out.
When I remove spark-defaults.conf from SPARK_HOME everything is
working, but with default settings.
What might be the problem?
2015-11-16 7:08 GMT-08:00 Mina Lee <mina...@nflabs.com>:
> Hi Egor,
>
> we recommand you to set SPARK_HOME because many spark configurations can
> be handled better in this way.
> SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh is valid only when you set
> SPARK_HOME.
> Since you get JVM error, if you have any JVM related configuration in
> your conf/zeppelin-env.sh can you share it?
>
> If you want to change driver memory without setting SPARK_HOME, you can
> try ZEPPELIN_INTP_MEM in conf/zeppelin-env.sh file as below:
> export ZEPPELIN_INTP_MEM="-Xmx10g" in conf/zeppelin-env.sh file.
> FYI, Zeppelin's default ZEPPELIN_INTP_MEM is "-Xmx1024m
> -XX:MaxPermSize=512m".
>
> Though Spark's UI won't display "Storage Memory" as 10g, because it will
> shows fraction of java heap to use for memory cache.
>
> I tested both with/without SPARK_HOME setting in yarn-client mode, and
> Spark UI shows different value.
>
> - With SPARK_HOME setting if I export SPARK_SUBMIT_OPTIONS="--driver-momory
> 10g", driver storage memory is 5.2G.
> - Without SPARK_HOME setting if I export ZEPPELIN_INT_MEM="-Xmx10g",
> driver storage memory is 4.8G.
>
> The difference here is that first one starts spark interpreter process by
> running spark_submit and second one runs with java.
>
> Hope this helps!
>
>
>
>
> On Thu, Nov 12, 2015 at 8:41 AM, Egor Pahomov <pahomov.e...@gmail.com>
> wrote:
>
>> I'm trying to configure driver memory size. So far I tied next parameters:
>>
>> -
>>
>> export JAVA_INTP_OPTS="-Xmx10g"
>>
>> -
>>
>> export SPARK_SUBMIT_OPTIONS="--driver-memory 10g --executor-memory 10g"
>>
>> -
>>
>> " -Xmx30000m \
>>
>> -
>>
>> Change SparkInterpreter.java: conf.set("spark.executor.memory", "10g");
>>
>> conf.set("spark.executor.cores", "2");
>> conf.set("spark.driver.memory", "10g");
>> conf.set("spark.shuffle.io.numConnectionsPerPeer", "5");
>> conf.set("spark.sql.autoBroadcastJoinThreshold", "200483647");
>> conf.set("spark.network.timeout", "400s");
>> conf.set("spark.driver.maxResultSize", "3g");
>> conf.set("spark.sql.hive.convertMetastoreParquet", "false");
>> conf.set("spark.kryoserializer.buffer.max", "200m");
>> conf.set("spark.serializer",
>> "org.apache.spark.serializer.KryoSerializer");
>> conf.set("spark.dynamicAllocation.enabled", "true");
>> conf.set("spark.shuffle.service.enabled", "true");
>> conf.set("spark.dynamicAllocation.minExecutors", "1");
>> conf.set("spark.dynamicAllocation.maxExecutors", "30");
>> conf.set("spark.dynamicAllocation.executorIdleTimeout", "60s");
>> //.set("spark.sql.hive.metastore.version", "1.1.0")
>> conf.set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "100s");
>>
>> -
>>
>> I tried setting SPARK_HOME, but it didn't even started, failed with
>> "Incompatible minimum and maximum heap sizes specified"
>>
>>
>> No matter, what I do I get in logs: "INFO [2015-11-11 14:55:24,453]
>> ({sparkDriver-akka.actor.default-dispatcher-14} Logging.scala[logInfo]:59)
>> - Registering block manager 192.168.12.121:45057 with 530.0 MB RAM,
>> BlockManagerId(driver, 192.168.12.121, 45057)" and on my spark UI:
>>
>> ​Has anyone faced this problem or knows what to do?
>>
>> --
>>
>>
>> *Sincerely yoursEgor Pakhomov*
>>
>
>
--
*Sincerely yoursEgor Pakhomov*
