Hi Calvin,
When you say until all the memory in the cluster is allocated and the job
gets killed, do you know what's going on? Spark apps should never be
killed for requesting / using too many resources? Any associated error
message?
Unfortunately there are no tools currently for tweaking the number of
executors in an automated manner. An option to use the entire YARN cluster
seems useful. I just filed a JIRA for it -
https://issues.apache.org/jira/browse/SPARK-3183.
-Sandy
On Tue, Aug 19, 2014 at 12:51 PM, Calvin iphcal...@gmail.com wrote:
I've set up a YARN (Hadoop 2.4.1) cluster with Spark 1.0.1 and I've
been seeing some inconsistencies with out of memory errors
(java.lang.OutOfMemoryError: unable to create new native thread) when
increasing the number of executors for a simple job (wordcount).
The general format of my submission is:
spark-submit \
--master yarn-client \
--num-executors=$EXECUTORS \
--executor-cores 1 \
--executor-memory 2G \
--driver-memory 3G \
count.py intput output
If I run without specifying the number of executors, it defaults to
two (3 containers: 2 executors, 1 driver). Is there any mechanism to
let a spark application scale to the capacity of the YARN cluster
automatically?
Similarly, for low numbers of executors I get what I asked for (e.g.,
10 executors results in 11 containers running, 20 executors results in
21 containers, etc) until a particular threshold... when I specify 50
containers, Spark seems to start asking for more and more containers
until all the memory in the cluster is allocated and the job gets
killed.
I don't understand that particular behavior—if anyone has any
thoughts, that would be great if you could share your experiences.
Wouldn't it be preferable to have Spark stop requesting containers if
the cluster is at capacity rather than kill the job or error out?
Does anyone have any recommendations on how to tweak the number of
executors in an automated manner?
Thanks,
Calvin
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
