Thank you for your reply.
I'm sorry confirmation is slow.
I'll try the tuning 'spark.yarn.executor.memoryOverhead'.
Thanks,
Yuichiro Sakamoto
On 2015/03/25 0:56, Sandy Ryza wrote:
Hi Yuichiro,
The way to avoid this is to boost spark.yarn.executor.memoryOverhead until the
executors have enough off-heap memory to avoid going over their limits.
-Sandy
On Tue, Mar 24, 2015 at 11:49 AM, Yuichiro Sakamoto <ks...@muc.biglobe.ne.jp
<mailto:ks...@muc.biglobe.ne.jp>> wrote:
Hello.
We use ALS(Collaborative filtering) of Spark MLlib on YARN.
Spark version is 1.2.0 included CDH 5.3.1.
1,000,000,000 records(5,000,000 users data and 5,000,000 items data) are
used for machine learning with ALS.
These large quantities of data increases virtual memory usage,
node manager of YARN kills Spark worker process.
Even though Spark run again after killing process, Spark worker process is
killed again.
As a result, the whole Spark processes are terminated.
# Spark worker process is killed, it seems that virtual memory usage
increased by
# 'Shuffle' or 'Disk writing' gets over the threshold of YARN.
To avoid such a case from occurring, we use the method that
'yarn.nodemanager.vmem-check-enabled' is false, then exit successfully.
But it does not seem to have an appropriate way.
If you know, please let me know about tuning method of Spark.
The conditions of machines and Spark settings are as follows.
1)six machines, physical memory is 32GB of each machine.
2)Spark settings
- spark.executor.memory=16g
- spark.closure.serializer=org.apache.spark.serializer.KryoSerializer
- spark.rdd.compress=true
- spark.shuffle.memoryFraction=0.4
Thanks,
Yuichiro Sakamoto
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-being-killed-by-YARN-node-manager-tp22199.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
<mailto:user-unsubscr...@spark.apache.org>
For additional commands, e-mail: user-h...@spark.apache.org
<mailto:user-h...@spark.apache.org>
--
*=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=*
Yuichiro SAKAMOTO
- ks...@muc.biglobe.ne.jp
- phonypian...@gmail.com
- http://www2u.biglobe.ne.jp/~yuichi/
*=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=*
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
