. J. Watson Research Center
http://researcher.ibm.com/person/us-wtan
From: Aaron Davidson ilike...@gmail.com
To: user@spark.apache.org,
Date: 06/15/2014 09:06 PM
Subject:Re: long GC pause during file.cache()
Note also that Java does not work well with very large JVMs due
Research Staff Member
IBM T. J. Watson Research Center
http://researcher.ibm.com/person/us-wtan
From: Wei Tan/Watson/IBM@IBMUS
To: user@spark.apache.org,
Date: 06/16/2014 10:47 AM
Subject:Re: long GC pause during file.cache()
Thanks you all for advice including (1) using CMS GC (2
Hi, Wei
You may try to set JVM opts in *spark-env.sh* as follow to prevent or
mitigate GC pause:
export SPARK_JAVA_OPTS=-XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC
-Xmx2g -XX:MaxPermSize=256m
There are more options you could add, please just Google :)
Regards,
Wang Hao(王灏)
CloudTeam |
SPARK_JAVA_OPTS is deprecated in 1.0, though it works fine if you don’t mind
the WARNING in the logs
you can set spark.executor.extraJavaOpts in your SparkConf obj
Best,
--
Nan Zhu
On Sunday, June 15, 2014 at 12:13 PM, Hao Wang wrote:
Hi, Wei
You may try to set JVM opts in
Is SPARK_DAEMON_JAVA_OPTS valid in 1.0.0?
On Sun, Jun 15, 2014 at 4:59 PM, Nan Zhu zhunanmcg...@gmail.com wrote:
SPARK_JAVA_OPTS is deprecated in 1.0, though it works fine if you
don’t mind the WARNING in the logs
you can set spark.executor.extraJavaOpts in your SparkConf obj
Best,
--
Yes, I think in the spark-env.sh.template, it is listed in the comments (didn’t
check….)
Best,
--
Nan Zhu
On Sunday, June 15, 2014 at 5:21 PM, Surendranauth Hiraman wrote:
Is SPARK_DAEMON_JAVA_OPTS valid in 1.0.0?
On Sun, Jun 15, 2014 at 4:59 PM, Nan Zhu
Note also that Java does not work well with very large JVMs due to this
exact issue. There are two commonly used workarounds:
1) Spawn multiple (smaller) executors on the same machine. This can be done
by creating multiple Workers (via SPARK_WORKER_INSTANCES in standalone
mode[1]).
2) Use Tachyon
Hi,
I have a single node (192G RAM) stand-alone spark, with memory
configuration like this in spark-env.sh
SPARK_WORKER_MEMORY=180g
SPARK_MEM=180g
In spark-shell I have a program like this:
val file = sc.textFile(/localpath) //file size is 40G
file.cache()
val output = file.map(line =
