Re: spark-submit memory too larger

2014-10-25 Thread marylucy 
Version: spark 1.1.0
42 workers,40g memory per worker 
Running graphx componentgraph ,use five hours

 在 Oct 25, 2014，1:27，Sameer Farooqui same...@databricks.com 写道：
 
 That does seem a bit odd. How many Executors are running under this Driver?
 
 Does the spark-submit process start out using ~60GB of memory right away or 
 does it start out smaller and slowly build up to that high? If so, how long 
 does it take to get that high?
 
 Also, which version of Spark are you using? 
 
 
 SameerF
 
 On Fri, Oct 24, 2014 at 8:07 AM, marylucy qaz163wsx_...@hotmail.com wrote:
 i used standalone spark，set spark.driver.memory=5g,but spark-submit process 
 use 57g memory, is this normal?how to decrease it?

spark-submit memory too larger

2014-10-24 Thread marylucy 
i used standalone spark，set spark.driver.memory=5g,but spark-submit process use 
57g memory, is this normal?how to decrease it?

Re: spark-submit memory too larger

2014-10-24 Thread Sameer Farooqui 
That does seem a bit odd. How many Executors are running under this Driver?

Does the spark-submit process start out using ~60GB of memory right away or
does it start out smaller and slowly build up to that high? If so, how long
does it take to get that high?

Also, which version of Spark are you using?


SameerF

On Fri, Oct 24, 2014 at 8:07 AM, marylucy qaz163wsx_...@hotmail.com wrote:

 i used standalone spark，set spark.driver.memory=5g,but spark-submit
 process use 57g memory, is this normal?how to decrease it?