Re: How to set Spark executor memory?

> You could set the spark.executor.memory in the code itself . new >>>>> SparkConf()..set("spark.executor.memory", "2g") >>>>> >>>>> Or you can try the -- spark.executor.memory 2g while submitting the >>>>> jar. >>&g

Re: How to set Spark executor memory?

>>> Hi Xi Shen, >>>> >>>> You could set the spark.executor.memory in the code itself . new >>>> SparkConf()..set("spark.executor.memory", "2g") >>>> >>>> Or you can try the -- spark.executor.memory 2g whil

Re: How to set Spark executor memory?

>>> >>> >>> >>> Regards >>> >>> Jishnu Prathap >>> >>> >>> >>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com] >>> *Sent:* Monday, March 16, 2015 2:06 PM >>> *To:* Xi Shen >>>

Re: How to set Spark executor memory?

gt; >> >> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com] >> *Sent:* Monday, March 16, 2015 2:06 PM >> *To:* Xi Shen >> *Cc:* user@spark.apache.org >> *Subject:* Re: How to set Spark executor memory? >> >> >> >> By default spark.executor.me

Re: How to set Spark executor memory?

ory", "2g") >> >> Or you can try the -- spark.executor.memory 2g while submitting the jar. >> >> >> >> Regards >> >> Jishnu Prathap >> >> >> >> From: Akhil Das [mailto:ak...@sigmoidanalytics.com] >> Sent: Monday,

Re: How to set Spark executor memory?

; > > > *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com] > *Sent:* Monday, March 16, 2015 2:06 PM > *To:* Xi Shen > *Cc:* user@spark.apache.org > *Subject:* Re: How to set Spark executor memory? > > > > By default spark.executor.memory is set to 512m, I'm assumin

RE: How to set Spark executor memory?

Sent: Monday, March 16, 2015 2:06 PM To: Xi Shen Cc: user@spark.apache.org Subject: Re: How to set Spark executor memory? By default spark.executor.memory is set to 512m, I'm assuming since you are submiting the job using spark-submit and it is not able to override the value since you are r

Re: How to set Spark executor memory?

There are a number of small misunderstandings here. In the first instance, the executor memory is not actually being set to 2g and the default of 512m is being used. If you are writing code to launch an app, then you are trying to duplicate what spark-submit does, and you don't use spark-submit. I

Re: How to set Spark executor memory?

Strange, even i'm having it while running in local mode. [image: Inline image 1] I set it as .set("spark.executor.memory", "1g") Thanks Best Regards On Mon, Mar 16, 2015 at 2:43 PM, Xi Shen wrote: > I set "spark.executor.memory" to "2048m". If the executor storage memory > is 0.6 of executor

Re: How to set Spark executor memory?

I set "spark.executor.memory" to "2048m". If the executor storage memory is 0.6 of executor memory, it should be 2g * 0.6 = 1.2g. My machine has 56GB memory, and 0.6 of that should be 33.6G...I hate math xD On Mon, Mar 16, 2015 at 7:59 PM Akhil Das wrote: > How much memory are you having on yo

Re: How to set Spark executor memory?

How much memory are you having on your machine? I think default value is 0.6 of the spark.executor.memory as you can see from here . Thanks Best Regards On Mon, Mar 16, 2015 at 2:26 PM, Xi Shen wrote: > Hi Akhil, > > Yes,

Re: How to set Spark executor memory?

Hi Akhil, Yes, you are right. If I ran the program from IDE as a normal java program, the executor's memory is increased...but not to 2048m, it is set to 6.7GB...Looks like there's some formula to calculate this value. Thanks, David On Mon, Mar 16, 2015 at 7:36 PM Akhil Das wrote: > By defau

Re: How to set Spark executor memory?

By default spark.executor.memory is set to 512m, I'm assuming since you are submiting the job using spark-submit and it is not able to override the value since you are running in local mode. Can you try it without using spark-submit as a standalone project? Thanks Best Regards On Mon, Mar 16, 201

Re: How to set Spark executor memory?

I set it in code, not by configuration. I submit my jar file to local. I am working in my developer environment. On Mon, 16 Mar 2015 18:28 Akhil Das wrote: > How are you setting it? and how are you submitting the job? > > Thanks > Best Regards > > On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen wrote

Re: How to set Spark executor memory?

How are you setting it? and how are you submitting the job? Thanks Best Regards On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen wrote: > Hi, > > I have set spark.executor.memory to 2048m, and in the UI "Environment" > page, I can see this value has been set correctly. But in the "Executors" > page, I

How to set Spark executor memory?

Hi, I have set spark.executor.memory to 2048m, and in the UI "Environment" page, I can see this value has been set correctly. But in the "Executors" page, I saw there's only 1 executor and its memory is 265.4MB. Very strange value. why not 256MB, or just as what I set? What am I missing here? T