Re: strange behavior in spark yarn-client mode
Please reply to the list. The web ui does not show the total size of the executor's heap. It shows the amount of memory available for caching data, which is, give or take, 60% of the heap by default. On Thu, Jan 14, 2016 at 11:03 AM, Sanjeev Verma wrote: > I am looking into the web ui of spark application master(tab executors). > > On Fri, Jan 15, 2016 at 12:08 AM, Marcelo Vanzin > wrote: >> >> On Thu, Jan 14, 2016 at 10:17 AM, Sanjeev Verma >> wrote: >> > now it spawn a single executors with 1060M size, I am not able to >> > understand >> > why this time it executes executors with 1G+overhead not 2G what I >> > specified. >> >> Where are you looking for the memory size for the container? >> >> -- >> Marcelo > > -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: strange behavior in spark yarn-client mode
On Thu, Jan 14, 2016 at 10:17 AM, Sanjeev Verma wrote: > now it spawn a single executors with 1060M size, I am not able to understand > why this time it executes executors with 1G+overhead not 2G what I > specified. Where are you looking for the memory size for the container? -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
strange behavior in spark yarn-client mode
I am seeing a strange behaviour while running spark in yarn client mode.I am observing this on the single node yarn cluster.in spark-default I have configured the executors memory as 2g and started the spark shell as follows bin/spark-shell --master yarn-client which trigger the 2 executors on the node with 1060MB of memory, I am able to figure out that if you wont specify the num-executors it will span 2 executors on the node by defaults. now when i try to run again it with the bin/spark-shell --master yarn-client --num-executors 1 now it spawn a single executors with 1060M size, I am not able to understand why this time it executes executors with 1G+overhead not 2G what I specified. why I am seeing this strange behavior?
