Hi Denny, This is due the spark.yarn.memoryOverhead parameter, depending on what version of Spark you are on the default of this may differ, but it should be the larger of 1024mb per executor or .07 * executorMemory.
When you set executor memory, the yarn resource request is executorMemory + yarnOverhead. - Arun On Sat, Dec 6, 2014 at 4:27 PM, Denny Lee <denny.g....@gmail.com> wrote: > This is perhaps more of a YARN question than a Spark question but i was > just curious to how is memory allocated in YARN via the various > configurations. For example, if I spin up my cluster with 4GB with a > different number of executors as noted below > > 4GB executor-memory x 10 executors = 46GB (4GB x 10 = 40 + 6) > 4GB executor-memory x 4 executors = 19GB (4GB x 4 = 16 + 3) > 4GB executor-memory x 2 executors = 10GB (4GB x 2 = 8 + 2) > > The pattern when observing the RM is that there is a container for each > executor and one additional container. From the basis of memory, it looks > like there is an additional (1GB + (0.5GB x # executors)) that is allocated > in YARN. > > Just wondering why is this - or is this just an artifact of YARN itself? > > Thanks! > >
