Thomas Graves created SPARK-1953:
------------------------------------
Summary: yarn client mode Application Master memory size is same
as driver memory size
Key: SPARK-1953
URL: https://issues.apache.org/jira/browse/SPARK-1953
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.0.0
Reporter: Thomas Graves
With Spark on yarn in client mode, the application master that gets created to
allocated containers gets the same amount of memory as the driver running on
the client. (--driver-memory option through spark-submit) This could
definitely be more then what is really needed, thus wasting resources. The
Application Master should be very small and require very little memory since
all its doing is allocating and starting containers.
We should allow the memory for the application master to be configurable
separate from the driver in client mode.
We probably need to be careful about how we do this as to not cause confusion
about what the options do in the various modes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
