[
https://issues.apache.org/jira/browse/SPARK-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324151#comment-14324151
]
Sean Owen edited comment on SPARK-5861 at 2/17/15 12:52 PM:
------------------------------------------------------------
Ah, I think this is a duplicate of
https://issues.apache.org/jira/browse/SPARK-1953
In 1.2.0, with an invocation like this, I see that the client command line is
like:
{code}
java ... -Xms6g -Xmx6g org.apache.spark.deploy.SparkSubmit --master yarn-client
--driver-memory 6g --class org.apache.spark.examples.SparkPi ...
{code}
But the application master also reserves 6+GB:
{code}
15/02/17 04:40:48 INFO Client: Will allocate AM container, with 6528 MB memory
including 384 MB overhead
{code}
But it should be resolved in 1.3.0. Before I mark it as a duplicate, however,
I'm curious.
In {{ClientArguments.scala}}, cluster mode is detected by the presence of
{{--class}}, but it's perfectly possible set {{--class}} with yarn-client mode.
Is this a case that should work after SPARK-1953?
CC [~WangTaoTheTonic] and [~tgraves] for thoughts
was (Author: srowen):
Ah, I think this is a duplicate of
https://issues.apache.org/jira/browse/SPARK-1953
In 1.2.0, with an invocation like this, I see that the client command line is
like:
{code}
java ... -Xms6g -Xmx6g org.apache.spark.deploy.SparkSubmit --master yarn-client
--driver-memory 6g --class org.apache.spark.examples.SparkPi ...
{code}
But the application master also reserves 6+GB:
{code
15/02/17 04:40:48 INFO Client: Will allocate AM container, with 6528 MB memory
including 384 MB overhead
{code}
But it should be resolved in 1.3.0. Before I mark it as a duplicate, however,
I'm curious.
In {{ClientArguments.scala}}, cluster mode is detected by the presence of
{{--class}}, but it's perfectly possible set {{--class}} with yarn-client mode.
Is this a case that should work after SPARK-1953?
CC [~WangTaoTheTonic] and [~tgraves] for thoughts
> [yarn-client mode] Application master should not use memory =
> spark.driver.memory
> ---------------------------------------------------------------------------------
>
> Key: SPARK-5861
> URL: https://issues.apache.org/jira/browse/SPARK-5861
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.2.1
> Reporter: Shekhar Bansal
>
> I am using
> {code}spark.driver.memory=6g{code}
> which creates application master of 7g
> (yarn.scheduler.minimum-allocation-mb=1024)
> Application manager don't need 7g in yarn-client mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
