[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN
[
https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799087#comment-15799087
]
Henry Kim commented on SPARK-15984:
---
the workaround that Saisai stated is to add the following config to your app
--conf spark.yarn.maxAppAttempts=2
> WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max
> attempts: 0 for application: 8 is invalid" when starting application on YARN
> --
>
> Key: SPARK-15984
> URL: https://issues.apache.org/jira/browse/SPARK-15984
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 2.0.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> When executing {{spark-shell}} on Spark on YARN 2.7.2 on Mac OS as follows:
> {code}
> YARN_CONF_DIR=hadoop-conf ./bin/spark-shell --master yarn -c
> spark.shuffle.service.enabled=true --deploy-mode client -c
> spark.scheduler.mode=FAIR
> {code}
> it ends up with the following WARN in the logs:
> {code}
> 2016-06-16 08:33:05,308 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new
> applicationId: 8
> 2016-06-16 08:33:07,305 WARN
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The specific
> max attempts: 0 for application: 8 is invalid, because it is out of the range
> [1, 2]. Use the global max attempts instead.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN
[
https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339136#comment-15339136
]
Saisai Shao commented on SPARK-15984:
-
I see. Here it is because user didn't specify this
{{individualMaxAppAttempts}}, in the RM it will get the default value 0, and
this value is out of range {{(1, globalMaxAppAttempts)}}, so you will see the
warning log. I think here the logic is expected and correct. If you really want
to fix it you could either set this {{individualMaxAppAttempts}} to
{{globalMaxAppAttempts}} as a default value in Spark side, or fix the log in
the yarn side.
> WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max
> attempts: 0 for application: 8 is invalid" when starting application on YARN
> --
>
> Key: SPARK-15984
> URL: https://issues.apache.org/jira/browse/SPARK-15984
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 2.0.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> When executing {{spark-shell}} on Spark on YARN 2.7.2 on Mac OS as follows:
> {code}
> YARN_CONF_DIR=hadoop-conf ./bin/spark-shell --master yarn -c
> spark.shuffle.service.enabled=true --deploy-mode client -c
> spark.scheduler.mode=FAIR
> {code}
> it ends up with the following WARN in the logs:
> {code}
> 2016-06-16 08:33:05,308 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new
> applicationId: 8
> 2016-06-16 08:33:07,305 WARN
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The specific
> max attempts: 0 for application: 8 is invalid, because it is out of the range
> [1, 2]. Use the global max attempts instead.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN
[
https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337693#comment-15337693
]
Jacek Laskowski commented on SPARK-15984:
-
The problem is that I am *not* changing Spark at all and so by default it gives
the warning. If it's a warning and Spark does it, it'd be better (?) to play
nicer with YARN. I could fix it easily if I was told it changes nothing else in
Spark. I don't know so that's why I reported it (since it's a warning anyway).
> WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max
> attempts: 0 for application: 8 is invalid" when starting application on YARN
> --
>
> Key: SPARK-15984
> URL: https://issues.apache.org/jira/browse/SPARK-15984
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 2.0.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> When executing {{spark-shell}} on Spark on YARN 2.7.2 on Mac OS as follows:
> {code}
> YARN_CONF_DIR=hadoop-conf ./bin/spark-shell --master yarn -c
> spark.shuffle.service.enabled=true --deploy-mode client -c
> spark.scheduler.mode=FAIR
> {code}
> it ends up with the following WARN in the logs:
> {code}
> 2016-06-16 08:33:05,308 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new
> applicationId: 8
> 2016-06-16 08:33:07,305 WARN
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The specific
> max attempts: 0 for application: 8 is invalid, because it is out of the range
> [1, 2]. Use the global max attempts instead.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN
[
https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337278#comment-15337278
]
Saisai Shao commented on SPARK-15984:
-
Is there any problem? I guess you might set max app attempt to 0, so you will
get such warning log, it is illegal to set 0 for max app attempt, it should be
>= 1.
Here is the yarn code.
{code}
int globalMaxAppAttempts =
conf.getInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS,
YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS);
int individualMaxAppAttempts = submissionContext.getMaxAppAttempts();
if (individualMaxAppAttempts <= 0 ||
individualMaxAppAttempts > globalMaxAppAttempts) {
this.maxAppAttempts = globalMaxAppAttempts;
LOG.warn("The specific max attempts: " + individualMaxAppAttempts
+ " for application: " + applicationId.getId()
+ " is invalid, because it is out of the range [1, "
+ globalMaxAppAttempts + "]. Use the global max attempts instead.");
} else {
this.maxAppAttempts = individualMaxAppAttempts;
}
{code}
> WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max
> attempts: 0 for application: 8 is invalid" when starting application on YARN
> --
>
> Key: SPARK-15984
> URL: https://issues.apache.org/jira/browse/SPARK-15984
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 2.0.0
>Reporter: Jacek Laskowski
>Priority: Minor
>
> When executing {{spark-shell}} on Spark on YARN 2.7.2 on Mac OS as follows:
> {code}
> YARN_CONF_DIR=hadoop-conf ./bin/spark-shell --master yarn -c
> spark.shuffle.service.enabled=true --deploy-mode client -c
> spark.scheduler.mode=FAIR
> {code}
> it ends up with the following WARN in the logs:
> {code}
> 2016-06-16 08:33:05,308 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new
> applicationId: 8
> 2016-06-16 08:33:07,305 WARN
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The specific
> max attempts: 0 for application: 8 is invalid, because it is out of the range
> [1, 2]. Use the global max attempts instead.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
