[
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292349#comment-15292349
]
Karthik Kambatla commented on YARN-4464:
----------------------------------------
bq. The patch is changing the config for max number of completed apps RM keeps
in memory, NOT in store. Am I missing something ?
Thanks for catching that, [~jianhe]. My bad, completely missed that.
bq. why do you think it's better to not let users see the historical apps ? I
feel it is sometimes convenient to see the finished apps after restart.
I agree that it is convenient to see finished apps. The current default of
10,000 is too large and we have seen issues with recovery due to the heavy load
on ZK. If we pick a smaller value that is not zero, it is true that users will
continue to see some finished apps. After upgrade, it is also quite possible
that users don't realize the number of jobs stored on restart has been lowered;
if they rely on the RM storing these, they might be in for a surprise later on.
Picking zero should expose this change in behavior in any test cluster as well,
and they could be a number appropriate for them. Long story short, I agree that
a number like 1000 might have been a good default to begin with. Now that we
are lowering, zero will be more in the face and avoid delayed surprises.
I am not wedded to this and am open to persuaded to a different default value.
> default value of yarn.resourcemanager.state-store.max-completed-applications
> should lower.
> ------------------------------------------------------------------------------------------
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
> Issue Type: Wish
> Components: resourcemanager
> Reporter: KWON BYUNGCHANG
> Assignee: Daniel Templeton
> Priority: Blocker
> Attachments: YARN-4464.001.patch, YARN-4464.002.patch,
> YARN-4464.003.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow. I have waited about 20min.
> realize missing
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.
> need to change lower value or document notice on [RM Restart
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
