[
https://issues.apache.org/jira/browse/SPARK-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or closed SPARK-3596.
----------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
Assignee: Weizhong
Target Version/s: 1.4.0
> Support changing the yarn client monitor interval
> --------------------------------------------------
>
> Key: SPARK-3596
> URL: https://issues.apache.org/jira/browse/SPARK-3596
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Thomas Graves
> Assignee: Weizhong
> Fix For: 1.4.0
>
>
> Right now spark on yarn has a monitor interval that can be configured by
> spark.yarn.report.interval. This is how often the client checks with the RM
> to get status on the running application in cluster mode. We should allow
> users to set this interval as some may not need to check so often. There is
> another jira filed to make it so the client doesn't have to stay around for
> cluster mode.
> With the changes in https://github.com/apache/spark/pull/2350, it further
> extends that to affect client mode.
> We may want to add in specific configs for that since the monitorApplication
> function is now used in multiple different scenarios it actually might make
> sense for it to take the timeout as a parameter. You could want different
> timeout for different situations.
> for instance how quickly we poll on client side and print information
> (cluster mode) vs how quickly we recognize the application quit and we want
> to terminate (client mode), I want the latter to happen quickly where as in
> cluster mode I might not care as much about how often it is printing updated
> info to the screen. I guess its private so we could leave it as is and change
> if we add support for that later.
> my suggestion for name would be something like
> spark.yarn.client.progress.pollinterval. If we were to add separate ones in
> the future then they could be something like
> spark.yarn.app.ready.pollinterval and spark.yarn.app.completion.pollinterval
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
