Mikko Kortelainen created SPARK-46238:
-----------------------------------------
Summary: spark-submit does not provide a method to
setApplicationTimeouts
Key: SPARK-46238
URL: https://issues.apache.org/jira/browse/SPARK-46238
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 3.5.0
Reporter: Mikko Kortelainen
{{ApplicationSubmissionContext#setApplicationTimeouts(Map<ApplicationTimeoutType,
Long> applicationTimeouts)}}
This can be used to enforce application maximum lifetime{{{}. {}}}
{{}}
{{Javadoc at}}
[https://hadoop.apache.org/docs/r3.0.2/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setApplicationTimeouts-java.util.Map-]{{{}{}}}
{{https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html}}
It would be {{preferable to be able to set this to enforce timeout of an
application.}}
{{}}
It is possible to set these cluster wide from yarn scheduler queues:
{{[https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html]}}
{{}}
{{}}
{{yarn.scheduler.capacity.<queue-path>.maximum-application-lifetime}}
{{yarn.scheduler.capacity.root.<queue-path>.default-application-lifetime}}
{{}}
{{however this is suboptimal in many cases as a submitted might have different
requirements for application lifetime than the queue defaults.}}
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
