tgravescs commented on code in PR #36991:
URL: https://github.com/apache/spark/pull/36991#discussion_r915948343
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:
##########
@@ -859,7 +859,8 @@ private[yarn] class YarnAllocator(
//
.com/apache/hadoop/blob/228156cfd1b474988bc4fedfbf7edddc87db41e3/had
//
oop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/ap
// ache/hadoop/yarn/util/Apps.java#L273 for details)
- if
(NOT_APP_AND_SYSTEM_FAULT_EXIT_STATUS.contains(other_exit_status)) {
+ if
(NOT_APP_AND_SYSTEM_FAULT_EXIT_STATUS.contains(other_exit_status) ||
+ SparkContext.getActive.forall(_.isStopped)) {
Review Comment:
this isn't going to work in Cluster mode on yarn where the application
master and yarn allocator are not in the same process as the SparkContext. I
assume in that case the getActive is returning None and we would do this all
the time when we really shouldn't.
Can we tell the allocator we are shutting down when the ApplicationMaster is
told to shutdown and do a similar check to prevent this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
