Github user Cangyl commented on the issue:
https://github.com/apache/spark/pull/22876
What changes were proposed in this pull request?
When set configs in yarn as below:
yarn.scheduler.minimum-allocation-mb 50mb
yarn.scheduler.increment-allocation-mb 50mb
And submit spark on yarn job using the command below:
In ApplicationMaster.scala, when a spark applicaiton failed in the
LastAttempt, it will try to unregister itself from yarn resourcemanager.
Normally, during ${spark.yarn.report.interval}, the unregister event will be
sent to resourcemanager before the failed container event, and overwrite the
container failded diagnostics with "Shutdown hook called before final status
was reported." This is the prolem code in Application.scala
<if (!unregistered) {
// we only want to unregister if we don't want the RM to retry
if (finalStatus == FinalApplicationStatus.SUCCEEDED ||
isLastAttempt) {
unregister(finalStatus, finalMsg)
cleanupStagingDir()
}
}>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
