[
https://issues.apache.org/jira/browse/SPARK-58536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luozonghai updated SPARK-58536:
-------------------------------
Fix Version/s: 4.1.2
(was: 4.1.1)
> getDefaultFinalStatus() should return SUCCEEDED in cluster mode matching its
> documentation
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-58536
> URL: https://issues.apache.org/jira/browse/SPARK-58536
> Project: Spark
> Issue Type: Bug
> Components: Bug
> Affects Versions: 4.1.0, 4.0.0, 4.1.1
> Reporter: luozonghai
> Priority: Major
> Fix For: 4.1.2
>
>
> In {{ApplicationMaster.scala}}, the {{getDefaultFinalStatus()}} method's
> docstring says:
> Set the default final application status for client mode to UNDEFINED
> to handle if YARN HA restarts the application so that it properly
> retries. Set the final status to SUCCEEDED in cluster mode to handle
> if the user calls System.exit from the application code.
> But the implementation returns {{FinalApplicationStatus.FAILED}} for cluster
> mode, contradicting the documented behavior.
> This causes batch-mode spark-sql (cluster mode) to report FAILED in YARN RM
> when using certain Hadoop client versions, because the shutdown hook for
> {{SparkContext}} fires before {{finish(SUCCEEDED)}} is called, and the
> default FAILED status is used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
