luozonghai created SPARK-58536:
----------------------------------
Summary: getDefaultFinalStatus() should return SUCCEEDED in
cluster mode matching its documentation
Key: SPARK-58536
URL: https://issues.apache.org/jira/browse/SPARK-58536
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 4.1.1, 4.0.0, 4.1.0
Reporter: luozonghai
Fix For: 4.1.1
In {{ApplicationMaster.scala}}, the {{getDefaultFinalStatus()}} method's
docstring says:
Set the default final application status for client mode to UNDEFINED
to handle if YARN HA restarts the application so that it properly
retries. Set the final status to SUCCEEDED in cluster mode to handle
if the user calls System.exit from the application code.
But the implementation returns {{FinalApplicationStatus.FAILED}} for cluster
mode, contradicting the documented behavior.
This causes batch-mode spark-sql (cluster mode) to report FAILED in YARN RM
when using certain Hadoop client versions, because the shutdown hook for
{{SparkContext}} fires before {{finish(SUCCEEDED)}} is called, and the default
FAILED status is used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
