Lobo2008 opened a new pull request, #57736:
URL: https://github.com/apache/spark/pull/57736
### What changes were proposed in this pull request?
Fix `getDefaultFinalStatus()` to return `FinalApplicationStatus.SUCCEEDED`
for cluster mode, matching its documentation.
### Why are the changes needed?
The current implementation returns `FAILED` for cluster mode, while the
docstring says it should return `SUCCEEDED`. The docstring correctly
describes the intent: when a user's application calls `System.exit(0)`,
the shutdown hook fires before `finish(SUCCEEDED)` can be reached. The
default status should be `SUCCEEDED` in this case, as the application
completed normally.
This is safe because genuinely failed applications call `finish(FAILED)`
explicitly, which sets `finished = true`, causing the shutdown hook to
skip the default status entirely.
### Does this PR introduce any user-facing change?
Yes — cluster mode batch spark-sql applications that exit via
`System.exit(0)` will now report SUCCEEDED to YARN RM instead of FAILED.
### How was this patch tested?
Manual testing with Spark 4.1.1 in cluster mode on Hadoop 3.2 RM
with Hadoop 3.4.2 client. Verified that YARN RM shows SUCCEEDED
after the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
