[
https://issues.apache.org/jira/browse/SPARK-54879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rob Reeves updated SPARK-54879:
-------------------------------
Description:
If a Spark job fails due to an issue on the driver side (e.g. during the commit
phase, custom driver code) this is not surfaced anywhere in SHS. This becomes
even more confusing when a query/job has run successfully. SHS will show all
jobs as completed successfully and the only way to know the job failed is to
look in the logs. Before this change this is what a failed driver side
application would look like in SHS. It looks like a success.
!image-2026-01-01-21-09-52-487.png|width=360,height=400!
To improve this the application failure status should be shown for failures.
was:
Spark History Server doesn't show the application final status. This can be
confusing for users in a few scenarios. Some examples:
# A subset of queries/jobs execute successfully, but something fails in the
commit or driver code. SHS will just show successfully completed jobs and you
need to go to the logs to see a failure.
# Spark driver crashes. SHS shows the app as still running.
To improve debugging the SHS application overview should include the
application final status.
> Add final application status to Spark History Server for failed executions
> --------------------------------------------------------------------------
>
> Key: SPARK-54879
> URL: https://issues.apache.org/jira/browse/SPARK-54879
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: Rob Reeves
> Priority: Minor
> Labels: pull-request-available
> Attachments: image-2026-01-01-21-09-52-487.png
>
>
> If a Spark job fails due to an issue on the driver side (e.g. during the
> commit phase, custom driver code) this is not surfaced anywhere in SHS. This
> becomes even more confusing when a query/job has run successfully. SHS will
> show all jobs as completed successfully and the only way to know the job
> failed is to look in the logs. Before this change this is what a failed
> driver side application would look like in SHS. It looks like a success.
> !image-2026-01-01-21-09-52-487.png|width=360,height=400!
> To improve this the application failure status should be shown for failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
