Mingkang Li created SPARK-48900:
-----------------------------------
Summary: Add `reason` string to all job / stage / job group
cancellation calls
Key: SPARK-48900
URL: https://issues.apache.org/jira/browse/SPARK-48900
Project: Spark
Issue Type: New Feature
Components: Spark Core
Affects Versions: 3.4.3
Reporter: Mingkang Li
Today it is difficult to determine _why_ a job, stage, or job group was
canceled. We should leverage existing Spark functionality to provide a
{{reason}} string explaining the cancellation cause, and should add new APIs to
let us provide this reason when canceling job groups.
{*}Details{*}:
* Since [SPARK-19549] Allow providing reasons for stage/job cancelling - ASF
JIRA (Spark 2.20), Spark’s {{cancelJob}} and {{cancelStage}} methods accept an
optional {{reason: String}} that is added to logging output and user-facing
error messages when jobs or stages are canceled. In our internal calls to these
methods, we should always supply a reason. For example, we should set an
appropriate reason when the “kill” links are clicked in the Spark UI (see
[code|https://github.com/apache/spark/blob/b14c1f036f8f394ad1903998128c05d04dd584a9/core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala#L54C1-L55]).
* Other APIs currently lack a {{reason}} field. For example,
{{cancelJobGroup}} and {{cancelJobsWithTag}} don’t provide any way to specify a
reason, so we only see generic logs like “asked to cancel job group <group
name>”. We should add an ability to pass in a group cancellation reason and
thread that through into the scheduler’s logging and job failure reasons.
This feature can be implemented in two PRs:
# Modify the current {{SparkContext}} and its downstream APIs to add the
{{reason}} string, such as {{cancelJobGroup}} and {{cancelJobsWithTag
}}
# Add reasons for all internal calls to these methods
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
