Jubin Soni created SPARK-58358:
----------------------------------
Summary: Tighten job cancellation test to assert on the
cancellation error instead of swallowing all exceptions
Key: SPARK-58358
URL: https://issues.apache.org/jira/browse/SPARK-58358
Project: Spark
Issue Type: Test
Components: PySpark, Tests
Affects Versions: 4.1.0
Reporter: Jubin Soni
The test helper {{check_job_cancellation}} in:
python/pyspark/sql/tests/test_job_cancellation.py
Because this catches {*}all exceptions{*}, unrelated failures (for example, a
UDF import error, runs a UDF-based job in a thread and currently classifies the
outcome as "cancelled" using a broad {{except Exception}} handler:
try:
setter(job_id)
...
self.spark.createDataFrame(...).collect()
is_job_cancelled[index] = False
except Exception:
# Assume that exception means job cancellation.
is_job_cancelled[index] = True
Because this catches {*}all exceptions{*}, unrelated failures (for example, a
UDF import error, serialization failure, or environment/configuration issue)
are incorrectly interpreted as successful job cancellation.
This can mask real test failures and allow cancellation tests such as
{{test_interrupt_all}} (and, once re-enabled, {{{}test_interrupt_tag{}}}) to
pass for the wrong reason.
*Proposal:*
Narrow the exception handling so that the test only treats
cancellation-specific failures as a successful cancellation. The helper should
verify that the raised exception corresponds to an actual Spark job
cancellation (for example, by checking the relevant cancellation
exception/error class) rather than assuming any exception indicates
cancellation.
Any non-cancellation failure should fail the test instead of being counted as a
pass.
*Impact:*
* Test-only change.
* No user-facing behavior change.
* Improves test correctness by preventing unrelated failures from being
silently ignored.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
