[
https://issues.apache.org/jira/browse/SPARK-58358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jubin Soni updated SPARK-58358:
-------------------------------
Summary: SparkSession.removeTag does not validate tags (Spark Classic)
(was: Tighten job cancellation test to assert on the cancellation error instead
of swallowing all exceptions)
> SparkSession.removeTag does not validate tags (Spark Classic)
> -------------------------------------------------------------
>
> Key: SPARK-58358
> URL: https://issues.apache.org/jira/browse/SPARK-58358
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 4.0.0
> Reporter: Jubin Soni
> Priority: Minor
>
> {{SparkSession.removeTag(tag)}} in Spark Classic does not validate its
> argument, even though:
> * Its API contract documents that tags {*}cannot be empty or contain a comma
> ({{{}','{}}}){*}.
> * The sibling {{addTag}} API validates input via
> {{{}SparkContext.throwIfInvalidTag{}}}.
> * The Spark Connect implementation validates both {{addTag}} and
> {{{}removeTag{}}}.
> * {{SparkContext.removeJobTags}} also validates its input.
> The current Spark Classic implementation removes the tag directly from the
> managed tag set:
> override def removeTag(tag: String): Unit = managedJobTags.get().remove(tag)
> As a result, calls such as {{removeTag("")}} and {{removeTag("a,b")}}
> silently succeed instead of throwing {{{}IllegalArgumentException{}}}.
> This is a correctness and consistency issue because:
> * Spark Classic does not enforce its documented API contract.
> * Spark Classic and Spark Connect exhibit different behavior for the same
> API.
> *Proposed Fix:*
> Update the Spark Classic implementation of `{{{}SparkSession.removeTag{}}}`
> to call:
> {code:java}
> SparkContext.throwIfInvalidTag(tag){code}
> before removing the tag, mirroring the existing behavior of {{{}addTag{}}}.
>
> *Test Coverage:*
> Add missing validation tests for both implementations:
> * *Scala ({{{}SparkSessionJobTaggingAndCancellationSuite{}}})*
> ** Verify {{addTag}} and {{removeTag}} reject empty and comma-containing
> tags.
> ** Verify invalid tags are not added to the current tag set.
> ** Verify removing a valid but absent tag remains a no-op.
> * *PySpark ({{{}test_job_cancellation{}}})*
> ** Verify the same API contract in both Spark Classic and Spark Connect (via
> the shared test mixin).
> ** Additionally, tighten the {{check_job_cancellation}} helper to re-raise
> non-cancellation exceptions instead of treating any exception as a successful
> cancellation.
> *Impact:*
> * Test and correctness improvement only.
> * No user-facing API changes beyond enforcing the existing documented
> contract consistently across Spark Classic and Spark Connect.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
