This is an automated email from the ASF dual-hosted git repository.

zsxwing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 11e5a465c516 [SPARK-60851][PYTHON][SS] Fix streaming listener job tags
11e5a465c516 is described below

commit 11e5a465c516f67cb6357e56a19b482d2ccfb138
Author: Li Zhang <[email protected]>
AuthorDate: Tue Nov 3 21:17:37 2026 -0800

    [SPARK-60851][PYTHON][SS] Fix streaming listener job tags
    
    ### What changes were proposed in this pull request?
    
    Convert streaming `QueryStartedEvent` job tags with Python `str()` and add 
a listener regression case with a nonempty job tag.
    
    ### Why are the changes needed?
    
    Py4J can convert a Java string to a Python `str` before 
`QueryStartedEvent.fromJObject` reads it. Calling `.toString()` on that value 
raises `AttributeError`, so the Python listener does not receive 
`onQueryStarted` when the query has job tags. A query started after 
`SparkContext.addJobTag(...)` exercises this path.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Python streaming listeners now receive `onQueryStarted` with the 
query's nonempty job tags. Previously the callback failed during event 
conversion.
    
    ### How was this patch tested?
    
    The existing streaming listener test now adds a job tag before starting a 
query, checks that the Python start event contains it, and clears the tag 
during cleanup. Both modified Python files passed `python3 -m py_compile`. The 
full Spark test suite has not been run locally. GitHub Actions stopped before 
test steps started, so CI validation is pending.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6).
    
    Closes #60066 from lzhang0616/fix-streaming-listener-job-tags.
    
    Authored-by: Li Zhang <[email protected]>
    Signed-off-by: Shixiong Zhu <[email protected]>
---
 python/pyspark/sql/streaming/listener.py                      | 2 +-
 python/pyspark/sql/tests/streaming/test_streaming_listener.py | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/sql/streaming/listener.py 
b/python/pyspark/sql/streaming/listener.py
index bb247e750a48..76fec6cf159b 100644
--- a/python/pyspark/sql/streaming/listener.py
+++ b/python/pyspark/sql/streaming/listener.py
@@ -195,7 +195,7 @@ class QueryStartedEvent:
         job_tags = set()
         java_iterator = jevent.jobTags().iterator()
         while java_iterator.hasNext():
-            job_tags.add(java_iterator.next().toString())
+            job_tags.add(str(java_iterator.next()))
 
         return cls(
             id=uuid.UUID(jevent.id().toString()),
diff --git a/python/pyspark/sql/tests/streaming/test_streaming_listener.py 
b/python/pyspark/sql/tests/streaming/test_streaming_listener.py
index ea250ec0b5c9..0c2b95bd1ac0 100644
--- a/python/pyspark/sql/tests/streaming/test_streaming_listener.py
+++ b/python/pyspark/sql/tests/streaming/test_streaming_listener.py
@@ -375,6 +375,7 @@ class StreamingListenerTests(StreamingListenerTestsMixin, 
ReusedSQLTestCase):
 
             try:
                 self.spark.streams.addListener(test_listener)
+                self.spark.sparkContext.addJobTag("listener-job-tag")
 
                 df = 
self.spark.readStream.format("rate").option("rowsPerSecond", 10).load()
 
@@ -399,6 +400,7 @@ class StreamingListenerTests(StreamingListenerTestsMixin, 
ReusedSQLTestCase):
                 
self.spark.sparkContext._jsc.sc().listenerBus().waitUntilEmpty()
 
                 self.check_start_event(start_event)
+                self.assertIn("listener-job-tag", start_event.jobTags)
                 self.check_progress_event(progress_event, True)
                 self.check_terminated_event(terminated_event)
 
@@ -423,6 +425,7 @@ class StreamingListenerTests(StreamingListenerTestsMixin, 
ReusedSQLTestCase):
                 )
 
             finally:
+                self.spark.sparkContext.clearJobTags()
                 self.spark.streams.removeListener(test_listener)
 
         verify(TestListenerV1())


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to