This is an automated email from the ASF dual-hosted git repository.
zsxwing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 11e5a465c516 [SPARK-60851][PYTHON][SS] Fix streaming listener job tags
11e5a465c516 is described below
commit 11e5a465c516f67cb6357e56a19b482d2ccfb138
Author: Li Zhang <[email protected]>
AuthorDate: Tue Nov 3 21:17:37 2026 -0800
[SPARK-60851][PYTHON][SS] Fix streaming listener job tags
### What changes were proposed in this pull request?
Convert streaming `QueryStartedEvent` job tags with Python `str()` and add
a listener regression case with a nonempty job tag.
### Why are the changes needed?
Py4J can convert a Java string to a Python `str` before
`QueryStartedEvent.fromJObject` reads it. Calling `.toString()` on that value
raises `AttributeError`, so the Python listener does not receive
`onQueryStarted` when the query has job tags. A query started after
`SparkContext.addJobTag(...)` exercises this path.
### Does this PR introduce _any_ user-facing change?
Yes. Python streaming listeners now receive `onQueryStarted` with the
query's nonempty job tags. Previously the callback failed during event
conversion.
### How was this patch tested?
The existing streaming listener test now adds a job tag before starting a
query, checks that the Python start event contains it, and clears the tag
during cleanup. Both modified Python files passed `python3 -m py_compile`. The
full Spark test suite has not been run locally. GitHub Actions stopped before
test steps started, so CI validation is pending.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
Closes #60066 from lzhang0616/fix-streaming-listener-job-tags.
Authored-by: Li Zhang <[email protected]>
Signed-off-by: Shixiong Zhu <[email protected]>
---
python/pyspark/sql/streaming/listener.py | 2 +-
python/pyspark/sql/tests/streaming/test_streaming_listener.py | 3 +++
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/streaming/listener.py
b/python/pyspark/sql/streaming/listener.py
index bb247e750a48..76fec6cf159b 100644
--- a/python/pyspark/sql/streaming/listener.py
+++ b/python/pyspark/sql/streaming/listener.py
@@ -195,7 +195,7 @@ class QueryStartedEvent:
job_tags = set()
java_iterator = jevent.jobTags().iterator()
while java_iterator.hasNext():
- job_tags.add(java_iterator.next().toString())
+ job_tags.add(str(java_iterator.next()))
return cls(
id=uuid.UUID(jevent.id().toString()),
diff --git a/python/pyspark/sql/tests/streaming/test_streaming_listener.py
b/python/pyspark/sql/tests/streaming/test_streaming_listener.py
index ea250ec0b5c9..0c2b95bd1ac0 100644
--- a/python/pyspark/sql/tests/streaming/test_streaming_listener.py
+++ b/python/pyspark/sql/tests/streaming/test_streaming_listener.py
@@ -375,6 +375,7 @@ class StreamingListenerTests(StreamingListenerTestsMixin,
ReusedSQLTestCase):
try:
self.spark.streams.addListener(test_listener)
+ self.spark.sparkContext.addJobTag("listener-job-tag")
df =
self.spark.readStream.format("rate").option("rowsPerSecond", 10).load()
@@ -399,6 +400,7 @@ class StreamingListenerTests(StreamingListenerTestsMixin,
ReusedSQLTestCase):
self.spark.sparkContext._jsc.sc().listenerBus().waitUntilEmpty()
self.check_start_event(start_event)
+ self.assertIn("listener-job-tag", start_event.jobTags)
self.check_progress_event(progress_event, True)
self.check_terminated_event(terminated_event)
@@ -423,6 +425,7 @@ class StreamingListenerTests(StreamingListenerTestsMixin,
ReusedSQLTestCase):
)
finally:
+ self.spark.sparkContext.clearJobTags()
self.spark.streams.removeListener(test_listener)
verify(TestListenerV1())
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]