[
https://issues.apache.org/jira/browse/SPARK-60851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu updated SPARK-60851:
---------------------------------
Affects Version/s: 4.2.0
4.3.0
> Python streaming listener misses QueryStartedEvent when job tags are present
> ----------------------------------------------------------------------------
>
> Key: SPARK-60851
> URL: https://issues.apache.org/jira/browse/SPARK-60851
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Structured Streaming
> Affects Versions: 4.1.0, 4.2.0, 4.3.0
> Reporter: Li Zhang
> Assignee: Li Zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Python StreamingQueryListener.onQueryStarted is not delivered when a
> streaming query has a nonempty job tag. QueryStartedEvent.fromJObject
> iterates jevent.jobTags() and calls .toString() on each item, but Py4J may
> already have converted a Java string to a Python str. The callback then
> raises AttributeError: 'str' object has no attribute 'toString'.
>
> Reproduction:
> # Register a Python StreamingQueryListener that records onQueryStarted.
> # Call spark.sparkContext.addJobTag("listener-job-tag").
> # Start a streaming query (for example, with the rate source).
>
> Expected: The Python listener receives onQueryStarted, and event.jobTags
> contains listener-job-tag.
>
> Proposed fix: Convert each tag with Python str() and add a listener
> regression test with a nonempty tag.
>
> Pull request: https://github.com/apache/spark/pull/60066
>
> Related: SPARK-51981 added job tags to QueryStartedEvent in Spark 4.1.0. It
> does not track this Python callback failure.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
