Hyukjin Kwon created SPARK-61022:
------------------------------------
Summary: Ask the server whether a DataFrame is streaming in
isStreaming of the Python client as PySpark does
Key: SPARK-61022
URL: https://issues.apache.org/jira/browse/SPARK-61022
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{df.isStreaming}} asks the server
({{AnalyzePlanRequest.is_streaming}}), which is true for a DataFrame with a
streaming source anywhere in its plan:
{{spark.readStream.format("rate").load().select("value").isStreaming}} is True,
as are the filters, unions and joins of a streaming DataFrame.
The Rust-backed Python client checks only whether the root of the plan is a
streaming read, so {{isStreaming}} is False for those DataFrames. Ask the
server as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
