Raghu Angadi created SPARK-45957:
------------------------------------
Summary: SQL on streaming Temp view fails
Key: SPARK-45957
URL: https://issues.apache.org/jira/browse/SPARK-45957
Project: Spark
Issue Type: Bug
Components: Connect, Structured Streaming
Affects Versions: 4.0.0
Reporter: Raghu Angadi
Fix For: 4.0.0
The following code fails in the last step with Spark Connect.
The root cause is that Connect server triggers physical plan on a streaming
Dataframe [in
SparkConnectPlanner.scala|https://github.com/apache/spark/blob/334d952f9555cbfad8ef84987d6f978eb6b37b9b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L2591].
Better to avoid that entirely, but at least for streaming it should be avoided
since it cannot be done with a batch execution engine.
{code:java}
df = spark.readStream.format("rate").option("numPartitions", "1").load()
df.createOrReplaceTempView("temp_view")
view_df = spark.sql("SELECT * FROM temp_view") // FAILS{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
