[
https://issues.apache.org/jira/browse/SPARK-45957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghu Angadi updated SPARK-45957:
---------------------------------
Description:
The following code fails in the last step with Spark Connect.
The root cause is that Connect server triggers physical plan on a streaming
Dataframe [in
SparkConnectPlanner.scala|https://github.com/apache/spark/blob/334d952f9555cbfad8ef84987d6f978eb6b37b9b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L2591].
Better to avoid that entirely, but at least for streaming it should be avoided
since it cannot be done with a batch execution engine.
{code:java}
df = spark.readStream.format("rate").option("numPartitions", "1").load()
df.createOrReplaceTempView("temp_view")
view_df = spark.sql("SELECT * FROM temp_view") // FAILS{code}
was:
The following code fails in the last step with Spark Connect.
The root cause is that Connect server triggers physical plan on a streaming
Dataframe [in
SparkConnectPlanner.scala|https://github.com/apache/spark/blob/334d952f9555cbfad8ef84987d6f978eb6b37b9b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L2591].
Better to avoid that entirely, but at least for streaming it should be avoided
since it cannot be done with a batch execution engine.
{code:java}
df = spark.readStream.format("rate").option("numPartitions", "1").load()
df.createOrReplaceTempView("temp_view")
view_df = spark.sql("SELECT * FROM temp_view") // FAILS{code}
> SQL on streaming Temp view fails
> --------------------------------
>
> Key: SPARK-45957
> URL: https://issues.apache.org/jira/browse/SPARK-45957
> Project: Spark
> Issue Type: Bug
> Components: Connect, Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Raghu Angadi
> Priority: Major
> Fix For: 4.0.0
>
>
> The following code fails in the last step with Spark Connect.
> The root cause is that Connect server triggers physical plan on a streaming
> Dataframe [in
> SparkConnectPlanner.scala|https://github.com/apache/spark/blob/334d952f9555cbfad8ef84987d6f978eb6b37b9b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L2591].
> Better to avoid that entirely, but at least for streaming it should be
> avoided since it cannot be done with a batch execution engine.
> {code:java}
> df = spark.readStream.format("rate").option("numPartitions", "1").load()
> df.createOrReplaceTempView("temp_view")
> view_df = spark.sql("SELECT * FROM temp_view") // FAILS{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
