This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new c174954e16cf [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame
arguments in a pipeline flow to flow analysis
c174954e16cf is described below
commit c174954e16cf00ed35e567244b846200f7c8f9f8
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 15:22:49 2026 -0700
[SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a
pipeline flow to flow analysis
### What changes were proposed in this pull request?
When `PipelinesHandler` handles a `DefineFlow` command, it transforms the
relation of the flow with `SparkConnectPlanner.transformRelation`.
`spark.sql(query, df=...)` with DataFrame arguments sends a SQL-with-references
`WITH_RELATIONS` relation, and `transformRelation` handles it with
`executeSQLWithRefs`. That method analyzes each reference, registers it as a
temporary view, and runs the root SQL with `session.sql`, all at flow
registration.
This PR adds `SparkConnectPlanner.transformPipelineFlowRelation` and passes
it to `PipelinesHandler` instead of `transformRelation`. While it transforms
the relation of a flow, `transformSqlWithRefs` does not execute a valid
SQL-with-references `WITH_RELATIONS`. It parses the root SQL and binds each
reference to its alias as a CTE relation (`UnresolvedWith`). `FlowAnalysis`
already substitutes CTEs before it resolves `UnresolvedRelation`s as pipeline
inputs, so it handles a dataset th [...]
`handleSqlCommand` and all other Spark Connect requests still use
`executeSQLWithRefs`, so `spark.sql` with DataFrame arguments outside of a flow
definition is unchanged.
This PR does not change the SDP guard in `handleSqlCommand`. For a
reference input that is not a SQL-with-references `WITH_RELATIONS` itself, the
guard still uses `transformRelation`, which executes any SQL-with-references
`WITH_RELATIONS` nested in that input. SPARK-57189 documented this as a known
limitation of `blockUnsupportedSqlCommandIn`. As a result, if a flow function
passes a further-transformed `spark.sql(..., df=...)` result, such as
`inner.where("id > 0")`, as a DataFrame [...]
### Why are the changes needed?
Inside a flow function, `spark.sql` does not run the SQL. The server
returns the relation as is (SPARK-54452), so that the flow analysis can resolve
it later. With DataFrame arguments, however, `DefineFlow` analyzes it right
away:
1. The flow fails to register when a DataFrame argument reads a dataset of
the same pipeline that does not exist in the catalog: a table or materialized
view on the first run, or a temporary view on every run.
```python
dp.materialized_view
def upstream():
return spark.range(5)
dp.materialized_view
def downstream():
return spark.sql("SELECT * FROM {df}",
df=spark.read.table("upstream"))
```
```
pyspark.errors.exceptions.connect.AnalysisException:
[TABLE_OR_VIEW_NOT_FOUND] The table or view `upstream` cannot be found. ...
```
2. When the table already exists, for example from a previous run, the flow
registers an analyzed plan. The flow analysis then does not see the read of
`upstream` and does not record it as an input of `downstream`. Without the
dependency, `downstream` can run before `upstream` is refreshed and read stale
data, without an error.
3. A command at the root that the SDP guard allows, such as `SHOW TABLES`,
is executed at flow registration.
The same query without DataFrame arguments, `spark.sql("SELECT * FROM
upstream")`, works because its relation is only parsed at flow registration.
In Spark 4.1.0 and 4.2.0, the SDP guard in `handleSqlCommand` also analyzed
the relation, so the error in (1) was raised at the `spark.sql` call in the
flow function. SPARK-57189 (#59425) moved it to flow registration.
### Does this PR introduce _any_ user-facing change?
Yes. In a flow function, `spark.sql` with DataFrame arguments can read
datasets of the same pipeline, and the flow analysis records them as inputs of
the flow. The SQL is analyzed during graph resolution instead of flow
registration, so an analysis error is reported as a flow resolution failure, as
for `spark.sql` without DataFrame arguments.
In a flow definition, DataFrame arguments are now bound as CTE relations
instead of temporary views. A statement that refers to a DataFrame argument
other than as a table in a query, such as `DESCRIBE TABLE {df}`, can no longer
resolve it when a flow function returns the statement.
### How was this patch tested?
Added two tests to `PythonPipelineSuite`, and a flow to an existing test
there. All of them fail without this fix.
- `SPARK-60494: referencing internal datasets through DataFrame arguments
of spark.sql`: a new pipeline reads a materialized view with batch and
streaming reads, and a temporary view with a batch read, all through DataFrame
arguments. Two more flows have the SQL with DataFrame arguments under another
relation: one applies `where` to the result, and the other passes the result as
a DataFrame argument of another `spark.sql` call. The test checks the inputs
and streaming-ness of the flow [...]
- `SPARK-60494: referencing an internal dataset through DataFrame arguments
of spark.sql tracks the dependency when its table already exists`: Without the
fix, the inputs of the flow are empty.
- `referencing internal datasets failed`: a new flow reads an undefined
dataset through a DataFrame argument. Like the other flows in the test, it
fails graph resolution with `TABLE_OR_VIEW_NOT_FOUND` for `src`. Without the
fix, its registration fails.
Also passed the existing suites in
`org.apache.spark.sql.connect.pipelines`, including `PythonPipelineSuite` and
`SparkDeclarativePipelinesServerSuite`, as well as `SparkConnectPlannerSuite`
and `SparkConnectProtoSuite`:
```
build/sbt 'connect/testOnly org.apache.spark.sql.connect.pipelines.*
*SparkConnectPlannerSuite *SparkConnectProtoSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59735 from dongjoon-hyun/SPARK-60494.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit f86fc93cc4cb644f020ef8e78c31e70b052b8e1b)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../sql/connect/planner/SparkConnectPlanner.scala | 36 +++++++++-
.../connect/pipelines/PythonPipelineSuite.scala | 76 +++++++++++++++++++++-
2 files changed, 108 insertions(+), 4 deletions(-)
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
index 7e6ebc9b5f41..429892a645bf 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
@@ -52,7 +52,7 @@ import
org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
import org.apache.spark.sql.catalyst.parser.{NamedParameterContext,
ParameterContext, ParseException, ParserUtils, PositionalParameterContext}
import org.apache.spark.sql.catalyst.plans.{Cross, FullOuter, Inner, JoinType,
LeftAnti, LeftOuter, LeftSemi, RightOuter, UsingJoin}
import org.apache.spark.sql.catalyst.plans.logical
-import org.apache.spark.sql.catalyst.plans.logical.{AppendColumns, Assignment,
CoGroup, CollectMetrics, CommandResult, DeduplicateAllColumnsAsKey,
DeduplicateKeyColumns, DeleteAction, DeserializeToObject, Except,
FlatMapGroupsWithState, InsertAction, InsertStarAction, Intersect, JoinWith,
LocalRelation, LogicalGroupState, LogicalPlan, MapGroups, MapPartitions,
MergeAction, Project, Sample, SerializeFromObject, Sort, SubqueryAlias,
TimeModes, TransformWithState, TypedFilter, Union, Unpivo [...]
+import org.apache.spark.sql.catalyst.plans.logical.{AppendColumns, Assignment,
CoGroup, CollectMetrics, CommandResult, DeduplicateAllColumnsAsKey,
DeduplicateKeyColumns, DeleteAction, DeserializeToObject, Except,
FlatMapGroupsWithState, InsertAction, InsertStarAction, Intersect, JoinWith,
LocalRelation, LogicalGroupState, LogicalPlan, MapGroups, MapPartitions,
MergeAction, Project, Sample, SerializeFromObject, Sort, SubqueryAlias,
TimeModes, TransformWithState, TypedFilter, Union, Unpivo [...]
import org.apache.spark.sql.catalyst.streaming.InternalOutputModes
import org.apache.spark.sql.catalyst.trees.{CurrentOrigin, Origin, TreePattern}
import org.apache.spark.sql.catalyst.types.DataTypeUtils
@@ -263,6 +263,28 @@ class SparkConnectPlanner(
plan
}
+ /**
+ * Whether [[transformSqlWithRefs]] builds an unresolved plan instead of
executing the SQL. Set
+ * only by [[transformPipelineFlowRelation]].
+ */
+ private var deferSqlWithRefs = false
+
+ /**
+ * Transforms the relation that a pipeline flow returns. Unlike
[[transformRelation]],
+ * [[transformSqlWithRefs]] does not execute a valid SQL-with-references
[[proto.WithRelations]]
+ * during this transformation: it parses the root SQL and binds each
reference to its alias as a
+ * CTE relation. The flow analysis then resolves the pipeline datasets that
the references read,
+ * and records them as inputs of the flow, as it does for a CTE in the SQL
text.
+ */
+ private def transformPipelineFlowRelation(rel: proto.Relation): LogicalPlan
= {
+ deferSqlWithRefs = true
+ try {
+ transformRelation(rel)
+ } finally {
+ deferSqlWithRefs = false
+ }
+ }
+
private def transformRelationPlugin(extension: ProtoAny): LogicalPlan = {
SparkConnectPluginRegistry.relationRegistry
// Lazily traverse the collection.
@@ -413,7 +435,15 @@ class SparkConnectPlanner(
if (!isValidSQLWithRefs(query)) {
throw InvalidInputErrors.invalidSQLWithReferences(query)
}
- executeSQLWithRefs(query).logicalPlan
+ if (deferSqlWithRefs) {
+ val cteRelations = query.getReferencesList.asScala.map { ref =>
+ val alias = ref.getSubqueryAlias.getAlias
+ (alias, SubqueryAlias(alias,
transformRelation(ref.getSubqueryAlias.getInput)), None)
+ }
+ UnresolvedWith(transformSql(query.getRoot.getSql), cteRelations.toSeq)
+ } else {
+ executeSQLWithRefs(query).logicalPlan
+ }
}
private def transformSubqueryAlias(alias: proto.SubqueryAlias): LogicalPlan
= {
@@ -2961,7 +2991,7 @@ class SparkConnectPlanner(
sessionHolder,
command,
responseObserver,
- transformRelation,
+ transformPipelineFlowRelation,
transformExpression)
executeHolder.eventsManager.postFinished()
responseObserver.onNext(
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
index 5d379bd693f5..0303e8d923b1 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
@@ -404,6 +404,76 @@ class PythonPipelineSuite
Set(graphIdentifier("b"), graphIdentifier("d")))
}
+ test("SPARK-60494: referencing internal datasets through DataFrame arguments
of spark.sql") {
+ // `d` and `e` have the SQL with DataFrame arguments under another
relation.
+ val unresolvedGraph = buildGraph("""
+ |@dp.materialized_view
+ |def src():
+ | return spark.range(5)
+ |
+ |@dp.temporary_view
+ |def view_src():
+ | return spark.range(5)
+ |
+ |@dp.materialized_view
+ |def a():
+ | return spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
+ |
+ |@dp.table
+ |def b():
+ | return spark.sql("SELECT * FROM {df}",
df=spark.readStream.table("src"))
+ |
+ |@dp.materialized_view
+ |def c():
+ | return spark.sql("SELECT * FROM {df}",
df=spark.read.table("view_src"))
+ |
+ |@dp.materialized_view
+ |def d():
+ | return spark.sql("SELECT * FROM {df}",
df=spark.read.table("src")).where("id > 0")
+ |
+ |@dp.materialized_view
+ |def e():
+ | inner = spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
+ | return spark.sql("SELECT * FROM {x}", x=inner)
+ |""".stripMargin)
+ val graph =
unresolvedGraph.resolve(sessionCaseSensitive).validate(sessionCaseSensitive)
+
+ // dependency is properly tracked
+ Seq("a", "b", "d", "e").foreach { flowName =>
+ assert(graph.resolvedFlow(graphIdentifier(flowName)).inputs ==
Set(graphIdentifier("src")))
+ }
+ assert(graph.resolvedFlow(graphIdentifier("c")).inputs ==
Set(TableIdentifier("view_src")))
+ assert(!graph.resolvedFlow(graphIdentifier("a")).df.isStreaming)
+ assert(graph.resolvedFlow(graphIdentifier("b")).df.isStreaming)
+
+ val updateContext =
+ new PipelineUpdateContextImpl(unresolvedGraph, _ => (), storageRoot =
storageRoot)
+ updateContext.pipelineExecution.runPipeline()
+ updateContext.pipelineExecution.awaitCompletion()
+ Seq("a", "b", "c", "e").foreach { table =>
+ assert(spark.table(table).collect().map(_.getLong(0)).toSet == (0L until
5L).toSet)
+ }
+ assert(spark.table("d").collect().map(_.getLong(0)).toSet == (1L until
5L).toSet)
+ }
+
+ test(
+ "SPARK-60494: referencing an internal dataset through DataFrame arguments
of spark.sql " +
+ "tracks the dependency when its table already exists") {
+ // The table of `src` exists from a previous run of the pipeline.
+ sql("CREATE TABLE spark_catalog.default.src AS SELECT * FROM RANGE(5)")
+ val graph = buildGraph("""
+ |@dp.materialized_view
+ |def src():
+ | return spark.range(5)
+ |
+ |@dp.materialized_view
+ |def a():
+ | return spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
+
|""".stripMargin).resolve(sessionCaseSensitive).validate(sessionCaseSensitive)
+
+ assert(graph.resolvedFlow(graphIdentifier("a")).inputs ==
Set(graphIdentifier("src")))
+ }
+
test("referencing external datasets") {
sql("CREATE TABLE spark_catalog.default.src AS SELECT * FROM RANGE(5)")
val graph = buildGraph("""
@@ -468,9 +538,13 @@ class PythonPipelineSuite
|@dp.table
|def e():
| return spark.sql("SELECT * FROM STREAM src")
+ |
+ |@dp.materialized_view
+ |def f():
+ | return spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
|""".stripMargin).resolve(sessionCaseSensitive)
- assert(graph.resolutionFailedFlows.size == 5)
+ assert(graph.resolutionFailedFlows.size == 6)
graph.resolutionFailedFlows.foreach { flow =>
assert(flow.failure.head.getMessage.contains("[TABLE_OR_VIEW_NOT_FOUND]"))
assert(flow.failure.head.getMessage.contains("`src`"))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]