This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f86fc93cc4cb [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame 
arguments in a pipeline flow to flow analysis
f86fc93cc4cb is described below

commit f86fc93cc4cb644f020ef8e78c31e70b052b8e1b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 15:22:49 2026 -0700

    [SPARK-60494][CONNECT][SDP] Defer SQL with DataFrame arguments in a 
pipeline flow to flow analysis
    
    ### What changes were proposed in this pull request?
    
    When `PipelinesHandler` handles a `DefineFlow` command, it transforms the 
relation of the flow with `SparkConnectPlanner.transformRelation`. 
`spark.sql(query, df=...)` with DataFrame arguments sends a SQL-with-references 
`WITH_RELATIONS` relation, and `transformRelation` handles it with 
`executeSQLWithRefs`. That method analyzes each reference, registers it as a 
temporary view, and runs the root SQL with `session.sql`, all at flow 
registration.
    
    This PR adds `SparkConnectPlanner.transformPipelineFlowRelation` and passes 
it to `PipelinesHandler` instead of `transformRelation`. While it transforms 
the relation of a flow, `transformSqlWithRefs` does not execute a valid 
SQL-with-references `WITH_RELATIONS`. It parses the root SQL and binds each 
reference to its alias as a CTE relation (`UnresolvedWith`). `FlowAnalysis` 
already substitutes CTEs before it resolves `UnresolvedRelation`s as pipeline 
inputs, so it handles a dataset th [...]
    
    `handleSqlCommand` and all other Spark Connect requests still use 
`executeSQLWithRefs`, so `spark.sql` with DataFrame arguments outside of a flow 
definition is unchanged.
    
    This PR does not change the SDP guard in `handleSqlCommand`. For a 
reference input that is not a SQL-with-references `WITH_RELATIONS` itself, the 
guard still uses `transformRelation`, which executes any SQL-with-references 
`WITH_RELATIONS` nested in that input. SPARK-57189 documented this as a known 
limitation of `blockUnsupportedSqlCommandIn`. As a result, if a flow function 
passes a further-transformed `spark.sql(..., df=...)` result, such as 
`inner.where("id > 0")`, as a DataFrame  [...]
    
    ### Why are the changes needed?
    
    Inside a flow function, `spark.sql` does not run the SQL. The server 
returns the relation as is (SPARK-54452), so that the flow analysis can resolve 
it later. With DataFrame arguments, however, `DefineFlow` analyzes it right 
away:
    
    1. The flow fails to register when a DataFrame argument reads a dataset of 
the same pipeline that does not exist in the catalog: a table or materialized 
view on the first run, or a temporary view on every run.
    
        ```python
        dp.materialized_view
        def upstream():
            return spark.range(5)
    
        dp.materialized_view
        def downstream():
            return spark.sql("SELECT * FROM {df}", 
df=spark.read.table("upstream"))
        ```
    
        ```
        pyspark.errors.exceptions.connect.AnalysisException: 
[TABLE_OR_VIEW_NOT_FOUND] The table or view `upstream` cannot be found. ...
        ```
    
    2. When the table already exists, for example from a previous run, the flow 
registers an analyzed plan. The flow analysis then does not see the read of 
`upstream` and does not record it as an input of `downstream`. Without the 
dependency, `downstream` can run before `upstream` is refreshed and read stale 
data, without an error.
    
    3. A command at the root that the SDP guard allows, such as `SHOW TABLES`, 
is executed at flow registration.
    
    The same query without DataFrame arguments, `spark.sql("SELECT * FROM 
upstream")`, works because its relation is only parsed at flow registration.
    
    In Spark 4.1.0 and 4.2.0, the SDP guard in `handleSqlCommand` also analyzed 
the relation, so the error in (1) was raised at the `spark.sql` call in the 
flow function. SPARK-57189 (#59425) moved it to flow registration.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. In a flow function, `spark.sql` with DataFrame arguments can read 
datasets of the same pipeline, and the flow analysis records them as inputs of 
the flow. The SQL is analyzed during graph resolution instead of flow 
registration, so an analysis error is reported as a flow resolution failure, as 
for `spark.sql` without DataFrame arguments.
    
    In a flow definition, DataFrame arguments are now bound as CTE relations 
instead of temporary views. A statement that refers to a DataFrame argument 
other than as a table in a query, such as `DESCRIBE TABLE {df}`, can no longer 
resolve it when a flow function returns the statement.
    
    ### How was this patch tested?
    
    Added two tests to `PythonPipelineSuite`, and a flow to an existing test 
there. All of them fail without this fix.
    
    - `SPARK-60494: referencing internal datasets through DataFrame arguments 
of spark.sql`: a new pipeline reads a materialized view with batch and 
streaming reads, and a temporary view with a batch read, all through DataFrame 
arguments. Two more flows have the SQL with DataFrame arguments under another 
relation: one applies `where` to the result, and the other passes the result as 
a DataFrame argument of another `spark.sql` call. The test checks the inputs 
and streaming-ness of the flow [...]
    - `SPARK-60494: referencing an internal dataset through DataFrame arguments 
of spark.sql tracks the dependency when its table already exists`: Without the 
fix, the inputs of the flow are empty.
    - `referencing internal datasets failed`: a new flow reads an undefined 
dataset through a DataFrame argument. Like the other flows in the test, it 
fails graph resolution with `TABLE_OR_VIEW_NOT_FOUND` for `src`. Without the 
fix, its registration fails.
    
    Also passed the existing suites in 
`org.apache.spark.sql.connect.pipelines`, including `PythonPipelineSuite` and 
`SparkDeclarativePipelinesServerSuite`, as well as `SparkConnectPlannerSuite` 
and `SparkConnectProtoSuite`:
    
    ```
    build/sbt 'connect/testOnly org.apache.spark.sql.connect.pipelines.* 
*SparkConnectPlannerSuite *SparkConnectProtoSuite'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59735 from dongjoon-hyun/SPARK-60494.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../sql/connect/planner/SparkConnectPlanner.scala  | 38 ++++++++++-
 .../connect/pipelines/PythonPipelineSuite.scala    | 76 +++++++++++++++++++++-
 2 files changed, 110 insertions(+), 4 deletions(-)

diff --git 
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
 
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
index a387409d532e..28efd70d44be 100644
--- 
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
+++ 
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
@@ -52,7 +52,7 @@ import 
org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
 import org.apache.spark.sql.catalyst.parser.{NamedParameterContext, 
ParameterContext, ParseException, ParserUtils, PositionalParameterContext}
 import org.apache.spark.sql.catalyst.plans.{Cross, FullOuter, Inner, JoinType, 
LeftAnti, LeftOuter, LeftSemi, RightOuter, UsingJoin}
 import org.apache.spark.sql.catalyst.plans.logical
-import org.apache.spark.sql.catalyst.plans.logical.{AppendColumns, Assignment, 
BinByOutputAliases, CoGroup, CollectMetrics, CommandResult, 
DeduplicateAllColumnsAsKey, DeduplicateKeyColumns, DeleteAction, 
DeserializeToObject, Except, FlatMapGroupsWithState, InsertAction, 
InsertStarAction, Intersect, JoinWith, LocalRelation, LogicalGroupState, 
LogicalPlan, MapGroups, MapPartitions, MergeAction, Project, Sample, 
SerializeFromObject, Sort, SubqueryAlias, TimeModes, TransformWithState, TypedF 
[...]
+import org.apache.spark.sql.catalyst.plans.logical.{AppendColumns, Assignment, 
BinByOutputAliases, CoGroup, CollectMetrics, CommandResult, 
DeduplicateAllColumnsAsKey, DeduplicateKeyColumns, DeleteAction, 
DeserializeToObject, Except, FlatMapGroupsWithState, InsertAction, 
InsertStarAction, Intersect, JoinWith, LocalRelation, LogicalGroupState, 
LogicalPlan, MapGroups, MapPartitions, MergeAction, Project, Sample, 
SerializeFromObject, Sort, SubqueryAlias, TimeModes, TransformWithState, TypedF 
[...]
 import org.apache.spark.sql.catalyst.streaming.InternalOutputModes
 import org.apache.spark.sql.catalyst.trees.{CurrentOrigin, Origin, TreePattern}
 import org.apache.spark.sql.catalyst.types.DataTypeUtils
@@ -264,6 +264,28 @@ class SparkConnectPlanner(
     plan
   }
 
+  /**
+   * Whether [[transformSqlWithRefs]] builds an unresolved plan instead of 
executing the SQL. Set
+   * only by [[transformPipelineFlowRelation]].
+   */
+  private var deferSqlWithRefs = false
+
+  /**
+   * Transforms the relation that a pipeline flow returns. Unlike 
[[transformRelation]],
+   * [[transformSqlWithRefs]] does not execute a valid SQL-with-references 
[[proto.WithRelations]]
+   * during this transformation: it parses the root SQL and binds each 
reference to its alias as a
+   * CTE relation. The flow analysis then resolves the pipeline datasets that 
the references read,
+   * and records them as inputs of the flow, as it does for a CTE in the SQL 
text.
+   */
+  private def transformPipelineFlowRelation(rel: proto.Relation): LogicalPlan 
= {
+    deferSqlWithRefs = true
+    try {
+      transformRelation(rel)
+    } finally {
+      deferSqlWithRefs = false
+    }
+  }
+
   private def transformRelationPlugin(extension: ProtoAny): LogicalPlan = {
     SparkConnectPluginRegistry.relationRegistry
       // Lazily traverse the collection.
@@ -414,7 +436,17 @@ class SparkConnectPlanner(
     if (!isValidSQLWithRefs(query)) {
       throw InvalidInputErrors.invalidSQLWithReferences(query)
     }
-    executeSQLWithRefs(query).logicalPlan
+    if (deferSqlWithRefs) {
+      val cteRelations = query.getReferencesList.asScala.map { ref =>
+        val alias = ref.getSubqueryAlias.getAlias
+        UnresolvedCTERelation(
+          alias,
+          SubqueryAlias(alias, 
transformRelation(ref.getSubqueryAlias.getInput)))
+      }
+      UnresolvedWith(transformSql(query.getRoot.getSql), cteRelations.toSeq)
+    } else {
+      executeSQLWithRefs(query).logicalPlan
+    }
   }
 
   private def transformSubqueryAlias(alias: proto.SubqueryAlias): LogicalPlan 
= {
@@ -3100,7 +3132,7 @@ class SparkConnectPlanner(
       sessionHolder,
       command,
       responseObserver,
-      transformRelation,
+      transformPipelineFlowRelation,
       transformExpression)
     executeHolder.eventsManager.postFinished()
     responseObserver.onNext(
diff --git 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
index 3cf2295235ae..d196e3c5ebe4 100644
--- 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
+++ 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
@@ -404,6 +404,76 @@ class PythonPipelineSuite
         Set(graphIdentifier("b"), graphIdentifier("d")))
   }
 
+  test("SPARK-60494: referencing internal datasets through DataFrame arguments 
of spark.sql") {
+    // `d` and `e` have the SQL with DataFrame arguments under another 
relation.
+    val unresolvedGraph = buildGraph("""
+      |@dp.materialized_view
+      |def src():
+      |  return spark.range(5)
+      |
+      |@dp.temporary_view
+      |def view_src():
+      |  return spark.range(5)
+      |
+      |@dp.materialized_view
+      |def a():
+      |  return spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
+      |
+      |@dp.table
+      |def b():
+      |  return spark.sql("SELECT * FROM {df}", 
df=spark.readStream.table("src"))
+      |
+      |@dp.materialized_view
+      |def c():
+      |  return spark.sql("SELECT * FROM {df}", 
df=spark.read.table("view_src"))
+      |
+      |@dp.materialized_view
+      |def d():
+      |  return spark.sql("SELECT * FROM {df}", 
df=spark.read.table("src")).where("id > 0")
+      |
+      |@dp.materialized_view
+      |def e():
+      |  inner = spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
+      |  return spark.sql("SELECT * FROM {x}", x=inner)
+      |""".stripMargin)
+    val graph = 
unresolvedGraph.resolve(sessionCaseSensitive).validate(sessionCaseSensitive)
+
+    // dependency is properly tracked
+    Seq("a", "b", "d", "e").foreach { flowName =>
+      assert(graph.resolvedFlow(graphIdentifier(flowName)).inputs == 
Set(graphIdentifier("src")))
+    }
+    assert(graph.resolvedFlow(graphIdentifier("c")).inputs == 
Set(TableIdentifier("view_src")))
+    assert(!graph.resolvedFlow(graphIdentifier("a")).df.isStreaming)
+    assert(graph.resolvedFlow(graphIdentifier("b")).df.isStreaming)
+
+    val updateContext =
+      new PipelineUpdateContextImpl(unresolvedGraph, _ => (), storageRoot = 
storageRoot)
+    updateContext.pipelineExecution.runPipeline()
+    updateContext.pipelineExecution.awaitCompletion()
+    Seq("a", "b", "c", "e").foreach { table =>
+      assert(spark.table(table).collect().map(_.getLong(0)).toSet == (0L until 
5L).toSet)
+    }
+    assert(spark.table("d").collect().map(_.getLong(0)).toSet == (1L until 
5L).toSet)
+  }
+
+  test(
+    "SPARK-60494: referencing an internal dataset through DataFrame arguments 
of spark.sql " +
+      "tracks the dependency when its table already exists") {
+    // The table of `src` exists from a previous run of the pipeline.
+    sql("CREATE TABLE spark_catalog.default.src AS SELECT * FROM RANGE(5)")
+    val graph = buildGraph("""
+      |@dp.materialized_view
+      |def src():
+      |  return spark.range(5)
+      |
+      |@dp.materialized_view
+      |def a():
+      |  return spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
+      
|""".stripMargin).resolve(sessionCaseSensitive).validate(sessionCaseSensitive)
+
+    assert(graph.resolvedFlow(graphIdentifier("a")).inputs == 
Set(graphIdentifier("src")))
+  }
+
   test("referencing external datasets") {
     sql("CREATE TABLE spark_catalog.default.src AS SELECT * FROM RANGE(5)")
     val graph = buildGraph("""
@@ -468,9 +538,13 @@ class PythonPipelineSuite
         |@dp.table
         |def e():
         |  return spark.sql("SELECT * FROM STREAM src")
+        |
+        |@dp.materialized_view
+        |def f():
+        |  return spark.sql("SELECT * FROM {df}", df=spark.read.table("src"))
         |""".stripMargin).resolve(sessionCaseSensitive)
 
-    assert(graph.resolutionFailedFlows.size == 5)
+    assert(graph.resolutionFailedFlows.size == 6)
     graph.resolutionFailedFlows.foreach { flow =>
       
assert(flow.failure.head.getMessage.contains("[TABLE_OR_VIEW_NOT_FOUND]"))
       assert(flow.failure.head.getMessage.contains("`src`"))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to