This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 709463440436 [SPARK-60498][CONNECT][SDP] Block CTE-prefixed DML and 
multi-insert statements in `spark.sql` of SDP
709463440436 is described below

commit 7094634404364f7f1095083843c31d45ba7bb928
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 15:53:36 2026 -0700

    [SPARK-60498][CONNECT][SDP] Block CTE-prefixed DML and multi-insert 
statements in `spark.sql` of SDP
    
    ### What changes were proposed in this pull request?
    
    This PR makes `PipelinesHandler.blockUnsupportedSqlCommand` check the 
statements under two parser wrappers at the top of a parsed plan.
    
    - `UnresolvedWith`: `AstBuilder.visitDmlStatement` puts a DML statement 
with a CTE prefix under `UnresolvedWith`, for example `WITH v AS (...) INSERT 
INTO t SELECT * FROM v`. A CTE prefix is allowed on `INSERT INTO`, `INSERT 
OVERWRITE [TABLE]`, `INSERT OVERWRITE [LOCAL] DIRECTORY`, `MERGE INTO`, 
`UPDATE`, `DELETE FROM`, and a multi-insert.
    - `Union`: `AstBuilder.visitMultiInsertQuery` builds a multi-insert with 
two or more `INSERT` clauses, `FROM src INSERT INTO t1 SELECT ... INSERT INTO 
t2 SELECT ...`, as a `Union` of its `INSERT`s.
    
    The guard now applies the same check to the child of `UnresolvedWith` and 
to each child of `Union`. As a result, these statements fail with 
`UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND`, like the same statements without the 
wrapper. A read-only query with a CTE or a `UNION` still passes because its 
children are queries.
    
    This PR unwraps only these two wrappers instead of checking every node of 
the plan. The grammar allows a CTE prefix only on a query or a DML statement, 
and a CTE definition can only be a query. A multi-insert is the only statement 
that the parser builds as a `Union` of DML statements. So, apart from commands 
such as `EXPLAIN` that are already blocked, these are the only nodes that the 
parser puts above a DML statement.
    
    ### Why are the changes needed?
    
    The SDP guard checks only the class of the top node of the parsed plan. A 
DML statement with a CTE prefix and a multi-insert have a non-command node at 
the top, so the guard lets them pass. Outside a flow function, `spark.sql(...)` 
then runs them eagerly. For example, each of the following lines in a pipeline 
definition file inserts rows into `t` while the pipeline is being registered.
    
    ```python
    spark.sql("WITH v AS (SELECT 1 AS id) INSERT INTO t SELECT * FROM v")
    spark.sql("FROM (SELECT 1 AS id) s INSERT INTO t SELECT * INSERT INTO t 
SELECT *")
    ```
    
    Inside a flow function, the guard lets the same statements pass as well.
    
    This is the top-node limitation that was discussed in 
https://github.com/apache/spark/pull/59425 and left out of scope in 
https://github.com/apache/spark/pull/59719.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These statements in `spark.sql(...)` of SDP now fail with the 
following error instead of modifying tables or directories. This is a bug fix 
for the SDP SQL guard, which has existed since Apache Spark 
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11).
    
    ```
    [UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'UnresolvedInsert' is not 
supported in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
    ```
    
    The error names the class of the wrapped statement, such as 
`UnresolvedInsert` or `MergeIntoTable`, like the error for the same statement 
without the wrapper.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases in `PythonPipelineSuite`.
    
    - `unsupportedSqlCommandList` has six more statements: `WITH ... INSERT 
INTO`, `WITH ... MERGE INTO`, `WITH ... UPDATE`, `WITH ... DELETE FROM`, a 
multi-insert, and a multi-insert with a CTE prefix. The last one is parsed into 
`UnresolvedWith(Union(...))`, so it is blocked only when the check goes through 
both wrappers. Like the existing entries, each of them runs both outside and 
inside a pipeline flow function.
    - `supportedSqlCommandList` has a read-only CTE query and a `UNION ALL` 
query, to make sure that the new check does not block them.
    
    Without the fix, the 12 new unsupported test cases fail. Outside a flow 
function, the statement passes the guard and then fails with 
`TABLE_OR_VIEW_NOT_FOUND` instead of `UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND`. 
Inside a flow function, no exception is thrown. The 4 new supported test cases 
pass with and without the fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59739 from dongjoon-hyun/SPARK-60498.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit cb40354bda43492635f83c6767ae2e40f6329d66)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/connect/pipelines/PipelinesHandler.scala       |  7 ++++++-
 .../spark/sql/connect/pipelines/PythonPipelineSuite.scala    | 12 +++++++++++-
 2 files changed, 17 insertions(+), 2 deletions(-)

diff --git 
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
 
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
index a7124fc0a1d7..64db31d59e7c 100644
--- 
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
+++ 
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
@@ -31,7 +31,7 @@ import org.apache.spark.sql.{AnalysisException, Column}
 import org.apache.spark.sql.catalyst.TableIdentifier
 import org.apache.spark.sql.catalyst.analysis.{PlanWithUnresolvedIdentifier, 
UnresolvedAttribute, UnresolvedExecuteImmediate, UnresolvedRelation}
 import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.plans.logical.{Command, CreateNamespace, 
CreateTable, CreateTableAsSelect, CreateView, DescribeRelation, 
DescribeTablePartition, DropView, ExecutableDuringAnalysis, InsertIntoDir, 
InsertIntoStatement, LogicalPlan, RenameTable, ShowColumns, ShowCreateTable, 
ShowFunctions, ShowTableProperties, ShowTables, ShowViews, UnresolvedInsert}
+import org.apache.spark.sql.catalyst.plans.logical.{Command, CreateNamespace, 
CreateTable, CreateTableAsSelect, CreateView, DescribeRelation, 
DescribeTablePartition, DropView, ExecutableDuringAnalysis, InsertIntoDir, 
InsertIntoStatement, LogicalPlan, RenameTable, ShowColumns, ShowCreateTable, 
ShowFunctions, ShowTableProperties, ShowTables, ShowViews, Union, 
UnresolvedInsert, UnresolvedWith}
 import org.apache.spark.sql.classic.ClassicConversions._
 import org.apache.spark.sql.connect.common.DataTypeProtoConverter
 import org.apache.spark.sql.connect.service.SessionHolder
@@ -156,6 +156,11 @@ private[connect] object PipelinesHandler extends Logging {
       // with a placeholder name and check it instead.
       case p: PlanWithUnresolvedIdentifier =>
         blockUnsupportedSqlCommand(p.planBuilder(Seq("placeholder"), 
p.children))
+      // The parser puts a DML statement with a CTE prefix, such as WITH ... 
INSERT INTO ..., under
+      // UnresolvedWith and builds a multi-insert, FROM ... INSERT ... INSERT 
..., as a Union of
+      // its INSERTs. Check the statements under them.
+      case w: UnresolvedWith => blockUnsupportedSqlCommand(w.child)
+      case u: Union => u.children.foreach(blockUnsupportedSqlCommand)
       case _ =>
     }
     val allowlistedCommands = Set(
diff --git 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
index d196e3c5ebe4..951dfd9595d0 100644
--- 
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
+++ 
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
@@ -1709,6 +1709,14 @@ class PythonPipelineSuite
     "CREATE VIEW some_view AS SELECT * FROM some_table",
     "CREATE TEMPORARY VIEW IDENTIFIER('some_view') AS SELECT 1",
     "INSERT INTO some_table VALUES (1)",
+    "WITH v AS (SELECT 1 AS id) INSERT INTO some_table SELECT * FROM v",
+    "WITH v AS (SELECT 1 AS id) MERGE INTO some_table t USING v ON t.id = v.id 
" +
+      "WHEN MATCHED THEN DELETE",
+    "WITH v AS (SELECT 1 AS id) UPDATE some_table SET id = 2 WHERE id IN 
(SELECT id FROM v)",
+    "WITH v AS (SELECT 1 AS id) DELETE FROM some_table WHERE id IN (SELECT id 
FROM v)",
+    "FROM some_view INSERT INTO some_table SELECT * INSERT INTO some_new_table 
SELECT *",
+    "WITH v AS (SELECT 1 AS id) FROM v INSERT INTO some_table SELECT * " +
+      "INSERT INTO some_new_table SELECT *",
     "ALTER TABLE some_table RENAME TO some_new_table",
     "CREATE NAMESPACE some_namespace",
     "DROP VIEW some_view",
@@ -1757,7 +1765,9 @@ class PythonPipelineSuite
     "SHOW CREATE TABLE spark_catalog.default.src",
     "SELECT * FROM RANGE(5)",
     "SELECT * FROM spark_catalog.default.src",
-    "TABLE IDENTIFIER('spark_catalog.default.' || 'src')")
+    "TABLE IDENTIFIER('spark_catalog.default.' || 'src')",
+    "WITH v AS (SELECT * FROM spark_catalog.default.src) SELECT * FROM v",
+    "SELECT * FROM RANGE(5) UNION ALL SELECT * FROM spark_catalog.default.src")
 
   gridTest("Supported SQL command outside query function should 
work")(supportedSqlCommandList) {
     supportedSqlCommand =>


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to