This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 1d536e531b33 [SPARK-60498][CONNECT][SDP] Block CTE-prefixed DML and
multi-insert statements in `spark.sql` of SDP
1d536e531b33 is described below
commit 1d536e531b3398bb25b83d519b5b8c596ef5cbcb
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 15:53:36 2026 -0700
[SPARK-60498][CONNECT][SDP] Block CTE-prefixed DML and multi-insert
statements in `spark.sql` of SDP
### What changes were proposed in this pull request?
This PR makes `PipelinesHandler.blockUnsupportedSqlCommand` check the
statements under two parser wrappers at the top of a parsed plan.
- `UnresolvedWith`: `AstBuilder.visitDmlStatement` puts a DML statement
with a CTE prefix under `UnresolvedWith`, for example `WITH v AS (...) INSERT
INTO t SELECT * FROM v`. A CTE prefix is allowed on `INSERT INTO`, `INSERT
OVERWRITE [TABLE]`, `INSERT OVERWRITE [LOCAL] DIRECTORY`, `MERGE INTO`,
`UPDATE`, `DELETE FROM`, and a multi-insert.
- `Union`: `AstBuilder.visitMultiInsertQuery` builds a multi-insert with
two or more `INSERT` clauses, `FROM src INSERT INTO t1 SELECT ... INSERT INTO
t2 SELECT ...`, as a `Union` of its `INSERT`s.
The guard now applies the same check to the child of `UnresolvedWith` and
to each child of `Union`. As a result, these statements fail with
`UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND`, like the same statements without the
wrapper. A read-only query with a CTE or a `UNION` still passes because its
children are queries.
This PR unwraps only these two wrappers instead of checking every node of
the plan. The grammar allows a CTE prefix only on a query or a DML statement,
and a CTE definition can only be a query. A multi-insert is the only statement
that the parser builds as a `Union` of DML statements. So, apart from commands
such as `EXPLAIN` that are already blocked, these are the only nodes that the
parser puts above a DML statement.
### Why are the changes needed?
The SDP guard checks only the class of the top node of the parsed plan. A
DML statement with a CTE prefix and a multi-insert have a non-command node at
the top, so the guard lets them pass. Outside a flow function, `spark.sql(...)`
then runs them eagerly. For example, each of the following lines in a pipeline
definition file inserts rows into `t` while the pipeline is being registered.
```python
spark.sql("WITH v AS (SELECT 1 AS id) INSERT INTO t SELECT * FROM v")
spark.sql("FROM (SELECT 1 AS id) s INSERT INTO t SELECT * INSERT INTO t
SELECT *")
```
Inside a flow function, the guard lets the same statements pass as well.
This is the top-node limitation that was discussed in
https://github.com/apache/spark/pull/59425 and left out of scope in
https://github.com/apache/spark/pull/59719.
### Does this PR introduce _any_ user-facing change?
Yes. These statements in `spark.sql(...)` of SDP now fail with the
following error instead of modifying tables or directories. This is a bug fix
for the SDP SQL guard, which has existed since Apache Spark
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11).
```
[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'UnresolvedInsert' is not
supported in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
```
The error names the class of the wrapped statement, such as
`UnresolvedInsert` or `MergeIntoTable`, like the error for the same statement
without the wrapper.
### How was this patch tested?
Pass the CIs with the newly added test cases in `PythonPipelineSuite`.
- `unsupportedSqlCommandList` has six more statements: `WITH ... INSERT
INTO`, `WITH ... MERGE INTO`, `WITH ... UPDATE`, `WITH ... DELETE FROM`, a
multi-insert, and a multi-insert with a CTE prefix. The last one is parsed into
`UnresolvedWith(Union(...))`, so it is blocked only when the check goes through
both wrappers. Like the existing entries, each of them runs both outside and
inside a pipeline flow function.
- `supportedSqlCommandList` has a read-only CTE query and a `UNION ALL`
query, to make sure that the new check does not block them.
Without the fix, the 12 new unsupported test cases fail. Outside a flow
function, the statement passes the guard and then fails with
`TABLE_OR_VIEW_NOT_FOUND` instead of `UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND`.
Inside a flow function, no exception is thrown. The 4 new supported test cases
pass with and without the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59739 from dongjoon-hyun/SPARK-60498.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit cb40354bda43492635f83c6767ae2e40f6329d66)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/connect/pipelines/PipelinesHandler.scala | 7 ++++++-
.../spark/sql/connect/pipelines/PythonPipelineSuite.scala | 12 +++++++++++-
2 files changed, 17 insertions(+), 2 deletions(-)
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
index 51b308d71729..4265b4589815 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
@@ -31,7 +31,7 @@ import org.apache.spark.sql.{AnalysisException, Column}
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.{PlanWithUnresolvedIdentifier,
UnresolvedAttribute, UnresolvedExecuteImmediate, UnresolvedRelation}
import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.plans.logical.{Command, CreateNamespace,
CreateTable, CreateTableAsSelect, CreateView, DescribeRelation,
DescribeTablePartition, DropView, ExecutableDuringAnalysis, InsertIntoDir,
InsertIntoStatement, LogicalPlan, RenameTable, ShowColumns, ShowCreateTable,
ShowFunctions, ShowTableProperties, ShowTables, ShowViews, UnresolvedInsert}
+import org.apache.spark.sql.catalyst.plans.logical.{Command, CreateNamespace,
CreateTable, CreateTableAsSelect, CreateView, DescribeRelation,
DescribeTablePartition, DropView, ExecutableDuringAnalysis, InsertIntoDir,
InsertIntoStatement, LogicalPlan, RenameTable, ShowColumns, ShowCreateTable,
ShowFunctions, ShowTableProperties, ShowTables, ShowViews, Union,
UnresolvedInsert, UnresolvedWith}
import org.apache.spark.sql.classic.ClassicConversions._
import org.apache.spark.sql.connect.common.DataTypeProtoConverter
import org.apache.spark.sql.connect.service.SessionHolder
@@ -156,6 +156,11 @@ private[connect] object PipelinesHandler extends Logging {
// with a placeholder name and check it instead.
case p: PlanWithUnresolvedIdentifier =>
blockUnsupportedSqlCommand(p.planBuilder(Seq("placeholder"),
p.children))
+ // The parser puts a DML statement with a CTE prefix, such as WITH ...
INSERT INTO ..., under
+ // UnresolvedWith and builds a multi-insert, FROM ... INSERT ... INSERT
..., as a Union of
+ // its INSERTs. Check the statements under them.
+ case w: UnresolvedWith => blockUnsupportedSqlCommand(w.child)
+ case u: Union => u.children.foreach(blockUnsupportedSqlCommand)
case _ =>
}
val allowlistedCommands = Set(
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
index 0303e8d923b1..7878bdd32169 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala
@@ -1623,6 +1623,14 @@ class PythonPipelineSuite
"CREATE VIEW some_view AS SELECT * FROM some_table",
"CREATE TEMPORARY VIEW IDENTIFIER('some_view') AS SELECT 1",
"INSERT INTO some_table VALUES (1)",
+ "WITH v AS (SELECT 1 AS id) INSERT INTO some_table SELECT * FROM v",
+ "WITH v AS (SELECT 1 AS id) MERGE INTO some_table t USING v ON t.id = v.id
" +
+ "WHEN MATCHED THEN DELETE",
+ "WITH v AS (SELECT 1 AS id) UPDATE some_table SET id = 2 WHERE id IN
(SELECT id FROM v)",
+ "WITH v AS (SELECT 1 AS id) DELETE FROM some_table WHERE id IN (SELECT id
FROM v)",
+ "FROM some_view INSERT INTO some_table SELECT * INSERT INTO some_new_table
SELECT *",
+ "WITH v AS (SELECT 1 AS id) FROM v INSERT INTO some_table SELECT * " +
+ "INSERT INTO some_new_table SELECT *",
"ALTER TABLE some_table RENAME TO some_new_table",
"CREATE NAMESPACE some_namespace",
"DROP VIEW some_view",
@@ -1671,7 +1679,9 @@ class PythonPipelineSuite
"SHOW CREATE TABLE spark_catalog.default.src",
"SELECT * FROM RANGE(5)",
"SELECT * FROM spark_catalog.default.src",
- "TABLE IDENTIFIER('spark_catalog.default.' || 'src')")
+ "TABLE IDENTIFIER('spark_catalog.default.' || 'src')",
+ "WITH v AS (SELECT * FROM spark_catalog.default.src) SELECT * FROM v",
+ "SELECT * FROM RANGE(5) UNION ALL SELECT * FROM spark_catalog.default.src")
gridTest("Supported SQL command outside query function should
work")(supportedSqlCommandList) {
supportedSqlCommand =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]