rdtr opened a new pull request, #59425: URL: https://github.com/apache/spark/pull/59425
## What changes were proposed in this pull request? This replaces https://github.com/apache/spark/pull/56241, which was closed as stale without a review. It is rebased on the current master with no other changes. In `SparkConnectPlanner.handleSqlCommand`, the SDP guard previously received a `queryPlan` built by `transformRelation(relation)`. For `WITH_RELATIONS` matching `isValidSQLWithRefs`, that chain routes through `executeSQLWithRefs` → `session.sql(...)` and eagerly executes the root SQL (and any Commands resolved from references), producing a `CommandResult` that bypasses the blocklist. The normal execution path then runs the same query again. This PR introduces a recursive helper `blockUnsupportedSqlCommandIn(rel)` that walks the relation tree and inspects each component the normal handler would execute, without triggering execution. For a valid SQL-with-references `WITH_RELATIONS`, it parses the root SQL with `transformSql` to get an unresolved plan and recursively inspects each reference's `SubqueryAlias.input` so nested `WITH_RELATIONS` cannot re-trigger the executing chain. For other relation types, the behavior is unchanged. ## Why are the changes needed? Two bugs defeat the SDP guard: 1. **Bypassed guard.** The blocklist checks for `Command` subclasses (`CreateTableAsSelect`, `InsertIntoStatement`, etc.), but after execution the plan is wrapped as `CommandResult` which is not on the list. Commands have already mutated state by the time the guard runs. Commands inside reference `SubqueryAlias` inputs also execute because `eagerlyExecuteCommands` walks the resolved plan tree. 2. **Double execution.** After the guard, `handleSqlCommand` falls through to the normal execution path and runs `executeSQLWithRefs` again, causing duplicate side effects. The guard should match the runtime's execution surface: inspect both the root SQL and each reference's input, without triggering execution itself. ## Does this PR introduce _any_ user-facing change? Yes. SDP requests containing blocked Commands inside `WITH_RELATIONS` (at the root or in a reference's input) now fail with `UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND` before any side effects occur. ## How was this patch tested? Added three unit tests in `SparkDeclarativePipelinesServerSuite`: - Command at the root of `WITH_RELATIONS` is blocked, and the target table is asserted not to exist (verifying no side effect). - Command embedded in a `SubqueryAlias` reference input is blocked, and the target table is asserted not to exist. - Valid `SELECT` `WITH_RELATIONS` in pipeline analysis context is allowed (positive case). Without the fix, the first two tests fail because the target table is created as a side effect of the guard. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
