This is an automated email from the ASF dual-hosted git repository.

peter-toth pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new b74863540ff3 [SPARK-58370][4.0][SQL] Check table write privileges when 
the write target is also read in the same statement
b74863540ff3 is described below

commit b74863540ff30c19e19110bc9340f74a718f0d57
Author: Peter Toth <[email protected]>
AuthorDate: Tue Jul 28 19:32:25 2026 +0200

    [SPARK-58370][4.0][SQL] Check table write privileges when the write target 
is also read in the same statement
    
    ### What changes were proposed in this pull request?
    
    Backport of #57563 to `branch-4.0`.
    
    `RelationResolution.resolveRelation` answers a hit in the per-query 
`AnalysisContext.relationCache` before reaching the branch that calls 
`CatalogV2Util.loadTable(catalog, ident, timeTravelSpec, writePrivileges)`. 
This PR stops serving a reference that carries `REQUIRED_WRITE_PRIVILEGES` from 
that cache, so a write target always goes through 
`TableCatalog.loadTable(ident, writePrivileges)`:
    
    ```scala
    val writePrivilegesString =
      Option(u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES))
    val cached = if (writePrivilegesString.isEmpty) {
      AnalysisContext.get.relationCache.get(key)
    } else {
      None
    }
    cached.map { cache => ... }.orElse { ... }
    ```
    
    Loaded relations are still *stored* in the cache, so a self-MERGE's source 
keeps sharing the target's `Table` instance (one snapshot).
    
    ### Backport notes
    
    The cherry-pick of `a1c3f310a8c` does not apply cleanly, so the change was 
re-applied by hand:
    
    - On `master` the affected code lives in 
`RelationResolution.tryResolvePersistent`, extracted alongside 
`relationResolutionSteps` and the `RelationCatalog.loadRelation` fast path. 
None of that exists on `branch-4.0`, where the same cache-then-load sequence 
still sits inline in `resolveRelation`.
    - `branch-4.0` also predates the `toCacheKey` / `adaptCachedRelation` 
helpers and the shared relation cache, so the cache key and the `PLAN_ID_TAG` 
handling are still written out inline. Those parts are left untouched.
    - The local is `writePrivilegesString: Option[String]` here, not a raw 
`String`, so the guard reads `if (writePrivilegesString.isEmpty)` rather than 
`if (writePrivileges == null)`.
    - The comment block is wrapped for this file's indentation.
    
    The test hunk applies unchanged.
    
    ### Why are the changes needed?
    
    `TableCatalog.loadTable(Identifier, Set<TableWritePrivilege>)` (`since 
3.5.3`) is where a catalog authorizes a write; the only other caller in the 
tree is `ResolveSchemaEvolution`. A write target hits the relation cache 
whenever another reference to the same table was resolved earlier in the same 
analyzer run, which is exactly the case for `InsertIntoStatement` and 
`V2WriteCommand` - their target is resolved at the command node *after* its 
query. So the catalog is never asked to autho [...]
    
    ```sql
    INSERT INTO t SELECT * FROM t
    INSERT OVERWRITE t SELECT * FROM t WHERE 1 = 0   -- empties the table
    INSERT INTO t REPLACE WHERE i = 0 SELECT * FROM t
    INSERT INTO t SELECT i FROM (SELECT i FROM t) x
    WITH c AS (SELECT i FROM t) INSERT INTO t SELECT i FROM c
    ```
    
    ```scala
    spark.table("t").writeTo("t").append()            // DataFrameWriterV2 
passes the unanalyzed plan
    ```
    
    `INSERT INTO t SELECT 1` on the same table is authorized normally, and 
`UPDATE` / `DELETE` / `MERGE` are unaffected (their target is a plain child, 
resolved before the source, so it is a cache miss), as is 
`DataFrameWriter.insertInto` (it passes the already-analyzed plan).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. On a catalog that enforces privileges in `loadTable(ident, 
writePrivileges)`, the statements above are now authorized like every other 
write, so a user without the required privilege gets the catalog's error 
instead of silently writing. Catalogs that ignore the privileges argument 
(including Spark's built-in `V2SessionCatalog`) are unaffected. The write 
target of a self-referencing INSERT is now loaded once more, which is what a 
non-self-referencing INSERT already does.
    
    ### How was this patch tested?
    
    New test in `DataSourceV2SQLSuite`, identical to the one on `master`. It 
fails on `branch-4.0` without the fix ("Expected exception 
java.lang.RuntimeException to be thrown, but no exception was thrown") and 
passes with it.
    
    Also green on this branch: 17 suites / 861 tests - `DataSourceV2SQLSuite` 
(V1+V2 filter), `DataSourceV2DataFrameSuite`, `DataSourceV2OptionSuite`, 
`PlanResolutionSuite`, `CachedTableSuite`, `*ViewTestSuite`, group/delta-based 
MERGE/UPDATE/DELETE suites, `MergeIntoDataFrameSuite`, `InsertSuite`. 
`dev/lint-scala` clean.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5)
    
    Closes #57600 from 
peter-toth/SPARK-58370-write-privileges-self-reference-4.0.
    
    Authored-by: Peter Toth <[email protected]>
    Signed-off-by: Peter Toth <[email protected]>
---
 .../sql/catalyst/analysis/RelationResolution.scala | 18 +++++--
 .../spark/sql/connector/DataSourceV2SQLSuite.scala | 59 ++++++++++++++++++++++
 2 files changed, 73 insertions(+), 4 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
index 08be456f090e..83cba9880740 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
@@ -112,8 +112,20 @@ class RelationResolution(override val catalogManager: 
CatalogManager)
               (catalog.name +: ident.namespace :+ 
ident.name).toImmutableArraySeq,
               finalTimeTravelSpec
             )
-          AnalysisContext.get.relationCache
-            .get(key)
+          val writePrivilegesString =
+            Option(u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES))
+          // A reference that requires write privileges is never served from 
the per-query relation
+          // cache. The catalog authorizes the write in `loadTable(ident, 
writePrivileges)` below,
+          // and a cache hit would skip that call entirely. The hit happens 
whenever the write
+          // target is also read in the same statement -- the target is 
resolved after its query
+          // (see `ResolveRelations`), so it finds the relation the query 
already put in the cache,
+          // e.g. for `INSERT INTO t SELECT * FROM t`.
+          val cached = if (writePrivilegesString.isEmpty) {
+            AnalysisContext.get.relationCache.get(key)
+          } else {
+            None
+          }
+          cached
             .map { cache =>
               val cachedRelation = cache.transform {
                 case multi: MultiInstanceRelation =>
@@ -130,8 +142,6 @@ class RelationResolution(override val catalogManager: 
CatalogManager)
                 .getOrElse(cachedRelation)
             }
             .orElse {
-              val writePrivilegesString =
-                
Option(u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES))
               val table =
                 CatalogV2Util.loadTable(catalog, ident, finalTimeTravelSpec, 
writePrivilegesString)
               val loaded = createRelation(
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
index 8d255e9efda5..383b8a2d3444 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
@@ -3783,6 +3783,65 @@ class DataSourceV2SQLSuiteV1Filter
     checkWriteOperations("read_only_cat")
   }
 
+  test("SPARK-58370: read-only catalog, write whose query reads the target 
table") {
+    def assertPrivilegeError(f: => Unit, privileges: String): Unit = {
+      val e = intercept[RuntimeException](f)
+      assert(e.getMessage == s"cannot write with $privileges")
+    }
+
+    def checkWritesReadingTheTarget(catalog: String): Unit = {
+      withSQLConf(s"spark.sql.catalog.$catalog" -> 
classOf[ReadOnlyCatalog].getName) {
+        val tbl = s"$catalog.default.t"
+        withTable(tbl) {
+          sql(s"CREATE TABLE $tbl (i INT)")
+
+          // The write target is resolved after its query, so it used to find 
the query's
+          // relation in the per-query relation cache and skip 
`loadTable(ident, writePrivileges)`.
+          assertPrivilegeError(sql(s"INSERT INTO $tbl SELECT * FROM $tbl"), 
"INSERT")
+          assertPrivilegeError(
+            sql(s"INSERT OVERWRITE $tbl SELECT * FROM $tbl"), "DELETE,INSERT")
+          assertPrivilegeError(
+            sql(s"INSERT OVERWRITE $tbl SELECT * FROM $tbl WHERE 1 = 0"), 
"DELETE,INSERT")
+          assertPrivilegeError(
+            sql(s"INSERT INTO $tbl REPLACE WHERE i = 0 SELECT * FROM $tbl"), 
"DELETE,INSERT")
+          // The reference to the target can be anywhere in the query.
+          assertPrivilegeError(
+            sql(s"INSERT INTO $tbl SELECT i FROM (SELECT i FROM $tbl) x"), 
"INSERT")
+          assertPrivilegeError(
+            sql(s"WITH c AS (SELECT i FROM $tbl) INSERT INTO $tbl SELECT i 
FROM c"), "INSERT")
+
+          // `DataFrameWriterV2` passes the unanalyzed query plan, so its 
target is resolved in the
+          // same analyzer run as the query's reference to the same table.
+          assertPrivilegeError(sql(s"SELECT * FROM 
$tbl").writeTo(tbl).append(), "INSERT")
+          assertPrivilegeError(
+            sql(s"SELECT * FROM $tbl").writeTo(tbl).overwritePartitions(), 
"DELETE,INSERT")
+          assertPrivilegeError(sql(s"SELECT * FROM 
$tbl").write.insertInto(tbl), "INSERT")
+
+          // Row-level operations resolve the target before the source / 
subquery, so these were
+          // already checked; they are here to keep both orders covered.
+          assertPrivilegeError(
+            sql(s"UPDATE $tbl SET i = 0 WHERE i IN (SELECT i FROM $tbl)"), 
"UPDATE")
+          assertPrivilegeError(
+            sql(s"DELETE FROM $tbl WHERE i IN (SELECT i FROM $tbl)"), "DELETE")
+          assertPrivilegeError(
+            sql(
+              s"""
+                 |MERGE INTO $tbl USING (SELECT i FROM $tbl) AS source
+                 |ON source.i = $tbl.i
+                 |WHEN MATCHED THEN UPDATE SET *
+                 |""".stripMargin),
+            "UPDATE")
+        }
+      }
+    }
+
+    // Reset CatalogManager to clear the materialized `spark_catalog` 
instance, so that we can
+    // configure a new implementation.
+    spark.sessionState.catalogManager.reset()
+    checkWritesReadingTheTarget(SESSION_CATALOG_NAME)
+    checkWritesReadingTheTarget("read_only_self_ref_cat")
+  }
+
   test("StagingTableCatalog without atomic support") {
     withSQLConf("spark.sql.catalog.fakeStagedCat" -> 
classOf[FakeStagedTableCatalog].getName) {
       withTable("fakeStagedCat.t") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to