This is an automated email from the ASF dual-hosted git repository.

peter-toth pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new d297b481bc05 [SPARK-58370][SQL] Check table write privileges when the 
write target is also read in the same statement
d297b481bc05 is described below

commit d297b481bc05419e5bfc3e9e289eec675ad4fd62
Author: Peter Toth <[email protected]>
AuthorDate: Tue Jul 28 09:43:06 2026 +0200

    [SPARK-58370][SQL] Check table write privileges when the write target is 
also read in the same statement
    
    ### What changes were proposed in this pull request?
    
    `RelationResolution.tryResolvePersistent` answers a hit in the per-query 
`AnalysisContext.relationCache` before reaching the branch that calls 
`CatalogV2Util.loadTable(catalog, ident, timeTravelSpec, 
Option(writePrivileges))`. This PR stops serving a reference that carries 
`REQUIRED_WRITE_PRIVILEGES` from that cache, so a write target always goes 
through `TableCatalog.loadTable(ident, writePrivileges)`:
    
    ```scala
    val writePrivileges = 
u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES)
    val cached = if (writePrivileges == null) relationCache.get(key) else None
    cached.map(adaptCachedRelation(_, planId)).orElse { ... }
    ```
    
    Loaded relations are still *stored* in the cache, so a self-MERGE's source 
keeps sharing the target's `Table` instance (one snapshot). `V2TableReference` 
already treats write targets this way 
(`V2TableReference.WriteTargetContext.cacheable = false`).
    
    ### Why are the changes needed?
    
    `TableCatalog.loadTable(Identifier, Set<TableWritePrivilege>)` (`since 
3.5.3`) is where a catalog authorizes a write; the only other caller in the 
tree is `ResolveSchemaEvolution`. A write target hits the relation cache 
whenever another reference to the same table was resolved earlier in the same 
analyzer run, which is exactly the case for `InsertIntoStatement` and 
`V2WriteCommand` - their target is resolved at the command node *after* its 
query. So the catalog is never asked to autho [...]
    
    ```sql
    INSERT INTO t SELECT * FROM t
    INSERT OVERWRITE t SELECT * FROM t WHERE 1 = 0   -- empties the table
    INSERT INTO t REPLACE WHERE i = 0 SELECT * FROM t
    INSERT INTO t SELECT i FROM (SELECT i FROM t) x
    WITH c AS (SELECT i FROM t) INSERT INTO t SELECT i FROM c
    ```
    
    ```scala
    spark.table("t").writeTo("t").append()            // DataFrameWriterV2 
passes the unanalyzed plan
    ```
    
    `INSERT INTO t SELECT 1` on the same table is authorized normally, and 
`UPDATE` / `DELETE` / `MERGE` are unaffected (their target is a plain child, 
resolved before the source, so it is a cache miss), as is 
`DataFrameWriter.insertInto` (it passes the already-analyzed plan).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. On a catalog that enforces privileges in `loadTable(ident, 
writePrivileges)`, the statements above are now authorized like every other 
write, so a user without the required privilege gets the catalog's error 
instead of silently writing. Catalogs that ignore the privileges argument 
(including Spark's built-in `V2SessionCatalog`) are unaffected. The write 
target of a self-referencing INSERT is now loaded once more, which is what a 
non-self-referencing INSERT already does.
    
    ### How was this patch tested?
    
    New test in `DataSourceV2SQLSuite` next to "SPARK-49246: read-only catalog" 
(whose write cases all use a constant source, which is why this gap was never 
covered), asserting the `ReadOnlyCatalog` privilege error for every statement 
above plus the UPDATE / DELETE / MERGE self-reference cases, for both a custom 
v2 catalog and a `ReadOnlyCatalog`-backed `spark_catalog`. It fails on master 
("Expected exception java.lang.RuntimeException to be thrown, but no exception 
was thrown") and pass [...]
    
    Also green: 17 suites / 1506 tests - `DataSourceV2SQLSuite` (V1+V2 filter), 
`DataSourceV2DataFrameSuite`, `DataSourceV2OptionSuite`, `PlanResolutionSuite`, 
`CachedTableSuite`, `*ViewTestSuite`, group/delta-based MERGE/UPDATE/DELETE 
suites, `InsertSuite`. `dev/lint-scala` clean.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5)
    
    Closes #57563 from peter-toth/SPARK-58370-write-privileges-self-reference.
    
    Authored-by: Peter Toth <[email protected]>
    Signed-off-by: Peter Toth <[email protected]>
    (cherry picked from commit a1c3f310a8c34a0c15661a41cb62cceae6032e1c)
    Signed-off-by: Peter Toth <[email protected]>
---
 .../sql/catalyst/analysis/RelationResolution.scala | 12 +++--
 .../spark/sql/connector/DataSourceV2SQLSuite.scala | 59 ++++++++++++++++++++++
 2 files changed, 68 insertions(+), 3 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
index 22bed3fbe769..5bd1256b631f 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
@@ -238,11 +238,17 @@ class RelationResolution(
       case CatalogAndIdentifier(catalog, ident) =>
         val key = toCacheKey(catalog, ident, finalTimeTravelSpec)
         val planId = u.getTagValue(LogicalPlan.PLAN_ID_TAG)
-        relationCache
-          .get(key)
+        val writePrivileges = 
u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES)
+        // A reference that requires write privileges is never served from the 
per-query relation
+        // cache. The catalog authorizes the write in `loadTable(ident, 
writePrivileges)` below, and
+        // a cache hit would skip that call entirely. The hit happens whenever 
the write target is
+        // also read in the same statement -- the target is resolved after its 
query (see
+        // `ResolveRelations`), so it finds the relation the query already put 
in the cache, e.g.
+        // for `INSERT INTO t SELECT * FROM t`.
+        val cached = if (writePrivileges == null) relationCache.get(key) else 
None
+        cached
           .map(adaptCachedRelation(_, planId))
           .orElse {
-            val writePrivileges = 
u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES)
             val finalOptions = u.clearWritePrivileges.options
             // For a `RelationCatalog` with no time-travel / write privileges, 
the single-RPC
             // `loadRelation` answers both "is there a table?" and "is there a 
view?" in one
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
index 78baca76aa75..79c46bdb4ccf 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
@@ -5209,6 +5209,65 @@ class DataSourceV2SQLSuiteV1Filter
     checkWriteOperations("read_only_cat")
   }
 
+  test("SPARK-58370: read-only catalog, write whose query reads the target 
table") {
+    def assertPrivilegeError(f: => Unit, privileges: String): Unit = {
+      val e = intercept[RuntimeException](f)
+      assert(e.getMessage == s"cannot write with $privileges")
+    }
+
+    def checkWritesReadingTheTarget(catalog: String): Unit = {
+      withSQLConf(s"spark.sql.catalog.$catalog" -> 
classOf[ReadOnlyCatalog].getName) {
+        val tbl = s"$catalog.default.t"
+        withTable(tbl) {
+          sql(s"CREATE TABLE $tbl (i INT)")
+
+          // The write target is resolved after its query, so it used to find 
the query's
+          // relation in the per-query relation cache and skip 
`loadTable(ident, writePrivileges)`.
+          assertPrivilegeError(sql(s"INSERT INTO $tbl SELECT * FROM $tbl"), 
"INSERT")
+          assertPrivilegeError(
+            sql(s"INSERT OVERWRITE $tbl SELECT * FROM $tbl"), "DELETE,INSERT")
+          assertPrivilegeError(
+            sql(s"INSERT OVERWRITE $tbl SELECT * FROM $tbl WHERE 1 = 0"), 
"DELETE,INSERT")
+          assertPrivilegeError(
+            sql(s"INSERT INTO $tbl REPLACE WHERE i = 0 SELECT * FROM $tbl"), 
"DELETE,INSERT")
+          // The reference to the target can be anywhere in the query.
+          assertPrivilegeError(
+            sql(s"INSERT INTO $tbl SELECT i FROM (SELECT i FROM $tbl) x"), 
"INSERT")
+          assertPrivilegeError(
+            sql(s"WITH c AS (SELECT i FROM $tbl) INSERT INTO $tbl SELECT i 
FROM c"), "INSERT")
+
+          // `DataFrameWriterV2` passes the unanalyzed query plan, so its 
target is resolved in the
+          // same analyzer run as the query's reference to the same table.
+          assertPrivilegeError(sql(s"SELECT * FROM 
$tbl").writeTo(tbl).append(), "INSERT")
+          assertPrivilegeError(
+            sql(s"SELECT * FROM $tbl").writeTo(tbl).overwritePartitions(), 
"DELETE,INSERT")
+          assertPrivilegeError(sql(s"SELECT * FROM 
$tbl").write.insertInto(tbl), "INSERT")
+
+          // Row-level operations resolve the target before the source / 
subquery, so these were
+          // already checked; they are here to keep both orders covered.
+          assertPrivilegeError(
+            sql(s"UPDATE $tbl SET i = 0 WHERE i IN (SELECT i FROM $tbl)"), 
"UPDATE")
+          assertPrivilegeError(
+            sql(s"DELETE FROM $tbl WHERE i IN (SELECT i FROM $tbl)"), "DELETE")
+          assertPrivilegeError(
+            sql(
+              s"""
+                 |MERGE INTO $tbl USING (SELECT i FROM $tbl) AS source
+                 |ON source.i = $tbl.i
+                 |WHEN MATCHED THEN UPDATE SET *
+                 |""".stripMargin),
+            "UPDATE")
+        }
+      }
+    }
+
+    // Reset CatalogManager to clear the materialized `spark_catalog` 
instance, so that we can
+    // configure a new implementation.
+    spark.sessionState.catalogManager.reset()
+    checkWritesReadingTheTarget(SESSION_CATALOG_NAME)
+    checkWritesReadingTheTarget("read_only_self_ref_cat")
+  }
+
   test("StagingTableCatalog without atomic support") {
     withSQLConf("spark.sql.catalog.fakeStagedCat" -> 
classOf[FakeStagedTableCatalog].getName) {
       withTable("fakeStagedCat.t") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to