plusplusjiajia opened a new pull request, #10906: URL: https://github.com/apache/paimon/pull/10906
### Purpose On a table with query auth, operations that read rows and write data back did not respect the caller's row filter and column masks: - Spark `UPDATE`, `DELETE` and `MERGE INTO` on append and Data Evolution tables find and read the candidate files through raw readers (`newSnapshotReader` and `KnownSplitsTable`). So `UPDATE t SET note = ssn` copied a masked column into a visible one, and rows hidden by the row filter were updated or deleted, for example `DELETE FROM t` without a condition. - On primary-key tables, Spark and Flink `UPDATE` and `MERGE INTO` read the filtered, masked rows and wrote the masked values back as real data. - Flink sort compaction and rescale, and Flink and Spark chain table compaction, read through the caller's filtered, masked view and overwrote partitions with it. The hidden rows were lost and the masks were stored. These operations now fail when query auth gives the caller a row filter or a column mask. The check is a shared helper next to the search rejection, `TableQueryAuthResult.rejectRewriteUnderQueryAuth`, which authorizes with all columns and refuses when the result has rules. Users with full access are not affected, which matches PyPaimon, where update, delete, merge and index builds require full table access. The check is called from: - Spark: the `UPDATE`, `DELETE` and `MERGE INTO` commands (including Data Evolution tables and the Spark 4.0 merge commands), the V2 row-level operations, the optimized `DELETE` / truncate, and `CompactChainTableProcedure`; - Flink: the row-level `UPDATE` / `DELETE` sink and the `MergeInto`, `DataEvolutionMergeInto`, `SortCompact`, `Rescale` and `CompactChainTable` actions. Rewrites that read the raw files, such as the Spark sort compaction, clustering and rescale procedures, keep the data intact and are unchanged. ### Tests - `TableQueryAuthResultTest#testRewriteWithoutQueryAuth` and `#testRewriteRequiresFullTableAccess`: the helper allows tables without query auth and users with full access, and refuses row filters and masks. - `SparkCatalogWithRestTest#testRowLevelOperationsRequireFullTableAccess`: Spark `UPDATE`, `DELETE` and `MERGE INTO` are refused for a restricted user on append tables (with V1 and V2 writes, and with deletion vectors), primary-key tables and Data Evolution tables, and still work with full access. - `RESTCatalogITCase#testRowLevelOperationsRequireFullTableAccess` and `#testMaintenanceRewritesRequireFullTableAccess`: Flink row-level operations, sort compaction and rescale are refused for a restricted user and still work with full access. - The new Spark cases fail on master. The related tests pass locally: 514 tests, including the existing Spark and Flink query auth and row-level suites. Spark 3, Spark 4 and Flink 1 compile without the fast-build profile. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
