SteNicholas opened a new issue, #520:
URL: https://github.com/apache/paimon-cpp/issues/520

   ## Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
   
   ## Motivation
   
   Java Paimon lets a REST catalog enforce row-level filters and column masks 
on reads through the table option `query-auth.enabled` 
(`CoreOptions.QUERY_AUTH_ENABLED`, default `false`):
   
   - When a table with `query-auth.enabled=true` plans a scan, 
`AbstractDataTableScan.authQuery()` asks the catalog for the rules through 
`RESTCatalog.authTableQuery()`, which sends `POST 
/v1/{prefix}/databases/{database}/tables/{table}/auth` with `{"select": 
[...]}`. The selected columns also include the columns referenced by the user 
filter and the partition filter, so their values are authorized before they are 
used for pruning.
   - The response is `{"filter": [...], "columnMasking": {...}}`. `filter` is a 
list of JSON-serialized `Predicate`s, combined with AND. `columnMasking` maps a 
column name to a JSON-serialized `Transform` (`TableQueryAuthResult`).
   - The splits are wrapped in `QueryAuthSplit` with the auth result, and 
`AbstractDataTableRead.createReader()` applies the row filter and the column 
masks when it reads them. With query auth enabled, `withFilter()` does not push 
the user predicate into the reader (it returns before `innerWithFilter()`).
   - The rules are re-validated against the latest schema on every plan, so a 
schema change under a live scan fails closed (`validateAgainstSchema`, 
`validateReadableWithoutRename`).
   - The option cannot be disabled through dynamic options once the table 
enables it (`AbstractFileStoreTable.checkImmutability`). Search that a column 
mask would invalidate is rejected 
(`TableQueryAuthResult.rejectSearchUnderQueryAuth`).
   - `CatalogEnvironment.tableQueryAuth()` skips the call when the table has no 
catalog loader, for example an external table of a REST catalog.
   
   paimon-cpp implements none of this:
   
   - `query-auth.enabled` is not referenced anywhere under `include/` or `src/`;
   - `RestApi` and `ResourcePaths` have no `auth` endpoint, and `Catalog` has 
no `authTableQuery()`;
   - there is no deserialization of the Java `Predicate` or `Transform` JSON.
   
   As a result, a paimon-cpp reader that opens a `query-auth.enabled=true` 
table through `RestCatalog` reads every row and every column unmasked. The row 
filters and column masks the catalog defines for the table are not applied, 
while a Java reader of the same table gets the filtered, masked result.
   
   ## Solution
   
   Fail closed first, then add the feature.
   
   1. Fail closed. Until the rules can be applied, reject scans and reads of a 
table with `query-auth.enabled=true` with a clear `NotImplemented` status, so 
the table can never be read without its rules. Also reject disabling the option 
through dynamic options, as Java does.
   
   2. REST protocol:
      - add `ResourcePaths::AuthTable(database, table)` -> 
`/v1/{prefix}/databases/{database}/tables/{table}/auth`;
      - add `AuthTableQueryRequest` (`select`, optional) and 
`AuthTableQueryResponse` (`filter`: list of strings, `columnMasking`: string 
map, both optional) to `rest_messages.h/.cpp`;
      - add `RestApi::AuthTableQuery(identifier, select)`. It is a read and is 
safe to retry;
      - map the errors the same way Java does: 404 is table not found, 403 is 
no permission, 501 is not supported.
   
   3. Catalog: expose the call to the scan, for example as an internal 
interface next to `VersionManagedCatalog` that `RestCatalog` implements. Reject 
system tables, and skip it for tables without catalog-managed versions 
(external tables), following `CatalogEnvironment.tableQueryAuth()`.
   
   4. Rule parsing: deserialize the Java `Predicate` JSON (leaf and compound 
predicates, literals of every supported type) and the `Transform` JSON used by 
column masks. Reject an empty column name, an empty transform, and JSON `null`, 
as `TableQueryAuthResult` does. An unknown predicate or transform must fail the 
query instead of being ignored.
   
   5. Scan and read:
      - at plan time, request the rules with the projected columns plus the 
columns referenced by the user filter and the partition filter;
      - validate the rules against the latest schema;
      - carry the result with the splits to the reader;
      - in the reader, apply the row filter and the masks, and do not push the 
user predicate or column statistics of masked columns down;
      - reject search that a mask would invalidate.
   
   Tests:
   
   - `rest_messages_test.cpp`: request/response JSON round trips with absent 
fields;
   - `rest_catalog_test.cpp`: the endpoint and error mapping, using 
`MockRestServer`;
   - a read test: rows filtered, columns masked, the user filter not pushed 
down, fail closed when the rules cannot be parsed or reference unknown columns, 
and the option not being disabled through dynamic options.
   
   Steps 1 and 2 can land on their own before the rest.
   
   ## Anything else?
   
   Compared against Java Paimon at apache/paimon@c2e001b7f2 and paimon-cpp at 
41ed4ead.
   
   Java references:
   
   - `paimon-api/.../rest/RESTApi.java` (`authTableQuery`);
   - `paimon-api/.../rest/ResourcePaths.java` (`authTable`);
   - `paimon-api/.../rest/requests/AuthTableQueryRequest.java`;
   - `paimon-api/.../rest/responses/AuthTableQueryResponse.java`;
   - `paimon-core/.../rest/RESTCatalog.java` (`authTableQuery`);
   - `paimon-core/.../catalog/TableQueryAuthResult.java`;
   - `paimon-core/.../table/CatalogEnvironment.java` (`tableQueryAuth`);
   - `paimon-core/.../table/source/AbstractDataTableScan.java` (`authQuery`);
   - `paimon-core/.../table/source/AbstractDataTableRead.java`;
   - `paimon-core/.../table/source/QueryAuthSplit.java`.
   
   ## Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to