JingsongLi opened a new pull request, #10539: URL: https://github.com/apache/paimon/pull/10539
## Dependency Depends on https://github.com/apache/paimon-rust/pull/1122. Keep this PR in draft until that Rust change is merged. Python Native CI continues to build paimon-rust main; the new Native authorization tests need the corresponding Rust core implementation. ## What changes Use Native planning and reads for existing REST query-authorized batch reads. Support the Java QueryAuthSplit type-5 wire format so the plan carries the captured policy alongside its data/indexed split metadata. Only original Native splits retain a live Rust authorization grant; a serialized plan or Native reader setup failure falls back through Python with the captured rules. Align Python execution with Java ReadTransform: authorization filter on original values, column masks, the user predicate, LIMIT, then output projection. Remove masked columns from raw partition/stats/PK/bucket pruning, include hidden query fields and their mask dependencies, and include hidden query/partition operands in the REST authorization request. Scan-only partition selectors referencing masked fields are rejected as Java requires. Fix PAD/OVERLAY integer validation and eager overflow checks despite NULL string inputs, and validate stored field families for the new string transforms. Keep row-kind metadata outside positional predicate operands. No new public Python or Rust binding API is introduced. The new mask execution stays in Rust core. ## Verification - Native authorization suite: 64 REST tests across Append, PK and Data Evolution, plus 3 Native explain tests. - Combined Native authorization / existing REST / resolved-schema suites: 184 passed before the 3 explain tests were added; 211 Native plans and 187 Native reads exercised. - Related Python unit suites: 423 passed, plus 69 subtests. Final parser/split golden subset: 119 passed and 25 subtests. - Actual Java-generated QueryAuthSplit golden round-trips exactly, preserving UTF-8/NUL policy text, UTF-16 map ordering, indexed row ranges and scores. - Regressions cover Native setup failure, disabled Native reads, wire/pickle round trips, policy changes after planning, hidden masked predicates before LIMIT, masked partitions, authorization request scope, and PK merge with row-kind metadata. - Changed-file flake8 and Python 3.6 syntax checks passed. ## Scope This enables REST batch authorization through Native for the supported Rust rules. Transformed authorization row filters and streaming authorization remain separate follow-up work. Explicit authorized partition selectors remain on Python until the binding can represent their scan-only semantics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
