plusplusjiajia opened a new pull request, #8582:
URL: https://github.com/apache/paimon/pull/8582
Stacked on #8570 — review the top commit only; will be rebased once #8570
merges.
Purpose
A column mask changes the value domain of its target column. A predicate
on a masked column therefore evaluates on the masked value (view semantics, as
in Snowflake/Ranger): matching the raw value would reveal
it. This makes every optimization that consumes the column's raw values or
raw statistics unsafe on masked columns — file/split pruning by min-max or
partition value, TopN selection, limit pushdown, and
reader-level row filtering would drop or leak rows relative to what the
query is entitled to see. This is a pre-existing issue of the merged masking
feature (#8447/#8458), independent of #8570.
The design follows the pattern ReadBuilderImpl already uses for read-level
TopN on query-auth.enabled tables: never hand unsafe state to the reader,
rather than pushing it and retracting it later.
- Scan side: the query filter is deferred to plan()/listPartitionEntries
and pushed once through the two-argument SnapshotReader.withFilter — the full
filter marks read-time filtering, while only the conjuncts
free of masked columns feed statistics and partition pruning. Masked
conjuncts (even partition-only ones) keep limit/TopN split pruning off; TopN
split pruning also skips masked ordering columns. A mask
appearing on an already-pushed filter column fails closed, since raw
statistics were already consumed.
- Read side: on query-auth.enabled tables the filter is stored (for
executeFilter) but not forwarded to reader internals; engines re-evaluate data
filters on the masked output. The conjuncts on masked columns
are evaluated inside the auth read, post-mask — engines do not re-evaluate
the partition filters they consumed. A masked filter column the query does not
project is widened into the read schema (same machinery
as mask inputs in #8570) and projected back out.
Scoped to query-auth.enabled tables: they conservatively lose reader-level
filter pushdown even without mask rules, matching the existing read-level TopN
behavior; scan-level pruning on safe conjuncts is
retained. Tables without query auth are unaffected.
Tests
- MockRESTCatalogTest: filter on a masked partition column (projected and
unprojected; engines consume partition filters, so the read must evaluate
them), raw-value match returns nothing, limit with a masked
partition-only filter, masked pk filter not applied on raw key ranges,
mask growth on a pushed filter column fails the scan closed while a fresh scan
+ existing reader return masked-correct results,
TopN/limit/stats-pruning disablement.
- Flink RESTCatalogITCase: partition-column mask + WHERE on it through SQL
(the consumed-filter path).
- Spark SparkCatalogWithRestTest: statistics-based aggregate pushdown
degrades on authed reads (regression anchor).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
