Looks good to me 👍
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Tue, Nov 17, 2026 at 4:00 PM huaxin gao <[email protected]> wrote: > Hi all, > > Friendly ping on this SPIP. The updated doc addresses the earlier > feedback, and the "Changes from the original proposal" section at the top > maps each review point to where it is answered: > > https://docs.google.com/document/d/1eS1NpjEa-K3UlDSuNENA6wBXNkwCrNfZiXMV024PFtg/edit?tab=t.0 > > Please take a look and share any comments here or in the doc. If there are > no major concerns, I plan to start a VOTE thread next week. > > Thanks, > Huaxin > > On Thu, Nov 12, 2026 at 12:44 PM Asif Shahid <[email protected]> > wrote: > >> Hi Huaxin, >> I had read your comments and agree . Sorry, I was not able to spare time >> to contribute further.. Best of luck with your efforts..Hope to see it in >> codebase. >> Regards >> Asif >> >> >> On Wed, Nov 11, 2026 at 3:19 PM huaxin gao <[email protected]> >> wrote: >> >>> Hi all, >>> >>> Thanks everyone for the reviews on the first version. I have updated the >>> SPIP based on the feedback: >>> >>> V2 doc: >>> https://docs.google.com/document/d/1eS1NpjEa-K3UlDSuNENA6wBXNkwCrNfZiXMV024PFtg/edit?tab=t.0 >>> >>> The main changes: >>> >>> - Each security guarantee is now stated with the threat it stops, >>> how Spark enforces it, and what happens when Spark cannot keep it (G1 to >>> G5), with matching tests. >>> - The policy row filter is a security barrier: only leakproof user >>> predicates can move below it. >>> - The policy is applied by a dedicated operator, PolicyEnforcement, >>> so optimizer rules that do not know about it cannot remove or weaken it. >>> - DELETE, UPDATE, MERGE, and overwrites fail for restricted callers; >>> appends are allowed. >>> - Leaks outside the query result (metadata columns, statistics, >>> metrics, plan output) and bypass paths (stored plans, path-based loads, >>> V1 >>> fallback) are covered. >>> - Older Spark versions fail closed through a new capability signal, >>> enforcesReadPolicies(). >>> - The fixed list of masking functions is removed; a mask can be any >>> Spark built-in or catalog ScalarFunction. >>> - Delivery is split into two phases: row filters first, then column >>> masks. >>> >>> >>> The "Changes from the original proposal" section at the top maps each >>> review comment to the section that answers it. @Martin Grund @Dongjoon Hyun >>> @Yang Jie @Peter Toth @Yuming Wang @Asif Shahid @Szehon Ho @Kaifei Yi >>> @Nicholas Chammas @Prashant Singh @Rory Qi, could you check whether your >>> points are addressed? >>> >>> Thanks, >>> Huaxin >>> >>> On Fri, Sep 25, 2026 at 8:27 PM huaxin gao <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> Thank you for the detailed feedback, both here and in the doc. There >>>> has been a lot of it and it is all useful. >>>> >>>> I am working through everything now and will address the comments in >>>> the doc and the comments on this thread, then post a revised doc. >>>> >>>> Thanks, >>>> Huaxin >>>> >>>> On Thu, Sep 24, 2026 at 9:45 PM Yuming Wang <[email protected]> wrote: >>>> >>>>> Huaxin, thank you for proposing this SPIP. I think this feature is >>>>> necessary, and Spark is the right place to enforce it. I left a comment in >>>>> the doc. >>>>> >>>>> On Thu, Sep 24, 2026 at 5:48 PM Peter Toth <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks for the proposal Huaxin. I like the direction. >>>>>> I agree with the points earlier reviewers raised and I left a few >>>>>> comments in the doc. >>>>>> >>>>>> Best, >>>>>> Peter >>>>>> >>>>>> On Wed, Sep 23, 2026 at 11:09 PM Martin Grund via dev < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Thanks for taking the time to write this proposal. It's an exciting >>>>>>> direction. >>>>>>> >>>>>>> The most important gap in the current proposal is to outline the >>>>>>> security boundaries of the execution. The current proposal focuses too >>>>>>> much >>>>>>> on a rather vague definition of how the filters or masks are not elided, >>>>>>> but that is not enough from a security perspective. >>>>>>> >>>>>>> As part of the SPIP, we should outline both what the guarantees are >>>>>>> and how we plan on enforcing them. I have seen a lot of weirdness in the >>>>>>> past when it comes to row filters and column masks. >>>>>>> >>>>>>> Just to give some examples: >>>>>>> >>>>>>> - How do we handle leaking expressions and what is the way to >>>>>>> prevent information disclosure? (see >>>>>>> https://rhaas.blogspot.com/2012/03/security-barrier-views.html) >>>>>>> - How do we handle type mismatches between masks and colum types? >>>>>>> - How do we handle time travel, clones, etc? >>>>>>> >>>>>>> Generally, I would recommend decoupling built-in masking functions >>>>>>> from the overall proposal. Masking functions are just standard Spark >>>>>>> built-ins, so once the SPIP is on its way, we can always add them to >>>>>>> Spark >>>>>>> without defining them in this proposal. I don't think having a fixed >>>>>>> list >>>>>>> of masking functions is useful. From my experience observing user >>>>>>> behavior, >>>>>>> there is a lot of complexity and variation in these functions. Giving >>>>>>> users >>>>>>> the ability to reference any UDF or built-in function is a much more >>>>>>> convenient approach. >>>>>>> >>>>>>> Looking forward to seeing the updated proposal. >>>>>>> >>>>>>> Martin >>>>>>> >>>>>>> On Wed, Sep 23, 2026 at 7:28 PM Holden Karau <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> So I think the security barrier / bypass path avoidance can be >>>>>>>> handled in a few different ways (I've got one proposal I'm working on a >>>>>>>> draft off but it's maybe a little early and shouldn't block this work). >>>>>>>> >>>>>>>> On 2026/09/23 16:48:33 Dongjoon Hyun wrote: >>>>>>>> > Hi, Huaxin. >>>>>>>> > >>>>>>>> > Thank you for proposing this SPIP. I support the direction. >>>>>>>> Moving row filtering and column masking from private Catalyst hacks to >>>>>>>> a >>>>>>>> public DSv2 contract would be a clear improvement for the ecosystem, >>>>>>>> and >>>>>>>> the proposed semantics are clear. >>>>>>>> > >>>>>>>> > Since the value of this SPIP rests on its "fail-closed" and >>>>>>>> "optimizer-safe" guarantees, I'd like the SPIP to address the >>>>>>>> following, >>>>>>>> grouped by priority. >>>>>>>> > >>>>>>>> > ### 1. Security guarantees (should be addressed before a vote) >>>>>>>> > >>>>>>>> > - **Security barrier**: The SPIP protects masks from the >>>>>>>> optimizer, but not hidden rows from user expressions. Leaky user >>>>>>>> predicates >>>>>>>> (e.g. ANSI errors, Python UDFs, predicates pushed to the connector) >>>>>>>> can be >>>>>>>> evaluated before the policy filter and expose rows that should be >>>>>>>> hidden. >>>>>>>> The SPIP should define the policy filter as a barrier, similar to >>>>>>>> PostgreSQL's `security_barrier` and `LEAKPROOF`. >>>>>>>> > - **DML read path**: DELETE/UPDATE/MERGE also read the target >>>>>>>> table. Applying the policy on that read can silently delete hidden >>>>>>>> rows or >>>>>>>> write masked values back. These operations should fail closed. >>>>>>>> > - **Older Spark versions**: An engine that does not know the new >>>>>>>> interface will silently ignore it, so the table is fail-open there. We >>>>>>>> need >>>>>>>> a handshake between the engine and the connector. >>>>>>>> > - **Bypass paths**: The policy is resolved once at analysis time >>>>>>>> and then stays in the plan. The SPIP should define enforcement for >>>>>>>> paths >>>>>>>> that reuse an analyzed plan or bypass the analyzer rewrite: global temp >>>>>>>> views, streaming, catalog-side `Table` caching, `TableProvider` loads, >>>>>>>> V1 >>>>>>>> fallback, aggregate pushdown, and statistics. >>>>>>>> > >>>>>>>> > ### 2. Design >>>>>>>> > >>>>>>>> > - Restate "non-removable Project" as an invariant rather than a >>>>>>>> special plan node. The invariant: a mask expression is never replaced >>>>>>>> by >>>>>>>> the original attribute, while unused mask outputs can still be pruned. >>>>>>>> > - Clarify the trust model when the policy filter is fully pushed >>>>>>>> down to the connector. >>>>>>>> > - Resolve mask functions safely. Temporary and session functions >>>>>>>> must not be able to shadow a mask. Reuse existing built-ins where >>>>>>>> possible, >>>>>>>> avoid the `hash` name collision, and drop weak hashes such as MD5. >>>>>>>> > - Clean up the API: consistent null vs. empty semantics, type >>>>>>>> rules for each mask function, a more specific name than >>>>>>>> `AccessControl`, >>>>>>>> and consider V2 connector expressions instead of String function names. >>>>>>>> > >>>>>>>> > ### 3. Clarifications >>>>>>>> > >>>>>>>> > - Semantics for views (invoker vs. definer), time travel, nested >>>>>>>> and metadata columns, and schema visibility of non-readable columns. >>>>>>>> > - Add test cases for group 1 to Q8, and revisit the Q7 timeline >>>>>>>> accordingly. >>>>>>>> > >>>>>>>> > I'm looking forward to the next revision. Thank you again for >>>>>>>> driving this, Huaxin. >>>>>>>> > >>>>>>>> > Dongjoon. >>>>>>>> > >>>>>>>> > On 2026/09/23 08:48:42 Yang Jie wrote: >>>>>>>> > > Thanks for putting this together. I'm supportive of the >>>>>>>> direction. >>>>>>>> > > >>>>>>>> > > This closes a real gap and I think putting enforcement in the >>>>>>>> engine, fail-closed and optimizer-aware, is the right call, and >>>>>>>> keeping it >>>>>>>> a vendor-neutral DSv2 capability lets Iceberg, Ranger, Polaris, and >>>>>>>> Delta >>>>>>>> retire their private-API workarounds instead of each carrying its own. >>>>>>>> > > >>>>>>>> > > I have a few questions on the enforcement semantics and the >>>>>>>> standard mask set that I will raise in the SPIP document. None of them >>>>>>>> change my view on the direction. >>>>>>>> > > >>>>>>>> > > Thanks, >>>>>>>> > > Jie Yang >>>>>>>> > > >>>>>>>> > > On 2026/09/23 03:39:25 huaxin gao wrote: >>>>>>>> > > > Hi all, >>>>>>>> > > > >>>>>>>> > > > I would like to start a discussion on a SPIP that adds a >>>>>>>> public DataSource >>>>>>>> > > > V2 >>>>>>>> > > > API for a table or catalog to declare a read access policy >>>>>>>> (readable >>>>>>>> > > > columns, >>>>>>>> > > > a row filter, and column masks) that Spark enforces in the >>>>>>>> query plan. Here >>>>>>>> > > > are >>>>>>>> > > > the jira <https://issues.apache.org/jira/browse/SPARK-59726> >>>>>>>> and SPIP doc >>>>>>>> > > > < >>>>>>>> https://docs.google.com/document/d/1hYEHORjHUnFzSoBY2DeOPILVJyu6iWXBusYwpF9FQJM/edit?tab=t.0 >>>>>>>> > >>>>>>>> > > > . >>>>>>>> > > > >>>>>>>> > > > Today Spark has no supported API for this. Integrators inject >>>>>>>> Catalyst rules >>>>>>>> > > > through SparkSessionExtensions and build on internal Catalyst >>>>>>>> APIs, as >>>>>>>> > > > Apache >>>>>>>> > > > Ranger (via the Kyuubi Spark AuthZ plugin) and a recent >>>>>>>> Apache Iceberg POC >>>>>>>> > > > both do. That is brittle across versions and unsafe: a >>>>>>>> masking projection >>>>>>>> > > > added naively to a plan can be removed or collapsed by the >>>>>>>> optimizer, >>>>>>>> > > > silently >>>>>>>> > > > returning unmasked data. >>>>>>>> > > > >>>>>>>> > > > The proposal adds SupportsAccessControl, enforced during >>>>>>>> analysis, with >>>>>>>> > > > three >>>>>>>> > > > guarantees: the masks cannot be optimized away, the row >>>>>>>> filter always sees >>>>>>>> > > > original pre-mask values, and anything Spark cannot resolve >>>>>>>> fails the read >>>>>>>> > > > rather than returning unprotected data. >>>>>>>> > > > >>>>>>>> > > > Details and proposed interfaces are in the doc. Feedback is >>>>>>>> very welcome. >>>>>>>> > > > >>>>>>>> > > > Thanks, >>>>>>>> > > > Huaxin >>>>>>>> > > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> --------------------------------------------------------------------- >>>>>>>> > > To unsubscribe e-mail: [email protected] >>>>>>>> > > >>>>>>>> > > >>>>>>>> > >>>>>>>> > >>>>>>>> --------------------------------------------------------------------- >>>>>>>> > To unsubscribe e-mail: [email protected] >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe e-mail: [email protected] >>>>>>>> >>>>>>>>
