Hi all, I have updated the proposal document and added a new tab to the Google doc [1]. The textual changes were too much to keep it "inline" and leverage gDoc versioning, so I opted to create a "v2" tab in the same doc.
Once the community agrees on the proposal "v2", we can go ahead and discuss the next steps. Best, Robert [1] https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM/edit?tab=t.7l861fq8jo38 On Fri, Sep 12, 2025 at 4:38 PM Robert Stupp <sn...@snazy.de> wrote: > > Hi all, > > Regarding the ongoing discussion about the Iceberg REST FGAC OpenAPI > proposal, we want to confirm our alignment with the community's > direction. > > The "Iceberg Expressions + Iceberg UDFs way" as the sole mechanism for > retrieving protection instructions is a good option in my opinion. > This approach aligns with the core principle of separating policy > definitions from protection instructions. > > As the initial proposal highlighted, as good practice / implementation > advice, catalogs should provide user and query-specific protection > instructions. > > We believe this approach effectively addresses the need for > interoperability and the strict separation between how policies are > defined and how they are enforced at query execution time. > > I will update the OpenAPI proposal by next week for broader review. > > Best regards, > Robert > > On Thu, Aug 21, 2025 at 1:22 AM Ryan Blue <rdb...@gmail.com> wrote: > > > > Thanks, Prashant! I agree, it's good to have a starting point we can > > iterate from. > > > > On Wed, Aug 20, 2025 at 3:25 PM Yufei Gu <flyrain...@gmail.com> wrote: > >> > >> Thanks Prashant for filing the PR. This is a solid start point for FGAC! > >> > >> The spec changes are straightforward and leave plenty of room for future > >> extensibility. While the current Transform [1] isn’t quite sufficient for > >> column masking, enhancements to Transform along with the upcoming UDF > >> proposal should address that gap. For instance, we could introduce a > >> "UDFTerm" once the UDF spec is finalized. > >> > >> [1] > >> https://github.com/polaris-catalog/polaris/blob/8d4cacb27c2a29353091b149e733b4fa7ce4ac53/spec/iceberg-rest-catalog-open-api.yaml#L2317-L2317 > >> > >> Yufei > >> > >> > >> On Wed, Aug 20, 2025 at 2:59 PM Prashant Singh <prashant010...@gmail.com> > >> wrote: > >>> > >>> Robert, > >>> > >>> > It might not change this particular proposal > >>> > having support for UDFs in Iceberg Expressions, which is a prerequisite > >>> > for this particular proposal > >>> > >>> Unfortunately, that's not my understanding of what community consensus > >>> was based on community sync (here) > >>> I have put out a SPEC change proposal based on my understanding of what > >>> community consensus is, as PR-13879, I would request other community > >>> members to chime here as well. > >>> > >>> As far as client server interaction with UDFs are concerned, based on my > >>> understanding of discussion so far in UDF syncs, we want to send all the > >>> versions / dialects that udf has from server to the client and the client > >>> picks the one which it wants, not the other way around. It is similar to > >>> what we have for Iceberg View already, this should handle both your > >>> questions. That being said if you have follow-up questions there is a > >>> dedicated Sync for UDF discussions. > >>> > >>> Thank you, > >>> Prashant Singh > >>> > >>> On Wed, Aug 20, 2025 at 2:48 AM Robert Stupp <sn...@snazy.de> wrote: > >>>> > >>>> Prashant, > >>>> > >>>> be ensured that we are still and actively working into this. We are > >>>> looking at this from multiple points of specific views but most > >>>> importantly from a high level view: how all the things work seamlessly > >>>> together. We are really looking forward to providing an update as soon > >>>> as we are confident that the whole approach works. It might not change > >>>> this particular proposal, but it is very important for a proper design > >>>> to think about upstream and downstream impacts. > >>>> > >>>> In the meantime, as it's orthogonal to this particular effort, it > >>>> would be great to get the ball rolling on having support for UDFs in > >>>> Iceberg Expressions, which is a prerequisite for this particular > >>>> proposal. We would be happy, if you could help getting that into > >>>> Apache Iceberg. > >>>> > >>>> As we are speaking about Iceberg UDFs, and implicitly agreed that > >>>> those become a prerequisite for this proposal, it would be very > >>>> beneficial for clients (query engines) being able to leverage the > >>>> following, which is where your help would be appreciated: > >>>> 1. Allows clients to request an Iceberg UDF only in the current > >>>> version, and only the representation (think: SQL dialect) the client > >>>> needs. > >>>> 2. Allow clients to request multiple Iceberg UDF representations in > >>>> one request. Optionally only returning the currentversion and > >>>> optionally restricted to the representation the client needs. > >>>> > >>>> Robert > >>>> > >>>> On Mon, Aug 18, 2025 at 6:05 PM Prashant Singh > >>>> <prashant010...@gmail.com> wrote: > >>>> > > >>>> > Hi all, > >>>> > > >>>> > It was really nice to go through the proposal in the Iceberg catalog > >>>> > community sync (7/30)! > >>>> > > >>>> > As next steps, I was expecting the doc to be updated according to the > >>>> > community feedback, specifically: > >>>> > > >>>> > Using loadTable API to return the evaluated policies result. > >>>> > > >>>> > Aligning on Iceberg expressions as the path forward for row filters, > >>>> > with UDFs to handle dialect-specific logic. > >>>> > > >>>> > I noticed the doc hasn’t yet been updated with this feedback. I > >>>> > completely understand the authors might be busy, so I’d like to offer > >>>> > help to push this forward. > >>>> > > >>>> > We’ve discussed this topic in depth across several syncs, so perhaps > >>>> > we could even move ahead by opening a spec PR to IRC and continue > >>>> > iterating there. I’m happy to help put that out if it would help keep > >>>> > things moving. > >>>> > > >>>> > Downstream projects such as Apache Polaris are eagerly waiting for > >>>> > this feature, so it would be great to make progress here! > >>>> > > >>>> > Thanks, > >>>> > Prashant Singh > >>>> > > >>>> > > >>>> > On Wed, Jul 23, 2025 at 7:45 AM Robert Stupp <sn...@snazy.de> wrote: > >>>> >> > >>>> >> Hi all, > >>>> >> > >>>> >> Following up on the “Iceberg REST FGAC proposal” discussion [1], we > >>>> >> are happy to share the more detailed proposal [2] to extend the Apache > >>>> >> Iceberg REST specification to include a new API for retrieving > >>>> >> fine-grained access control (FGAC) "protection instructions" > >>>> >> (row-level filters and column transformations) from an Iceberg REST > >>>> >> Catalog. > >>>> >> > >>>> >> The aim is to standardize how query engines obtain these instructions > >>>> >> based on user identity, simplifying data protection enforcement. > >>>> >> > >>>> >> The proposal focuses solely on the new Iceberg REST API endpoint to > >>>> >> retrieve protection instructions, intentionally omitting catalog > >>>> >> specific policy management APIs. > >>>> >> > >>>> >> Having a truly interoperable way to represent the protection > >>>> >> instructions for both row filters and column transformations is a huge > >>>> >> benefit. This is why the support for Iceberg expressions is marked as > >>>> >> mandatory in the proposal. We think that it is a fair option to allow > >>>> >> people to use SQL expressions, not required by the proposal, to > >>>> >> satisfy their needs, assuming they are okay to accept that not all > >>>> >> catalogs or engines support SQL expressions or not all SQL > >>>> >> conformance/dialects. > >>>> >> > >>>> >> Thanks to all of those who have helped review & contribute - JB > >>>> >> Onofre, Prashant Singh, Russell Spitzer, Roy Hansson, & Kevin Liu. We > >>>> >> are excited about the community support! > >>>> >> > >>>> >> Cheers, > >>>> >> Robert, Laurent, Alex, Dmitri > >>>> >> > >>>> >> [1] https://lists.apache.org/thread/nfw1t0glfdfj1hwmzzzzwwyrfnq44yr5 > >>>> >> [2] > >>>> >> https://docs.google.com/document/d/108Y0E8XsZi91x-UY0_aHLEbmXDNmxmS5BnDjunEKvTM
