I’m also not sure to standardize AI functions in the Iceberg expression spec. The spec deliberately defines a small, engine-neutral expression model, with common predicates and a limited set of portable functions closely tied to Iceberg’s needs (ie, data organization and skipping). It leaves richer computation to SQL-standard functions, engine built-ins, and dialect-specific UDFs. This keeps the shared contract small and makes adoption easier.
Standardizing AI functions would significantly expand that scope and require agreement on semantics such as model identity, versioning, inference configuration, and determinism. I’d prefer to use the existing function/UDF mechanisms unless there is a clear Iceberg-specific need for a standardized contract. On Mon, Nov 9, 2026 at 12:01 PM Alex Stephen via dev <[email protected]> wrote: > My proposal definitely has some novelty that I could have explained > better. > > Part of my hope here is to understand if there's any interest from the > Iceberg community in shipping a set of SQL functions as part of the Iceberg > codebase. We have some precedent with the partition transform functions, > but this is definitely more complex than those. > > If there's interest in that, there's a number of shapes that these > functions could take: > - Built into the Iceberg Spark / Flink connectors > - Creating these as "default" UDFs that engines could then implement as > part of the UDF spec > > I definitely understand the novelty here and that this may not be a > perfect fit for the project. Just curious to see what everyone thinks > > On Mon, Nov 9, 2026 at 7:36 AM Russell Spitzer <[email protected]> > wrote: > >> I'm not sure I understand the goal here. Is the goal to create a common >> vocabulary between Spark and Flink? I'm a little hesitant just because i'm >> not sure what the use case is here because you would most likely have a >> number of engine specific things in each API outside of function names and >> argos. >> >> On Mon, Nov 2, 2026 at 2:45 PM Alex Stephen via dev < >> [email protected]> wrote: >> >>> Hello all, >>> >>> As part of our ongoing work with the FILE + VECTOR types, I'd like to >>> propose adding a generative AI function + AI embedding function to the >>> Iceberg Spark / Flink connectors. This would allow users to use a >>> vendor-agnostic set of functions for interacting with their multimodal >>> Iceberg data and creating AI embeddings. There's a wide variety of prior >>> work across both the OSS and proprietary landscapes. >>> >>> Please take a look at the proposal document below. Thanks a lot! >>> >>> Proposal document: https://s.apache.org/iceberg-ai-functions >>> >>> -- Alex Stephen >>> >>
