I am fairly new to Iceberg. Thanks for the encouragement. Here is a first draft (still WIP) https://docs.google.com/document/d/1u93kyIrTc8bt9VSTB4O0GM9S7zuXekqixJR6Id2FZC4/edit?usp=sharing. Please take a look !
https://github.com/apache/iceberg/issues/15923 @Andrei Tserakhau <[email protected]> Your summary has helped a lot in catching up with parallel work and also the history - Thanks a lot ! @[email protected] <[email protected]> and @[email protected] <[email protected]> Happy to collaborate - was trying to add you as editor - let me know your google account enabled id. In the proposal document above, I have also tried to address two problem which are common to multiple v4 features 1. Expressions reuse 2. Backward compatibility: V4-features need the ability to know data was modified by an engine unaware of the feature and how does the engine tell that? For example: derived columns feature on optimizer rewrite could give incorrect results. On Fri, Aug 7, 2026 at 9:09 PM Stefan Grafberger via dev < [email protected]> wrote: > Hi all, > > Thanks a lot for bringing up this topic! > > I think one important topic to consider here is multi-modal workloads. > Both Lance and other databases like BigQuery use derived columns also > for embedding generation, where materializing the output of > expressions like embedding functions provides huge cost and compute > savings. > > - Lance Embedding Generation: https://docs.lancedb.com/embedding > - BigQuery Autonomous Embedding Generation: > https://docs.cloud.google.com/bigquery/docs/autonomous-embedding-generation > > To support these multi-modal AI patterns in Iceberg, it would be > beneficial not to broadly prohibit all non-deterministic expressions > in the spec. > > Instead, it might be worth distinguishing between: > 1. Purely deterministic expressions (e.g., LOWER(col)), > 2. Purely non-deterministic expressions (e.g., RAND(), UUID()), and > 3. Non-deterministic but reusable expressions (e.g., AI.EMBED() in > BigQuery), where the value generation is AI based, so there's some > non-determinism involved, but once generated, the materialized result > acts as a reusable derived property. > > By making this distinction at the spec level, we could allow reusable > non-deterministic expressions without sacrificing the predictability > needed for query optimization. > > Curious to hear what others think about supporting such AI use cases. > > Best, > Stefan > > On Thu, Aug 6, 2026 at 8:09 PM Szehon Ho <[email protected]> wrote: > > > > Hi, nice, I have also been looking at implementing this from the Spark > DSV2 side, would love to collaborate on this as well. > > > > Yes it seems like we can go further with the expression spec. I found > the determinism problem to be deeper than it seemed at first glance, a lot > of popular generation expressions in Delta, Spark are in fact dependent on > engine configs like timezone, which makes data skipping a bit tricky, we've > only recently started thinking about this issue. > > > > Thanks, > > Szehon > > > > On Thu, Aug 6, 2026 at 9:07 AM Daniel Weeks <[email protected]> wrote: > >> > >> What Andrei captured is a great summary of related proposals and > ongoing work. > >> > >> While there's no formal proposal, discussions have occurred regarding > using expressions and the default value expressions concept to expose > generated columns. > >> > >> We'd love for someone to create a formal proposal to expand upon these > existing proposals/features as it's a pretty natural extension from what we > have in progress already. > >> > >> I'd be happy to work with you on this if you would like to pick this up, > >> -Dan > >> > >> On Thu, Aug 6, 2026 at 8:38 AM Andrei Tserakhau via dev < > [email protected]> wrote: > >>> > >>> Thanks for bringing this to the list, Prashant. > >>> > >>> On your question: no, there's no active work on a format-level > >>> derived/computed column concept in Iceberg, and as far as I can tell > >>> from the archives none has been proposed before -- so you're not > >>> duplicating anything. Some adjacent prior art that may be useful. > >>> > >>> Delta has had generated columns since OSS Delta 1.0.0 (May 2021), with > >>> your exact motivation: a column materialized from an expression, used > >>> as a partition column, lets the engine infer a filter on it from a > >>> filter on the source and skip files. Two costs showed up in the five > >>> years since. First, the expression is stored as a Spark SQL string > >>> (delta.generationExpression), so anything that isn't Spark has to parse > >>> Spark SQL to know what the column means -- which is roughly where > >>> presto.derived-columns.spec.json would put Iceberg, one dialect over. > >>> Second, Delta gates it behind a writer feature, so a writer that > >>> doesn't understand the invariant is refused rather than allowed to > >>> silently break it. Your RFC is candid that stale values produce > >>> incorrect results once rewrite is enabled, and table properties are > >>> defined as hints, so an unaware writer is within its rights to ignore > >>> them. Worth deciding early which of those two you want. > >>> > >>> On both counts there's an existing pattern in Iceberg you'd probably > >>> rather join than parallel. The expressions spec merged in June > >>> (https://github.com/apache/iceberg/pull/16652) -- field-ID references > >>> that survive renames, dialect-specific behavior pushed into UDF > >>> references. Two features are already building on it: Huaxin's > >>> constraint support ( > https://lists.apache.org/thread/mv0gjm1o9v6wzx6dxkdw6yp5b9119w2d), > >>> which stores CHECK expressions as Expression objects rather than SQL > >>> strings, and Dan's default value expressions in V4 > >>> (https://lists.apache.org/thread/w0xqrm0dpnsgvw0dyvy4r34y0dtzmn7f, PR > >>> 16777). The constraint proposal also tracks validity per snapshot > >>> rather than on the constraint, which might be a lighter answer to your > >>> staleness problem than a format-version bump -- an unaware writer would > >>> degrade you to "rewrite disabled" rather than "wrong answer." > >>> > >>> Two scope notes. Your item 3 (stats and a bloom filter with no stored > >>> data) overlaps the secondary index work fairly directly -- Péter has a > >>> draft spec (https://github.com/apache/iceberg/pull/16961) and there's > >>> an active sync series. And it's worth stating explicitly that this > >>> covers deterministic expressions only: Delta put deterministic > >>> expression-derived columns and non-deterministic value generation > >>> (identity/sequence) under one GENERATED ALWAYS AS umbrella, and > >>> separating them afterwards was painful, since pushdown rewrite needs > >>> reproducibility and value generation needs the opposite. Your RFC > >>> already excludes non-deterministic functions, so this is just making > >>> that explicit in the spec. > >>> > >>> Last thing: hidden-partition transforms already give metadata-level > >>> skipping for monotonic cases with no extra stored column. lower(col) > >>> genuinely isn't covered, so the motivation is fair -- but saying where > >>> transforms already suffice and where a stored column is actually needed > >>> would make the proposal harder to argue with. > >>> > >>> Best, > >>> Andrei > >>> > >>> On Thu, Aug 6, 2026 at 12:28 PM Prashant Sharma <[email protected]> > wrote: > >>>> > >>>> Hello All, > >>>> > >>>> Recently, I have been involved in bringing derived columns(aka > generated columns) for the Iceberg connector in Presto Link: > https://github.com/prestodb/rfcs/pull/61. We currently use table > properties to store the extra metadata for derived columns. However, we > cannot achieve the following: > >>>> > >>>> 1. Cross engine compatibility: How do we know if the derived columns > metadata is in sync i.e. if other engines unaware of derived columns write > to the table/alter the columns etc... > >>>> 2. UDF portability and versioning: which is already addressed by > (Iceberg UDF spec) > >>>> 3. VIRTUAL columns: We store all the column metadata/metrics/stats > including a bloom filter but not the actual data. A poor man's index. > >>>> > >>>> We are considering a Iceberg spec level support to address all of the > above. Just wondering, is there work already happening in this direction > elsewhere? > >>>> > >>>> If not, I have started an issue: > https://github.com/apache/iceberg/issues/15923 , will be adding more > details soon. > >>>> > >>>> Thanks, > >>>> Prashant >
