Hi, nice, I have also been looking at implementing this from the Spark DSV2 side, would love to collaborate on this as well.
Yes it seems like we can go further with the expression spec. I found the determinism problem to be deeper than it seemed at first glance, a lot of popular generation expressions in Delta, Spark are in fact dependent on engine configs like timezone, which makes data skipping a bit tricky, we've only recently started thinking about this issue. Thanks, Szehon On Thu, Aug 6, 2026 at 9:07 AM Daniel Weeks <[email protected]> wrote: > What Andrei captured is a great summary of related proposals and ongoing > work. > > While there's no formal proposal, discussions have occurred regarding > using expressions and the default value expressions concept to expose > generated columns. > > We'd love for someone to create a formal proposal to expand upon these > existing proposals/features as it's a pretty natural extension from what we > have in progress already. > > I'd be happy to work with you on this if you would like to pick this up, > -Dan > > On Thu, Aug 6, 2026 at 8:38 AM Andrei Tserakhau via dev < > [email protected]> wrote: > >> Thanks for bringing this to the list, Prashant. >> >> On your question: no, there's no active work on a format-level >> derived/computed column concept in Iceberg, and as far as I can tell >> from the archives none has been proposed before -- so you're not >> duplicating anything. Some adjacent prior art that may be useful. >> >> Delta has had generated columns since OSS Delta 1.0.0 (May 2021), with >> your exact motivation: a column materialized from an expression, used >> as a partition column, lets the engine infer a filter on it from a >> filter on the source and skip files. Two costs showed up in the five >> years since. First, the expression is stored as a Spark SQL string >> (delta.generationExpression), so anything that isn't Spark has to parse >> Spark SQL to know what the column means -- which is roughly where >> presto.derived-columns.spec.json would put Iceberg, one dialect over. >> Second, Delta gates it behind a writer feature, so a writer that >> doesn't understand the invariant is refused rather than allowed to >> silently break it. Your RFC is candid that stale values produce >> incorrect results once rewrite is enabled, and table properties are >> defined as hints, so an unaware writer is within its rights to ignore >> them. Worth deciding early which of those two you want. >> >> On both counts there's an existing pattern in Iceberg you'd probably >> rather join than parallel. The expressions spec merged in June >> (https://github.com/apache/iceberg/pull/16652) -- field-ID references >> that survive renames, dialect-specific behavior pushed into UDF >> references. Two features are already building on it: Huaxin's >> constraint support ( >> https://lists.apache.org/thread/mv0gjm1o9v6wzx6dxkdw6yp5b9119w2d), >> which stores CHECK expressions as Expression objects rather than SQL >> strings, and Dan's default value expressions in V4 >> (https://lists.apache.org/thread/w0xqrm0dpnsgvw0dyvy4r34y0dtzmn7f, PR >> 16777). The constraint proposal also tracks validity per snapshot >> rather than on the constraint, which might be a lighter answer to your >> staleness problem than a format-version bump -- an unaware writer would >> degrade you to "rewrite disabled" rather than "wrong answer." >> >> Two scope notes. Your item 3 (stats and a bloom filter with no stored >> data) overlaps the secondary index work fairly directly -- Péter has a >> draft spec (https://github.com/apache/iceberg/pull/16961) and there's >> an active sync series. And it's worth stating explicitly that this >> covers deterministic expressions only: Delta put deterministic >> expression-derived columns and non-deterministic value generation >> (identity/sequence) under one GENERATED ALWAYS AS umbrella, and >> separating them afterwards was painful, since pushdown rewrite needs >> reproducibility and value generation needs the opposite. Your RFC >> already excludes non-deterministic functions, so this is just making >> that explicit in the spec. >> >> Last thing: hidden-partition transforms already give metadata-level >> skipping for monotonic cases with no extra stored column. lower(col) >> genuinely isn't covered, so the motivation is fair -- but saying where >> transforms already suffice and where a stored column is actually needed >> would make the proposal harder to argue with. >> >> Best, >> Andrei >> >> On Thu, Aug 6, 2026 at 12:28 PM Prashant Sharma <[email protected]> >> wrote: >> >>> Hello All, >>> >>> Recently, I have been involved in bringing derived columns(aka generated >>> columns) for the Iceberg connector in Presto Link: >>> https://github.com/prestodb/rfcs/pull/61. We currently use table >>> properties to store the extra metadata for derived columns. However, we >>> cannot achieve the following: >>> >>> 1. Cross engine compatibility: How do we know if the derived columns >>> metadata is in sync i.e. if other engines unaware of derived columns write >>> to the table/alter the columns etc... >>> 2. UDF portability and versioning: which is already addressed by >>> (Iceberg UDF spec) >>> 3. VIRTUAL columns: We store all the column metadata/metrics/stats >>> including a bloom filter but not the actual data. A poor man's index. >>> >>> We are considering a Iceberg spec level support to address all of the >>> above. Just wondering, is there work already happening in this direction >>> elsewhere? >>> >>> If not, I have started an issue: >>> https://github.com/apache/iceberg/issues/15923 , will be adding more >>> details soon. >>> >>> Thanks, >>> Prashant >>> >>
