Thanks for bringing this to the list, Prashant. On your question: no, there's no active work on a format-level derived/computed column concept in Iceberg, and as far as I can tell from the archives none has been proposed before -- so you're not duplicating anything. Some adjacent prior art that may be useful.
Delta has had generated columns since OSS Delta 1.0.0 (May 2021), with your exact motivation: a column materialized from an expression, used as a partition column, lets the engine infer a filter on it from a filter on the source and skip files. Two costs showed up in the five years since. First, the expression is stored as a Spark SQL string (delta.generationExpression), so anything that isn't Spark has to parse Spark SQL to know what the column means -- which is roughly where presto.derived-columns.spec.json would put Iceberg, one dialect over. Second, Delta gates it behind a writer feature, so a writer that doesn't understand the invariant is refused rather than allowed to silently break it. Your RFC is candid that stale values produce incorrect results once rewrite is enabled, and table properties are defined as hints, so an unaware writer is within its rights to ignore them. Worth deciding early which of those two you want. On both counts there's an existing pattern in Iceberg you'd probably rather join than parallel. The expressions spec merged in June (https://github.com/apache/iceberg/pull/16652) -- field-ID references that survive renames, dialect-specific behavior pushed into UDF references. Two features are already building on it: Huaxin's constraint support ( https://lists.apache.org/thread/mv0gjm1o9v6wzx6dxkdw6yp5b9119w2d), which stores CHECK expressions as Expression objects rather than SQL strings, and Dan's default value expressions in V4 (https://lists.apache.org/thread/w0xqrm0dpnsgvw0dyvy4r34y0dtzmn7f, PR 16777). The constraint proposal also tracks validity per snapshot rather than on the constraint, which might be a lighter answer to your staleness problem than a format-version bump -- an unaware writer would degrade you to "rewrite disabled" rather than "wrong answer." Two scope notes. Your item 3 (stats and a bloom filter with no stored data) overlaps the secondary index work fairly directly -- Péter has a draft spec (https://github.com/apache/iceberg/pull/16961) and there's an active sync series. And it's worth stating explicitly that this covers deterministic expressions only: Delta put deterministic expression-derived columns and non-deterministic value generation (identity/sequence) under one GENERATED ALWAYS AS umbrella, and separating them afterwards was painful, since pushdown rewrite needs reproducibility and value generation needs the opposite. Your RFC already excludes non-deterministic functions, so this is just making that explicit in the spec. Last thing: hidden-partition transforms already give metadata-level skipping for monotonic cases with no extra stored column. lower(col) genuinely isn't covered, so the motivation is fair -- but saying where transforms already suffice and where a stored column is actually needed would make the proposal harder to argue with. Best, Andrei On Thu, Aug 6, 2026 at 12:28 PM Prashant Sharma <[email protected]> wrote: > Hello All, > > Recently, I have been involved in bringing derived columns(aka generated > columns) for the Iceberg connector in Presto Link: > https://github.com/prestodb/rfcs/pull/61. We currently use table > properties to store the extra metadata for derived columns. However, we > cannot achieve the following: > > 1. Cross engine compatibility: How do we know if the derived columns > metadata is in sync i.e. if other engines unaware of derived columns write > to the table/alter the columns etc... > 2. UDF portability and versioning: which is already addressed by (Iceberg > UDF spec) > 3. VIRTUAL columns: We store all the column metadata/metrics/stats > including a bloom filter but not the actual data. A poor man's index. > > We are considering a Iceberg spec level support to address all of the > above. Just wondering, is there work already happening in this direction > elsewhere? > > If not, I have started an issue: > https://github.com/apache/iceberg/issues/15923 , will be adding more > details soon. > > Thanks, > Prashant >
