Thanks for taking a look, Peter and Gianluca. For the result data type, I was thinking that we would require it if it is not a case where we know it can be derived. For instance, we know the output type for partition fields so we don't need to specify it and it can change. For cases where you may not know the output, like when you have an expression, we would require the data type. And I agree that we would use bound or id-based references.
I don't think that there is much value in having a special case for identity. You already have a field ID for the data, so I don't think there is a situation in which you would ever do this. Also, I see the distinction that Gianluca points out, but this is not an issue because there should be no functions in the second category. Maybe hilbert was a poor choice for an example. I don't think that we need to design for expressions that have hidden inputs. Ryan On Tue, Sep 1, 2026 at 4:17 AM Péter Váry <[email protected]> wrote: > Hi Ryan, > > Thanks for putting this together. This would make index definitions > significantly simpler. > > A couple of thoughts: > > - *Schema evolution*: In Iceberg expressions, the result type is > currently defined by the expression itself and may evolve as the table > schema changes (e.g., int -> long). Storing an explicit data-type would > make reading metadata and index data much simpler, but it does remove some > of that flexibility. > - *FieldId references*: It would probably be a good idea to require > expressions to reference fields by ID (e.g., field(15)) rather than by name > (col_a) to make them resilient to column renames. > - *Identity expressions*: I expect index definitions to frequently > contain entries like: > > > * {"field-id": 104, "type": "expr-value", "data-type": > "long", "expr": "<identity(col_a)>"} * > Would it be worth introducing a dedicated type for this case, such as: > > > * {"field-id": 104, "type": "identity-value", > "original-field-id": 15} * > It doesn't seem broadly useful outside indexing, but identity > projections in indexes are likely common enough that a more compact > representation may be worth considering. > > Thanks, > Peter > > Gianluca Graziadei <[email protected]> ezt írta (időpont: > 2026. szept. 1., K, 7:02): > >> Hi Ryan, >> I like the proposal; it is concise and clear. >> >> I would suggest making a clear distinction between two classes of >> expressions: >> >> 1. Expressions that produce deterministic values intrinsically tied to >> the input value, e.g. to_lower_case(s).This class is relatively >> straightforward to reason about, since the result depends only on the >> individual row. >> 2. Expressions such as Hilbert that produce values whose meaning depends >> on the input value + the whole distribution. Here the main issue is that >> the underlying distribution matters. If the distribution changes over time, >> hilbert(c11,c12) computed at time t may no longer be valid at t+1. This >> makes the second class harder to handle, because we need to establish for >> how long a computed value remains valid (=if in the current snapshot values >> computed on a previous snapshot are still valid). >> >> For this second class, I don't think persistence of the computed value >> alone is sufficient. We may also need to persist the distribution (or its >> CDF) against which the value was computed (I am not confident that min/max >> can be good enough in this case). >> >> Before I comment: >> 1. Are you targeting to manage these two classes in the same way? It >> seems that they have potentially different validity scope. >> 2. What about adding a snapshot level CDF struct per column? >> >> Cheers, >> Gianluca >> >> Il giorno mar 1 set 2026 alle ore 00:25 Ryan Blue <[email protected]> ha >> scritto: >> >>> Hi everyone, >>> >>> One of the remaining open questions for v4 metadata is how we will >>> assign table field IDs for values that are not written into the table. I >>> want to propose a solution that I think is going to be flexible, while not >>> introducing a lot of churn in the table or REST specs. >>> >>> Columnar field stats are written into metadata using a simple function >>> from table field ID to metadata field ID. We want to reuse what we already >>> have working for table fields and keep the spec simple. That means we need >>> a way to assign a table field ID to a non-materialized column so that we >>> can track its stats. There are a few cases we’ve identified: >>> >>> - Partition field output for non-monotonic functions, like bucket(1024, >>> id) >>> - Clustering expressions, like to_lower_case(last_name) >>> - Collation sequence lower and upper bounds >>> >>> We also discussed a new case this morning in the index sync: we need a >>> field ID for a derived value used to organize an index, like hilbert(col_a, >>> col_b), because we intend to use table field IDs in index schemas. >>> >>> Initially, I suggested that we keep a table of expressions and assign >>> each one a field ID. But as we started thinking about the use cases where >>> we need expressions it became clear that denormalizing *all* >>> expressions was adding a lot of complexity for little benefit. For example, >>> CHECK constraints won’t have reusable expressions and it makes little >>> sense to create them in two parts (expression and constraint). Similarly, >>> it is awkward to model a collation sequence as an expression, and we don’t >>> need to rebuild partition specs just to assign field IDs. However, we also >>> don’t want to just embed table field IDs in every one of these structures. >>> >>> My proposal is to directly model what we want: one table of fields that >>> are not materialized in the table, but are assigned IDs for stats or other >>> purposes. This would take a few forms: >>> >>> - Partition output value: {"field-id": 102, "type": >>> "partition-value", "partition-field-id": 1000} >>> - Collation sequence: {"field-id": 103, "type": "collation-bounds", >>> "collation-seq-id": 1} >>> - Value expression results: {"field-id": 104, "type": "expr-value", >>> "data-type": "long", "expr": <hilbert(col_a, col_b) expr>} >>> >>> This representation leaves existing structures alone and is a single >>> place outside of schema to allocate table field IDs. This can be expanded >>> with new types later when we want to add new structures, like a cluster-by >>> spec. >>> >>> I think this is a fairly clean way to move forward and solve two >>> challenges that we’re currently hitting. We'll discuss this in the next v4 >>> metadata sync, but in the meantime please reply with feedback if you have >>> an opinion. >>> >>> Thanks, >>> >>> Ryan >>> >>
