What if we handle non-materialized columns using the derived, but virtual columns. See: https://docs.google.com/document/d/1u93kyIrTc8bt9VSTB4O0GM9S7zuXekqixJR6Id2FZC4 ?
Péter Váry <[email protected]> ezt írta (időpont: 2026. szept. 4., P, 16:58): > Index would also use a "Sort Order" functionality. > If you take a look at the proposed spec, I have defined Cluster spec > <https://github.com/apache/iceberg/pull/16961/changes#diff-327b0c238ffecd1fd8a6c9a87cdc9bc9c5d552203851997acdaf5c1dc0c1f795R143> > as > a list of materialized, and non-materialized fields, and sort the data in > the index based on that. If we redefine sort order, then we can reuse > that with the index too. > We could also reuse the Group max value > <https://github.com/apache/iceberg/pull/16961/changes#diff-327b0c238ffecd1fd8a6c9a87cdc9bc9c5d552203851997acdaf5c1dc0c1f795R322> > and > maybe add Group min value statistics too, which would define the ranges > available in the given data file. > > This proposal would make expressing these things way easier! > > Thanks, > Peter > > Gianluca Graziadei <[email protected]> ezt írta (időpont: > 2026. szept. 3., Cs, 22:13): > >> +1 (non-binding) >> Thanks, that's exactly the distinction I was trying to get at. If the >> parametrization is fully captured by the expression in metadata, then we're >> aligned: there are no hidden inputs and the expression remains a >> well-defined transformation of the row. >> >> - Gianluca >> >> Il giorno gio 3 set 2026 alle ore 21:58 Ryan Blue <[email protected]> ha >> scritto: >> >>> Thanks for the background and hilbert details. I understand what you're >>> saying, but I don't think it affects this work. However we parameterize >>> this function, what we store in metadata needs to completely describe the >>> function's operation. We are concerned with functions that are well-defined >>> transformations of a single input row. That could be something like >>> `hilbert(zvalue(col1, 0, 1), zvalue(col2, 100, 50))` to account for what >>> you're talking about. >>> >>> On Thu, Sep 3, 2026 at 12:17 PM Gianluca Graziadei < >>> [email protected]> wrote: >>> >>>> Hi Ryan, >>>> >>>> You're right, and apologies: I left the actual case implicit and used >>>> hilbert(c1, c2) as shorthand for it. With a fixed per-column bit width the >>>> curve index is a pure function of the row, so as written my example doesn't >>>> support my point. Let me state the real case explicitly. >>>> >>>> Hilbert clustering over heterogeneous columns doesn't work on raw >>>> values. Each column has a different domain, scale and distribution, and if >>>> you feed them to the curve as-is, most of the bits of one dimension end up >>>> unused and the clustering degenerates along it. >>>> This is not hypothetical: it came out of the discussion with Russell on >>>> the Hilbert PR, and it's what the PR 17893 implements on top of the >>>> MultiColumn Term refactor (see also this draft >>>> https://github.com/GGraziadei/iceberg/pull/1). >>>> >>>> So in practice each input goes through a per-column transform that maps >>>> it onto the available bits, and the expression is: *hilbert(linear(c1), >>>> zstdnorm(c2), quantile(c3)) *where linear is a min/max rescaling, >>>> zstdnorm a z-score standardization, and quantile a mapping onto quantile >>>> boundaries. >>>> >>>> I don't think this needs a new structure because the proposal already >>>> fits it, provided two things are explicit in the spec (RFC 2119 ): >>>> >>>> - expr MUST be a deterministic function of the referenced field values >>>> only; any constant, including fitted parameters, MUST be embedded as a >>>> literal in the expression. This is what makes "no hidden inputs" a >>>> checkable property rather than an assumption. >>>> - Entries in the non-materialized fields table MUST be immutable: >>>> changing an expression, including re-fitting its parameters, allocates a >>>> new field ID, and IDs MUST NOT be reused (maybe this is implicit). >>>> >>>> I think this is also where Russell's two points land. A sort order over >>>> an expression like the one above is precisely the case where you want to >>>> save the computed values, and the lineage question is the transformation >>>> re-fit case: under rule 2 the field ID itself carries the lineage, so I >>>> don't think a separate ExpressionsID is needed. If instead an entry can be >>>> updated in place under the same ID, then we do need versioning. Is >>>> immutability the intent? >>>> >>>> I'll park the CDF suggestion (I will detach in a separate thread that >>>> could be useful also for optimizing the Spark physical plan): it's about >>>> stats representation, not id allocation however it is important at this >>>> level to understand this exigence (=validity scope for expressions) and >>>> plan according to it. >>>> >>>> Cheers, >>>> Gianluca >>>> >>>> Il giorno gio 3 set 2026 alle ore 20:23 Russell Spitzer < >>>> [email protected]> ha scritto: >>>> >>>>> This makes sense for me. I think as we do this we should also define >>>>> an output for "Sort Orders" since that's the other main function I think >>>>> we >>>>> have that we would want to save values for and it should look pretty >>>>> similar to partition output (just mimicking the input values).) >>>>> >>>>> >>>>> Since these wouldn't live in the "Schema' are we also tracking the >>>>> lineage of these expressions over time? Does a table have both a SchemaID >>>>> and ExpressionsID? >>>>> >>>>> On Thu, Sep 3, 2026 at 1:17 PM Ryan Blue <[email protected]> wrote: >>>>> >>>>>> Thanks for taking a look, Peter and Gianluca. >>>>>> >>>>>> For the result data type, I was thinking that we would require it if >>>>>> it is not a case where we know it can be derived. For instance, we know >>>>>> the >>>>>> output type for partition fields so we don't need to specify it and it >>>>>> can >>>>>> change. For cases where you may not know the output, like when you have >>>>>> an >>>>>> expression, we would require the data type. And I agree that we >>>>>> would use bound or id-based references. >>>>>> >>>>>> I don't think that there is much value in having a special case for >>>>>> identity. You already have a field ID for the data, so I don't think >>>>>> there >>>>>> is a situation in which you would ever do this. >>>>>> >>>>>> Also, I see the distinction that Gianluca points out, but this is not >>>>>> an issue because there should be no functions in the second category. >>>>>> Maybe >>>>>> hilbert was a poor choice for an example. I don't think that we need to >>>>>> design for expressions that have hidden inputs. >>>>>> >>>>>> Ryan >>>>>> >>>>>> On Tue, Sep 1, 2026 at 4:17 AM Péter Váry < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Ryan, >>>>>>> >>>>>>> Thanks for putting this together. This would make index definitions >>>>>>> significantly simpler. >>>>>>> >>>>>>> A couple of thoughts: >>>>>>> >>>>>>> - *Schema evolution*: In Iceberg expressions, the result type is >>>>>>> currently defined by the expression itself and may evolve as the >>>>>>> table >>>>>>> schema changes (e.g., int -> long). Storing an explicit data-type >>>>>>> would >>>>>>> make reading metadata and index data much simpler, but it does >>>>>>> remove some >>>>>>> of that flexibility. >>>>>>> - *FieldId references*: It would probably be a good idea to >>>>>>> require expressions to reference fields by ID (e.g., field(15)) >>>>>>> rather than >>>>>>> by name (col_a) to make them resilient to column renames. >>>>>>> - *Identity expressions*: I expect index definitions to >>>>>>> frequently contain entries like: >>>>>>> >>>>>>> >>>>>>> * {"field-id": 104, "type": "expr-value", "data-type": >>>>>>> "long", "expr": "<identity(col_a)>"} * >>>>>>> Would it be worth introducing a dedicated type for this case, >>>>>>> such as: >>>>>>> >>>>>>> >>>>>>> * {"field-id": 104, "type": "identity-value", >>>>>>> "original-field-id": 15} * >>>>>>> It doesn't seem broadly useful outside indexing, but identity >>>>>>> projections in indexes are likely common enough that a more compact >>>>>>> representation may be worth considering. >>>>>>> >>>>>>> Thanks, >>>>>>> Peter >>>>>>> >>>>>>> Gianluca Graziadei <[email protected]> ezt írta >>>>>>> (időpont: 2026. szept. 1., K, 7:02): >>>>>>> >>>>>>>> Hi Ryan, >>>>>>>> I like the proposal; it is concise and clear. >>>>>>>> >>>>>>>> I would suggest making a clear distinction between two classes of >>>>>>>> expressions: >>>>>>>> >>>>>>>> 1. Expressions that produce deterministic values intrinsically tied >>>>>>>> to the input value, e.g. to_lower_case(s).This class is relatively >>>>>>>> straightforward to reason about, since the result depends only on the >>>>>>>> individual row. >>>>>>>> 2. Expressions such as Hilbert that produce values whose meaning >>>>>>>> depends on the input value + the whole distribution. Here the main >>>>>>>> issue is >>>>>>>> that the underlying distribution matters. If the distribution changes >>>>>>>> over >>>>>>>> time, hilbert(c11,c12) computed at time t may no longer be valid at >>>>>>>> t+1. >>>>>>>> This makes the second class harder to handle, because we need to >>>>>>>> establish >>>>>>>> for how long a computed value remains valid (=if in the current >>>>>>>> snapshot >>>>>>>> values computed on a previous snapshot are still valid). >>>>>>>> >>>>>>>> For this second class, I don't think persistence of the computed >>>>>>>> value alone is sufficient. We may also need to persist the >>>>>>>> distribution (or >>>>>>>> its CDF) against which the value was computed (I am not confident that >>>>>>>> min/max can be good enough in this case). >>>>>>>> >>>>>>>> Before I comment: >>>>>>>> 1. Are you targeting to manage these two classes in the same way? >>>>>>>> It seems that they have potentially different validity scope. >>>>>>>> 2. What about adding a snapshot level CDF struct per column? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Gianluca >>>>>>>> >>>>>>>> Il giorno mar 1 set 2026 alle ore 00:25 Ryan Blue <[email protected]> >>>>>>>> ha scritto: >>>>>>>> >>>>>>>>> Hi everyone, >>>>>>>>> >>>>>>>>> One of the remaining open questions for v4 metadata is how we will >>>>>>>>> assign table field IDs for values that are not written into the >>>>>>>>> table. I >>>>>>>>> want to propose a solution that I think is going to be flexible, >>>>>>>>> while not >>>>>>>>> introducing a lot of churn in the table or REST specs. >>>>>>>>> >>>>>>>>> Columnar field stats are written into metadata using a simple >>>>>>>>> function from table field ID to metadata field ID. We want to reuse >>>>>>>>> what we >>>>>>>>> already have working for table fields and keep the spec simple. That >>>>>>>>> means >>>>>>>>> we need a way to assign a table field ID to a non-materialized column >>>>>>>>> so >>>>>>>>> that we can track its stats. There are a few cases we’ve identified: >>>>>>>>> >>>>>>>>> - Partition field output for non-monotonic functions, like >>>>>>>>> bucket(1024, >>>>>>>>> id) >>>>>>>>> - Clustering expressions, like to_lower_case(last_name) >>>>>>>>> - Collation sequence lower and upper bounds >>>>>>>>> >>>>>>>>> We also discussed a new case this morning in the index sync: we >>>>>>>>> need a field ID for a derived value used to organize an index, like >>>>>>>>> hilbert(col_a, >>>>>>>>> col_b), because we intend to use table field IDs in index schemas. >>>>>>>>> >>>>>>>>> Initially, I suggested that we keep a table of expressions and >>>>>>>>> assign each one a field ID. But as we started thinking about the use >>>>>>>>> cases >>>>>>>>> where we need expressions it became clear that denormalizing *all* >>>>>>>>> expressions was adding a lot of complexity for little benefit. For >>>>>>>>> example, >>>>>>>>> CHECK constraints won’t have reusable expressions and it makes >>>>>>>>> little sense to create them in two parts (expression and constraint). >>>>>>>>> Similarly, it is awkward to model a collation sequence as an >>>>>>>>> expression, >>>>>>>>> and we don’t need to rebuild partition specs just to assign field IDs. >>>>>>>>> However, we also don’t want to just embed table field IDs in every >>>>>>>>> one of >>>>>>>>> these structures. >>>>>>>>> >>>>>>>>> My proposal is to directly model what we want: one table of fields >>>>>>>>> that are not materialized in the table, but are assigned IDs for >>>>>>>>> stats or >>>>>>>>> other purposes. This would take a few forms: >>>>>>>>> >>>>>>>>> - Partition output value: {"field-id": 102, "type": >>>>>>>>> "partition-value", "partition-field-id": 1000} >>>>>>>>> - Collation sequence: {"field-id": 103, "type": >>>>>>>>> "collation-bounds", "collation-seq-id": 1} >>>>>>>>> - Value expression results: {"field-id": 104, "type": >>>>>>>>> "expr-value", "data-type": "long", "expr": <hilbert(col_a, col_b) >>>>>>>>> expr>} >>>>>>>>> >>>>>>>>> This representation leaves existing structures alone and is a >>>>>>>>> single place outside of schema to allocate table field IDs. This can >>>>>>>>> be >>>>>>>>> expanded with new types later when we want to add new structures, >>>>>>>>> like a >>>>>>>>> cluster-by spec. >>>>>>>>> >>>>>>>>> I think this is a fairly clean way to move forward and solve two >>>>>>>>> challenges that we’re currently hitting. We'll discuss this in the >>>>>>>>> next v4 >>>>>>>>> metadata sync, but in the meantime please reply with feedback if you >>>>>>>>> have >>>>>>>>> an opinion. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Ryan >>>>>>>>> >>>>>>>>
