Thanks Ryan, I could argue with the other reasons, but the "should not be displayed as part of the table schema" seals the discussion.
+1 Peter On Fri, Oct 23, 2026, 18:29 Ryan Blue <[email protected]> wrote: > I don't think that we are trying to solve the problem of generated columns > right now. I think that should be part of the schema so it is separate from > this work. > > You might be suggesting that we use generated columns to store stats > instead of the current proposal. I don't think that is a good idea because > it adds extra fields to the schema with different behavior that breaks > assumptions for schemas. > > For instance, stats-only fields are not fields in the schema and should > not be displayed as part of the table schema. Generated columns are part of > table data, which is why they belong in the schema. If we add stats-only > fields to the schema, then anyone consuming the schema needs to know what > to filter out before passing it along. > > Similarly, schema fields have specific types that change only when a user > modifies them. Stats fields are derived and can change when schema changes > affect them, so we would not be able to produce all field types without > binding the schema to a subset of itself. > > This would also complicate composition. Right now, stats-only fields are > not nested within the schema, they are independent. If we were allowing a > field to be a stat-only field then we have to come up with rules around > where they can be used (not in map keys?) and then check those rules. How > would you handle a struct with only one field that is a stats-only field > that is not actually present in the schema? > > The proposal is much simpler because we don't need to worry about many of > these concerns. > > Ryan > > On Thu, Oct 22, 2026 at 10:15 PM Péter Váry <[email protected]> > wrote: > >> Hi Ryan, >> >> Thanks for putting this together! >> >> I had the same impression when I took a look at the derived columns >> proposal, and we updated that. >> >> Now, a schema would look like this: >> >> schema { >> 1: id: required int >> 2: name: optional string >> 3: hilbert: optional long (generated: stats-only: hilbert(1, 2)) >> } >> >> Or in JSON: >> { >> [...] >> "fields": [ >> { >> "id": 3, >> "name": "id", >> "required": false, >> "type": "long", >> "generated": >> { >> "strategy": "stats-only", // on-write, on-read(?) >> "expression": [...] >> } >> }, >> [...] >> ] >> } >> >> This would help removing the extra rules around the id generation, stat >> generation and we likely need the generated columns too. >> >> Schema already contains expressions, elements that are not stored in the >> files, and stats-only fields are optional so just return `null` when >> queried. >> >> WDYT? >> >> Thanks, >> Peter >> >> >> On Fri, Oct 23, 2026, 00:56 Ryan Blue <[email protected]> wrote: >> >>> Hi everyone, >>> >>> We've been working on the spec changes for non-materialized columns in >>> v4, which was previously discussed here: >>> https://lists.apache.org/thread/4h53r1nvvgxqymfvx7vcyhqgh1fk8180 >>> >>> I think the spec is about ready. Please take a look here: >>> https://github.com/apache/iceberg/pull/18405 >>> >>> I ended up using "stats-only fields" since I think that's a more >>> straightforward name than "non-materialized columns". The changes are what >>> was discussed in the last thread. >>> >>> I also considered whether it would be a good idea to unify this with >>> materialized columns that are generated by an expression, like default >>> value expressions or generated columns. I decided not to do that for the >>> same reasons we chose not to separate expressions from check constraints. >>> That would require adding an expression and then adding a column that >>> depends on it. That indirection isn't needed and I think generated columns >>> should have an expression in the definition. >>> >>> Please take a look at the PR and comment. I'd like to start a vote soon >>> if there is consensus. Thanks! >>> >>> Ryan >>> >>>
