rdblue commented on code in PR #18405: URL: https://github.com/apache/iceberg/pull/18405#discussion_r4395521589
########## format/spec.md: ########## @@ -474,6 +474,27 @@ Two rows are the "same"---that is, the rows represent the same entity---if the i Identifier fields may be nested in structs but cannot be nested within maps or lists. Float, double, and optional fields cannot be used as identifier fields and a nested field cannot be used as an identifier field if it is nested in an optional struct, to avoid null values in identifiers. +#### Stats-only Fields + +Stats are tracked in manifests by field ID. Schema fields have assigned IDs, but additional field IDs may be assigned to track stats for derived values. For instance, lower and upper bounds for `to_lower_case(name)` are useful for case-insensitive file pruning. + +These additional _stats-only fields_ are used to track stats for derived values that are not part of the table schema and are not materialized. A stats-only field consists of: + +* A **`field-id`** assigned by incrementing the table's `last-column-id` +* A **`type`** that can be `partition-value` or `expr-value` +* An optional **`data-type`** that determines the bound type; must be a primitive or variant +* Type-specific fields that define how derived values are produced + +The `partition-value` type stores stats for the output of a partition field, identified by a `partition-field-id` type-specific field. The bound type is the partition field's result type and `data-type` is omitted. This may be used to store bucket partition values in stats for filtering. + +The `expr-value` type stores stats for the result of a [value expression](expressions-spec.md#value-expressions), stored in the `expr` field. Expressions must use only ID references. The output type of the value expression must be stored in the `data-type` field. + +Readers must not fail when an unsupported stats-only field `type` is found; stats for unsupported types must be ignored. Review Comment: The purpose of this statement is to ensure that we can add types to the spec later without breaking readers. For example, I think we would want to add a specific new `type` for collations rather than having an expression that gets bound. (Specifics here are a future discussion.) We could do the same thing for expressions, but it seems obvious that these can be ignored rather than used. That requirement would also mean that implementations cannot fail in some reasonable cases, like when an expression references an ID that is not present in the schema. When that happens, it's reasonable for readers to choose to fail or warn because the configuration is malformed. I probably wouldn't fail, but it makes sense to me to preserve the ability for implementations to make that choice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
