danielcweeks commented on code in PR #16972: URL: https://github.com/apache/iceberg/pull/16972#discussion_r4109285534
########## format/spec.md: ########## @@ -347,9 +348,74 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may carry a **collation**, an attribute that changes how the field's values are compared and ordered without changing how they are stored. Collations enable case-insensitive, accent-insensitive, and locale-aware comparison and sorting. A collation only affects comparison: the stored value is returned unchanged (a value written as `'appLE'` is read back as `'appLE'`). + +This change defines exactly two things: the `collation` annotation on string fields, and collation-aware file pruning through collation bounds stored in `content_stats`. It does not give collation semantics to any other part of the table format, because collation order and equality are not stable across collation implementation versions or engines, while the rest of the format must be deterministic. Specifically: + +* Byte-order `lower_bounds`/`upper_bounds`, partition transforms (including `truncate`), and `bucket`/hash all operate on the binary UTF-8 value and are collation-unaware. Collation-equal but byte-distinct values may therefore land in different partitions or buckets, and a reader must not use partition or bucket pruning to eliminate candidates for a predicate evaluated under a collation. +* Equality-delete matching and identifier-field equality are binary value equality by field id; a `collation` annotation does not make either collation-aware. An engine that needs a collation-dependent delete must resolve it to position deletes or an explicit set of values. Review Comment: Equality deletes should not be relevant since they're slated for removal in v4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
