danielcweeks commented on code in PR #16972: URL: https://github.com/apache/iceberg/pull/16972#discussion_r4109295571
########## format/spec.md: ########## @@ -347,9 +348,74 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may carry a **collation**, an attribute that changes how the field's values are compared and ordered without changing how they are stored. Collations enable case-insensitive, accent-insensitive, and locale-aware comparison and sorting. A collation only affects comparison: the stored value is returned unchanged (a value written as `'appLE'` is read back as `'appLE'`). + +This change defines exactly two things: the `collation` annotation on string fields, and collation-aware file pruning through collation bounds stored in `content_stats`. It does not give collation semantics to any other part of the table format, because collation order and equality are not stable across collation implementation versions or engines, while the rest of the format must be deterministic. Specifically: + +* Byte-order `lower_bounds`/`upper_bounds`, partition transforms (including `truncate`), and `bucket`/hash all operate on the binary UTF-8 value and are collation-unaware. Collation-equal but byte-distinct values may therefore land in different partitions or buckets, and a reader must not use partition or bucket pruning to eliminate candidates for a predicate evaluated under a collation. +* Equality-delete matching and identifier-field equality are binary value equality by field id; a `collation` annotation does not make either collation-aware. An engine that needs a collation-dependent delete must resolve it to position deletes or an explicit set of values. +* Map key identity is by stored (byte) value, not by collation. A collation on a map key does not merge keys: `map['ABC']` and `map['abc']` remain distinct entries and both may be present in the same map. A key collation affects only how predicates over keys compare — collation-aware key lookup may therefore match more than one stored key, and how that is resolved is engine-defined. +* This change does not define collation-aware sort orders. + +SQL-level comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. Engines read the `collation` annotation and apply their own semantics; they must not infer broader behavior from the table-format metadata alone. Review Comment: > they must not infer broader behavior from the table-format metadata alone Can you clarify this point? I don't understand what it is referring to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
