laskoviymishka commented on code in PR #16972: URL: https://github.com/apache/iceberg/pull/16972#discussion_r4060198075
########## format/spec.md: ########## @@ -345,9 +346,55 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may carry a **collation**, an attribute that changes how the field's values are compared and ordered without changing how they are stored. Collations enable case-insensitive, accent-insensitive, and locale-aware comparison and sorting. A collation only affects comparison: the stored value is returned unchanged (a value written as `'appLE'` is read back as `'appLE'`). + +This change defines exactly two things: the `collation` annotation on string fields, and collation-aware file pruning through collation metric fields stored in `content_stats`. It does not give collation semantics to any other part of the table format, because collation order and equality are not stable across collation implementation versions or engines, while the rest of the format must be deterministic. Specifically: + +* Byte-order `lower_bounds`/`upper_bounds`, partition transforms (including `truncate`), and `bucket`/hash all operate on the binary UTF-8 value and are collation-unaware. Collation-equal but byte-distinct values may therefore land in different partitions or buckets, and a reader must not use partition or bucket pruning to eliminate candidates for a predicate evaluated under a collation. +* Equality-delete matching and identifier-field equality are binary value equality by field id; a `collation` annotation does not make either collation-aware. An engine that needs a collation-dependent delete must resolve it to position deletes or an explicit set of values. +* This change does not define collation-aware sort orders. + +SQL-level comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. Engines read the `collation` annotation and apply their own semantics; they must not infer broader behavior from the table-format metadata alone. + +A collation may be attached to any field of `string` type, top-level or nested: as a `collation` attribute on a struct field, or as an `element-collation`, `key-collation`, or `value-collation` attribute on an enclosing `list` or `map` (see [Appendix C](#appendix-c-json-serialization)). It may be attached only to `string` types. A string field with no collation defaults to UTF-8 byte-order comparison, the behavior of all prior versions. Nesting does not change how collation bounds work: because every nested string position has its own field id, a collation on a nested string is annotated and given collation metrics exactly as a top-level string field (see [Collation Bounds](#collation-bounds)). Review Comment: Good question. I went back and forth on this. The main question is whether collation changes **map key identity**, or only how keys are **compared**. I think it should only affect comparison. So under a case-insensitive collation, `"ABC"` and `"abc"` are still two different map keys and both can exist in the same map. The collation only matters when the engine compares or looks up those keys. Why not make them the same key? 1. **It would make validity depend on the ICU version.** -- A map that has no duplicate keys under one version could suddenly have duplicates under another. 2. **It makes migration dangerous.** -- If you add a collation to an existing map column, keys that were distinct before could start colliding. The writer would then have to drop or merge data. 3. **Different engines could disagree.** -- If they use different ICU versions, they may not agree on which keys are duplicates. Keeping key identity byte-based avoids all of that. The stored map stays stable, and collation is only applied by the engine when it compares values. The trade-off is that a case-insensitive lookup could match both `"ABC"` and `"abc"`. I’d leave that behavior to the engine rather than define it in the format. I added this to the scope section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
