laskoviymishka commented on code in PR #16972: URL: https://github.com/apache/iceberg/pull/16972#discussion_r3501479758
########## format/spec.md: ########## @@ -345,6 +346,33 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may carry a **collation**, an attribute that changes how the field's values are compared and ordered without changing how they are stored. Collations enable case-insensitive, accent-insensitive, and locale-aware comparison and sorting. A collation only affects comparison: the stored value is returned unchanged (a value written as `'appLE'` is read back as `'appLE'`). + +A field's collation is stored as a `collation` attribute on the field (see [Appendix C](#appendix-c-json-serialization)). The attribute is allowed only on `string` fields. If a field has no `collation` attribute, comparison defaults to UTF-8 byte order, which is the behavior of all prior versions. + +A collation is identified by a provider-qualified name of the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider names the library that defines the collation (`icu` for collations defined by the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) over [CLDR](https://cldr.unicode.org/) locale data; other providers may define engine-specific collations such as case-folding variants). The name selects a locale and optional modifiers for case sensitivity (`ci`/`cs`), accent sensitivity (`ai`/`as`), trimming, and case folding. The reserved name `utf8` denotes UTF-8 byte-order comparison. + +The schema stores the collation **name without a version**, so any engine that supports the collation can read the table. UCA, DUCET, CLDR, and ICU collation orders are [not stable across versions](https://unicode.org/reports/tr10/#Non-Goals), so collation-aware metrics carry the implementation version they were produced under (see below) and a reader uses them only when it can produce the same order. + +##### Collation Bounds + +Because a collation can reorder values (for example `'a' < 'B'` under a case-insensitive collation, but `'a' > 'B'` in byte order), the byte-order `lower_bounds` and `upper_bounds` cannot be used to evaluate predicates on a collated column. Collation-aware bounds are stored separately in `data_file.collation_bounds`, a map from column id to a list of `collation_bound` structs: + +| Field id, name | Type | Description | +|----------------|------|-------------| +| **`151 collation`** | `string` | Collation the bounds were produced for, e.g. `icu.en_US-ci` | Review Comment: Agree on the generic multi-version direction - a file should be able to carry bounds for several collations and versions, resolved at planning time. On the keying, I'd push back on one field id per (collation, version). A collation version bump only changes a file's min/max if it actually reorders the specific values in that file. Most bumps touch a small set of characters, so for the large majority of columns the min and max are unchanged across adjacent versions. Keying by version stores the identical bound once per version — linear growth in ids and in stored bytes, with no new information. It also makes every version bump a schema change, which is awkward since the version is a property of the stats, not of the column's logical type. The cheaper encoding falls out of that observation: let a bounds entry declare the set of versions it's valid for (valid_for: [v71, v72, ...]) and add a new entry only when a bump actually changes the bound. One entry covers the common no-reorder case. A middle ground that keeps your field-id model: register one field id per collation in the schema (not per version), and let that metric entry hold the version-tagged bounds with the valid-for set inside. The field id stays discoverable at planning time and you can still carry several collations per file, but versions don't each cost an id or a schema change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
