sfc-gh-aloeser commented on code in PR #16972: URL: https://github.com/apache/iceberg/pull/16972#discussion_r4405259116
########## format/spec.md: ########## @@ -386,9 +387,81 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may have a collation that defines how its values are compared and ordered, for example case-insensitive, accent-insensitive, or locale-aware. A collation does not change how values are stored: a value written as `'appLE'` is read as `'appLE'`. + +Any `string` field may have a collation, including nested fields. A struct field uses the `collation` attribute; a list or map uses `element-collation`, `key-collation`, or `value-collation` (see [Appendix C](#appendix-c-json-serialization)). A string field without a collation uses UTF-8 byte order. + +A collation changes comparison only. The rest of the format stays byte-based, because collation order differs between implementations and versions: + +* Partition transforms, `bucket`, and byte-order bounds use UTF-8 bytes. Values that are equal under a collation can be in different partitions or buckets, so readers must not use partition values to prune a predicate evaluated under a collation. Review Comment: I agree that transforms can be problematic for collations. Are we defining the behavior here (instead of prohibiting it) because we're worried that some non-collation aware engine could add the partitioning? ########## format/spec.md: ########## @@ -386,9 +387,81 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may have a collation that defines how its values are compared and ordered, for example case-insensitive, accent-insensitive, or locale-aware. A collation does not change how values are stored: a value written as `'appLE'` is read as `'appLE'`. + +Any `string` field may have a collation, including nested fields. A struct field uses the `collation` attribute; a list or map uses `element-collation`, `key-collation`, or `value-collation` (see [Appendix C](#appendix-c-json-serialization)). A string field without a collation uses UTF-8 byte order. + +A collation changes comparison only. The rest of the format stays byte-based, because collation order differs between implementations and versions: + +* Partition transforms, `bucket`, and byte-order bounds use UTF-8 bytes. Values that are equal under a collation can be in different partitions or buckets, so readers must not use partition values to prune a predicate evaluated under a collation. +* Identifier fields compare bytes. +* Map keys are distinct by bytes: `map['ABC']` and `map['abc']` are separate entries under a case-insensitive key collation. A lookup under the collation can match both, and the result is engine-defined. +* Sort orders do not use collations. + +SQL comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. + +##### Collation Names + +A collation name has the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider is the library that defines the collation. This spec defines one provider, `icu`: the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) with [CLDR](https://cldr.unicode.org/) locale data. Adding a provider requires a spec change. A reader must not use a collation from a provider it does not recognize. + +``` +specifier = icu-spec | utf8-spec +icu-spec = "icu" "." locale [ "-" case-mod ] [ "-" accent-mod ] [ "-" trim-mod ] +utf8-spec = "utf8" [ "-" trim-mod ] +locale = language [ "_" region ] ; e.g. en, de, en_US, de_DE +case-mod = "cs" | "ci" ; case sensitive (default) / insensitive +accent-mod = "as" | "ai" ; accent sensitive (default) / insensitive +trim-mod = "ltrim" | "rtrim" | "trim" ; trim leading / trailing / both spaces (U+0020) +``` + +Modifier order and case are fixed, so equal collations have equal names. Case- and accent-insensitive collations require a locale: `ł` and `l` are equal under `icu.en-ai` but not under `icu.pl-ai`. Review Comment: > equal collations have equal names I think there might still be some variance: since `cs` is the default value for `case-mod`, `icu.en` would behave the same as `icu.en-cs`. I think we can permit both, we'd just need to adjust the wording ########## format/spec.md: ########## @@ -386,9 +387,81 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may have a collation that defines how its values are compared and ordered, for example case-insensitive, accent-insensitive, or locale-aware. A collation does not change how values are stored: a value written as `'appLE'` is read as `'appLE'`. + +Any `string` field may have a collation, including nested fields. A struct field uses the `collation` attribute; a list or map uses `element-collation`, `key-collation`, or `value-collation` (see [Appendix C](#appendix-c-json-serialization)). A string field without a collation uses UTF-8 byte order. + +A collation changes comparison only. The rest of the format stays byte-based, because collation order differs between implementations and versions: + +* Partition transforms, `bucket`, and byte-order bounds use UTF-8 bytes. Values that are equal under a collation can be in different partitions or buckets, so readers must not use partition values to prune a predicate evaluated under a collation. +* Identifier fields compare bytes. +* Map keys are distinct by bytes: `map['ABC']` and `map['abc']` are separate entries under a case-insensitive key collation. A lookup under the collation can match both, and the result is engine-defined. Review Comment: this feels a little inconsistent to me, I'm not sure if there would be use cases. How do you feel about simply prohibiting collation on map keys? Map values should be fine ########## format/spec.md: ########## @@ -386,9 +387,81 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may have a collation that defines how its values are compared and ordered, for example case-insensitive, accent-insensitive, or locale-aware. A collation does not change how values are stored: a value written as `'appLE'` is read as `'appLE'`. + +Any `string` field may have a collation, including nested fields. A struct field uses the `collation` attribute; a list or map uses `element-collation`, `key-collation`, or `value-collation` (see [Appendix C](#appendix-c-json-serialization)). A string field without a collation uses UTF-8 byte order. + +A collation changes comparison only. The rest of the format stays byte-based, because collation order differs between implementations and versions: + +* Partition transforms, `bucket`, and byte-order bounds use UTF-8 bytes. Values that are equal under a collation can be in different partitions or buckets, so readers must not use partition values to prune a predicate evaluated under a collation. +* Identifier fields compare bytes. +* Map keys are distinct by bytes: `map['ABC']` and `map['abc']` are separate entries under a case-insensitive key collation. A lookup under the collation can match both, and the result is engine-defined. +* Sort orders do not use collations. + +SQL comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. + +##### Collation Names + +A collation name has the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider is the library that defines the collation. This spec defines one provider, `icu`: the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) with [CLDR](https://cldr.unicode.org/) locale data. Adding a provider requires a spec change. A reader must not use a collation from a provider it does not recognize. + +``` +specifier = icu-spec | utf8-spec +icu-spec = "icu" "." locale [ "-" case-mod ] [ "-" accent-mod ] [ "-" trim-mod ] +utf8-spec = "utf8" [ "-" trim-mod ] +locale = language [ "_" region ] ; e.g. en, de, en_US, de_DE +case-mod = "cs" | "ci" ; case sensitive (default) / insensitive +accent-mod = "as" | "ai" ; accent sensitive (default) / insensitive +trim-mod = "ltrim" | "rtrim" | "trim" ; trim leading / trailing / both spaces (U+0020) +``` + +Modifier order and case are fixed, so equal collations have equal names. Case- and accent-insensitive collations require a locale: `ł` and `l` are equal under `icu.en-ai` but not under `icu.pl-ai`. + +Without a trim modifier, trailing spaces are significant (NO PAD). `rtrim` gives PAD SPACE behavior, for example `icu.en_US-ci-rtrim`. + +`utf8` without a modifier is UTF-8 byte order, the same as no collation. + +##### Collation Versions + +Collation order can change between versions of a provider's data ([UTS #10](https://unicode.org/reports/tr10/#Non-Goals)). The schema stores the collation name without a version, and collation bounds record the version they were produced under. For `icu`, the version is the [ICU collator version](https://unicode-org.github.io/icu/userguide/collation/architecture.html#versioning), which changes when the collation order can change. Versions are compared as opaque strings. + +##### Collation Bounds + +Byte-order bounds cannot prune a collated field: under a case-insensitive collation `'a' < 'B'`, but in byte order `'a' > 'B'`. Collation bounds are kept in [stats-only fields](#stats-only-fields), outside the schema. A collated field has: + +* One `collation-value` stats-only field per collation version. Its type-specific fields are `source-id`, the id of the collated field; `collation`, the collation name; and `version`, the collation version. Its bounds are original string values, compared under that collation and version. The bound type is `string` and `data-type` is omitted. +* Two `expr-value` stats-only fields with `data-type` `int`, for `min_code_point(ref)` and `max_code_point(ref)` (see [Iceberg functions](expressions-spec.md#appendix-a-iceberg-functions)). The lower bound of the first and the upper bound of the second give the file's code-point range. They do not depend on the collation or its version. + +For example, for a field `city` with id `5` and collation `icu.de_DE` (`expr` in short form): + +``` +{ "field-id": 101, "type": "collation-value", "source-id": 5, "collation": "icu.de_DE", "version": "153.120.0.0" } +{ "field-id": 102, "type": "expr-value", "expr": min_code_point(ref(5)), "data-type": "int" } +{ "field-id": 103, "type": "expr-value", "expr": max_code_point(ref(5)), "data-type": "int" } +``` + +`collation-value` bounds must be valid but need not be tight: under the collation and version, `lower_bound` sorts at or before every value in the file, and `upper_bound` at or after. + +A writer should write byte-order bounds for a collated field, so readers without collation support can still prune. A writer that supports the collation should also write `collation-value` bounds for its version and code-point bounds, and adds these stats-only fields when they do not exist. It must not add a second `collation-value` field with the same `source-id`, `collation`, and `version`, and must not write `collation-value` bounds for a version it does not support. Review Comment: > A writer should write byte-order bounds for a collated field should we harden this to state that if a writer provides bounds for a collated field (which is desirable), those MUST be based on UTF-8 ordering? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
