sfc-gh-aloeser commented on code in PR #16972:
URL: https://github.com/apache/iceberg/pull/16972#discussion_r4631669098
##########
format/expressions-spec.md:
##########
@@ -228,6 +228,68 @@ Note that `year`, `month`, and `hour` transforms produce
ordinal values and not
[bucket-ref]: spec.md#bucket-transform-details
[truncate-ref]: spec.md#truncate-transform-details
+### Collation functions
+
+A collation defines how `string` values are compared and ordered, for example
case-insensitive, accent-insensitive, or locale-aware. These functions track
collations and collation-aware bounds in [stats-only
fields][stats-only-fields-ref], without changes to the table schema.
+
+| Function name | Description
| Source types | Result type |
+|--------------------------------------|-----------------------------------------------------------------------------|--------------|-------------|
+| `default_collate(value, collation)` | `value`, unchanged
| `string` | `string` |
+| `collate(value, collation, version)` | Sort key of `value` under `collation`
and `version`, followed by the UTF-8 bytes of `value` | `string` | `binary`
|
+| `min_code_point(value)` | Smallest Unicode code point in
`value` | `string` | `int` |
+| `max_code_point(value)` | Largest Unicode code point in `value`
| `string` | `int` |
+
+`collation` and `version` must be string constants. All four functions produce
`null` for a `null` input. `min_code_point` and `max_code_point` also produce
`null` for an empty string.
+
+#### Collation names
+
+A collation name has the form `<provider>.<name>`, for example `icu.en_US-ci`.
The provider is the library that defines the collation. This spec defines one
provider, `icu`: the [Unicode Collation
Algorithm](https://unicode.org/reports/tr10/) with
[CLDR](https://cldr.unicode.org/) locale data. Adding a provider requires a
spec change.
+
+```
+specifier = icu-spec | utf8-spec
+icu-spec = "icu" "." locale [ "-" case-mod ] [ "-" accent-mod ] [ "-"
trim-mod ]
+utf8-spec = "utf8" [ "-" trim-mod ]
+locale = language [ "_" region ] ; e.g. en, de, en_US, de_DE
+case-mod = "cs" | "ci" ; case sensitive (default) /
insensitive
+accent-mod = "as" | "ai" ; accent sensitive (default) /
insensitive
+trim-mod = "ltrim" | "rtrim" | "trim" ; trim leading / trailing / both
spaces (U+0020)
+```
+
+Modifier order and case are fixed, so equal collations have equal names. Case-
and accent-insensitive collations require a locale: `ł` and `l` are equal under
`icu.en-ai` but not under `icu.pl-ai`.
+
+Without a trim modifier, trailing spaces are significant (NO PAD). `rtrim`
gives PAD SPACE behavior, for example `icu.en_US-ci-rtrim`.
+
+`utf8` without a modifier is UTF-8 byte order, the same as no collation.
+
+#### Collation versions
+
+Collation order and sort keys can change between versions of a provider's
data. `version` must identify the sort keys exactly: two implementations given
the same `collation`, `version`, and value must produce the same bytes. For
`icu`, `version` is the ICU version followed by the collator version, for
example `76.1/153.120.0.0`. Versions are compared as opaque strings. An
implementation that cannot produce the sort keys for a version must treat
`collate` with that version as unsupported.
+
+For `icu`, the sort key is the output of ICU's `getSortKey` with the settings
given by the collation name. It ends with a single `0x00` byte and contains no
other `0x00` bytes, so the value can be read back after the first `0x00`.
+
+#### Declaring a collation
+
+A `default_collate(ref(id), collation)` stats-only field declares that the
`string` field `id` uses `collation`. This applies to v4 tables only. Any
`string` field can be declared, including list elements and map keys and
values. A field has at most one declaration. Changing a collation replaces the
declaration in the same commit, and removing the declaration returns the field
to UTF-8 byte order. A field without a declaration uses UTF-8 byte order.
Review Comment:
recording an offline discussion with Andrei:
Ideally, we would like `collation` to be a part of the NestedField's
attributes, so the collation can be directly seen by looking at he field. We
started with that proposal originally, too.
The reason we changed it: It is currently not clear whether collations will
be part of the initial v4 release. If they're not, we can't (to the best of our
knowledge) just retroactively add a collation property to NestedField. There
might already be engines that support v4 tables but don't know about
collations. If any such engine rewrites the schema of a table, it might
silently drop collations (by not preserving them), without even realizing it.
Given the impact on performance and query behavior, the risk of such an
accidental drop seems too high.
So we've been looking for another, safer way to say "this column has
collation X" - in a way that engines supporting v4 can't accidentally drop.
Using a stat-only field is the best approach we found so far, but I think we're
both happy to go another route, if there's a better way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]