sfc-gh-aloeser commented on code in PR #16972:
URL: https://github.com/apache/iceberg/pull/16972#discussion_r4631552764
##########
format/expressions-spec.md:
##########
@@ -228,6 +228,68 @@ Note that `year`, `month`, and `hour` transforms produce
ordinal values and not
[bucket-ref]: spec.md#bucket-transform-details
[truncate-ref]: spec.md#truncate-transform-details
+### Collation functions
+
+A collation defines how `string` values are compared and ordered, for example
case-insensitive, accent-insensitive, or locale-aware. These functions track
collations and collation-aware bounds in [stats-only
fields][stats-only-fields-ref], without changes to the table schema.
+
+| Function name | Description
| Source types | Result type |
+|--------------------------------------|-----------------------------------------------------------------------------|--------------|-------------|
+| `default_collate(value, collation)` | `value`, unchanged
| `string` | `string` |
+| `collate(value, collation, version)` | Sort key of `value` under `collation`
and `version`, followed by the UTF-8 bytes of `value` | `string` | `binary`
|
Review Comment:
IIRC we agreed to include the original string already. Do we gain much by
also including the sort key?
- pro: bounds are directly binary comparable
- con: bounds will be 2-3 larger (sort keys are longer than original string,
in most cases), and I believe it does not affect how much we can prune
Am I missing anything? If not, I'm leaning towards just keeping the original
strings. Engines that operate internally with sort keys could still derive them
from the original string. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]