Hi all, This continues the collation work in #16972 [3]. We already aligned on keeping multiple collation versions per file in [4]. The remaining question is how to store the bounds.
Alex and I went through the options offline. I prototyped the two earlier layouts in iceberg-go [1]. On real manifests they end up within ~2.5% of each other after compression, so I don’t think size should drive the choice. Ryan’s stats-only fields proposal [2] gives us a cleaner way to do this without adding collation-specific fields to `content_stats`, which was Russell’s main concern with the current PR. The proposed shape is: - Collation bounds are stored as a new stats-only field type. The field points to the source column and records the collation + collation version. Lower/upper bounds are original string values ordered by that collation. They must be valid bounds, but do not need to be tight. - `min_code_point(col)` and `max_code_point(col)` are separate `expr-value` stats-only fields. They are ints and do not depend on collation version. - A collated column therefore has 2 + N stats-only fields, where N is the number of collation versions that have written stats. A file normally carries bounds for only the version its writer used. - If a writer uses a version that does not have a stats-only field yet, it adds one through a table metadata update. No schema change. - A reader uses bounds for its own version when present. It can also use another version when the file’s code-point range has the same ordering in both versions, for example some ASCII-only cases. Otherwise it just scans the file. - Readers without collation support ignore these fields like any other unsupported stats-only field type. - Old version fields can be removed later. That only means losing pruning on files that still have stats for that version. I don’t think plain `expr-value` is enough for the collation bounds themselves. The bounds are ordered by the collation, not by the physical type. Returning sort keys would make byte comparison work, but sort keys can change across collation versions even when the relative order does not, which would prevent reuse across versions. Which code-point ranges keep their order across versions is up to the engine. ICU does not publish this, so the format records the range and the engine decides whether it can reuse a bound. An engine that cannot tell scans the file. I updated #16972 [3] to match this direction: no collation-specific `content_stats`, one new stats-only field type for collation bounds, and `min_code_point` / `max_code_point` in the functions appendix. It depends on #18405. Comments welcome here or on the PR. [1] https://github.com/laskoviymishka/iceberg-go/tree/collation-size/inline and https://github.com/laskoviymishka/iceberg-go/tree/collation-size/derived [2] https://lists.apache.org/thread/t4nm0blbo33y5fnkcok6lrmdc4k4toxr and https://github.com/apache/iceberg/pull/18405 [3] https://github.com/apache/iceberg/pull/16972 [4] https://lists.apache.org/thread/4p8345gb8jo03l27qh86kbbpvolwc09j Thanks, Andrei
