Hi all, Splitting out new thread from collation [1] main discussion - the key open decision left:
Short context about collations: collations are the rules for how strings are compared and sorted - e.g. treating 'Apple' = 'apple' (case-insensitive), 'résumé' = 'resume' (accent-insensitive), or sorting by a language's alphabet instead of raw byte order. Where we are: the proposal [2] / PR [3] has converged on collations annotation on the string field, bounds stored as original values, the ICU version kept with the metrics, rather than in the column type, and equality deletes / sort orders are out of scope for v1. What’s left is how a file’s collation bound and their version are presented in metadata. Here is a concrete problem example: ICU ordering isn't stable across versions, so collation bounds are tagged with the version they were computed under, and a reader can only use them if it can reproduce that version. Let's say Engine A writes data files on ICU 75 and records bounds under that version. A reader on ICU 76 opens one. Can it prune? Only if the file also carries ICU-76 bounds. If a file can hold just one version's bounds, the answer is no - the ICU-76 reader full-scans. And since engines upgrade ICU on their own cadence (rolling, mixed-version fleets), that skew is the normal state, not an edge case. The question the example raises: should one file be able to carry bounds for more than one version? If yes, there are two ways to store them. Option 1/ the schema declares a metric field per version. Multi-version by construction, and the reader resolves a usable version at planning time. Cost: the schema grows per version, and a writer on a version the schema doesn't list needs a schema change before it can produce bounds. Option 2/ inline, the version and bounds travel together in content_stats, as a repeated field so a file can hold several. No schema changes when a new version appears, and the version sits in the stats the reader already reads at planning time. The difference is what happens when a new ICU version appears: Option 1 needs a schema change to register it before anyone can write its bounds; Option 2 doesn't. My lean is the same as Alex’s recommendation [4]: keep this inline (opt 2), but mulit-version. It avoids the per-version schema evolution and solves gradual rollout skew. Longer term per file code point stats could let a reader reuse an older version’s bound when it can prove the ordering didn’t change for that file’s data (it can be inferred), this can be part of implementation details, what matters now layout. To wrap up a main question: 1/ one file staying prunable as engines cross ICU version - is this something we want in v1 or acceptable as later optimization? 2/ inside deployments of the engine is a single known ICU version realistic or this is expected to have several one live at once? The answers here will pick the layout, and i’d like to settle this with the broader community. [1] https://lists.apache.org/thread/44todz4x460g8pb89y8rpozlnmo8vdhc [2] https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit [3] https://github.com/apache/iceberg/pull/16972 [4] https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.50g4t7yl7w7t#bookmark=id.nxhabrnakurq Best, Andrei
