Hi all,
Splitting one open decision out of the collation thread [1] / PR #16972 [2]
into its own thread, because it fixes the on-disk metadata layout and I'd
like to lock it.
Some background: a collated column stores collation-aware min/max bounds so
it stays prunable. ICU ordering isn't stable across versions, so each bound
is tagged with the collation version it was computed under. A reader uses
that bound only if it can reproduce the same version; otherwise it scans
the file. That only affects pruning, not correctness.
The decision: should one data file be able to carry bounds for more than
one collation version?
-
Yes: bounds are a per-file list of {version, lower, upper}. A file
written under ICU X can still be pruned by a reader on ICU Y once a Y bound
is added, for example by compaction. Cost is a slightly larger stats struct.
-
No: one version per file. Simpler struct, but a reader on another ICU
version cannot prune that file until it is rewritten. In mixed-version
deployments that can mean a lot more scans during upgrades.
My preference is multi-version. In Databricks Runtime customers don't pin
an ICU version, so different versions can be live at the same time during
upgrades - under one-version-per-file that becomes the common case, not an
edge case. That's just the Databricks view, though; I'm asking below
because if other engines share the constraint, multi-version is the safer
default. The PR already implements it.
Two questions, especially for other engines:
1.
In your deployments, is a single known ICU version realistic, or do you
expect several live at once?
2.
Any objection to locking the multi-version layout so we can settle the
field IDs and take the PR out of draft?
[1] https://lists.apache.org/thread/44todz4x460g8pb89y8rpozlnmo8vdhc
[2] https://github.com/apache/iceberg/pull/16972
Thanks,
Andrei
