Hi Alex,
Thanks, these are the right questions. Let me answer them, but I
think all three are really facets of one decision worth pulling out,
so I'll do that at the end.
Original values vs sort keys. I don't think the two limitations are
symmetric. You're right that a bound stored under version X may not
hold under Y for either representation, and a naive reader prunes
only on an exact version match either way. But they degrade
differently: a sort key from X is incomparable under Y (nothing a Y
reader can do with it) while an original value is the actual string,
so a Y reader can re-interpret it. In the case you describe at the
end of your mail, where only a small code-point range moved and a
file's values fall outside it, that reader can prove the X bound
still holds and prune across versions. Sort keys foreclose that;
original values keep it open. So original values are a superset:
worst case they match sort keys, best case they prune across versions.
Your two sort-key advantages are real, I just don't think they belong
in the format. Truncation: agreed it's hard for collated strings
(your abcเก contraction case is exactly the trap), so I sidestepped
it, collation bounds must be tight, a writer that can't store the
exact min/max omits the bound. Collation-aware truncation with
CollationElementIterator is a possible later optimization. Compare
cost: pruning is per-file at planning time, not per-row, so collator
vs byte compare is in the noise; and an engine that wants the byte
path can derive and cache the sort key from the stored value.
Original values don't block that, they just don't bake a
version-specific encoding into the format.
Column vs file-level version. As you say, every engine can read
regardless, so this is a pruning-performance choice, not correctness.
In the schema-registered-metrics design a file carries bounds under a
declared (collation, version), and a reader prunes any file with a
metric for a version it can produce, not only files it wrote. So
convergence on one ICU version gives full cross-engine pruning, same
as column-level, and a writer or compaction can populate several
versions at once. It gives the column-level benefit by convention
without making a version bump a format-breaking change.
[One data point from the engine side, since I'm coming at this from
the Databricks runtime: our runtime is effectively versionless
(customers don't pin an ICU version, and upgrades happen under them)
so "the same table read by clients on different ICU versions" isn't a
corner case for us, it's the default. That's what pushes me toward
per-file versioning: pinning one version per table or column means
either forcing the whole fleet to upgrade in lockstep or breaking
pruning on every bump, and neither survives a versionless fleet. And
in practice most version bumps we've gone through don't reorder the
data in a given column at all, which is exactly why keeping original
values, and eventually your code-point-range idea, lets a reader keep
pruning across a bump instead of falling back to a scan.]
Providers. Agreed. I'll tighten the spec to a registered set like
geo, icu to start, utf8 reserved, non-ICU collations added by spec
change rather than ad hoc. Interop is the whole point and an open
namespace undercuts it.
-----------
Stepping back: I think the three above collapse into one question
that needs broader alignment than the two of us:
> how much cross-engine pruning interoperability should the format
guarantee, versus leave to convention?
Original-vs-sortkey, column-vs-file version, and open-vs-restricted
providers are all that same tradeoff from different angles. It's a
values call more than a correctness one, and it binds every engine
that hasn't weighed in yet: Trino, Flink, Spark, PyIceberg, rust.
From the DBR side I can say the multi-version case is real rather
than theoretical, but that's one engine's vantage point. I'd like to
get the interop question in front of the other implementers before we
fix field IDs, the dev list is probably enough for now, and a
community sync is there if it needs more than async. No rush on that;
I'd rather let the thread settle the mechanics first.
Best,
Andrei
On Fri, Jul 3, 2026 at 12:22 AM Alexander Löser
<[email protected]> wrote:
Hi Andrei,
Thanks for putting together the spec PR and the detailed
write-up! The approach mostly looks solid to me. I have a few
questions/initial thoughts regarding the changes you proposed
(compared to the original proposal):
> 1 - Bounds store original values, not sort keys, tagged with a
per-file collation version. ICU/CLDR sort keys aren't stable
across versions, so storing keys ties every reader to one exact
version; original values plus a per-file version (readers prune
only on an exact match) degrade gracefully instead of breaking.
The schema keeps the collation name unversioned so anyone can read
If I understand correctly, we’re talking about two separate
things here:
1. Tagging a column vs a single file with a certain ICU version
2. Using collation keys vs original strings (the ones that will
produce the min/max collation keys)
For 1, it comes down to a tradeoff:
* If we tag the column with the ICU version, we’d force engines
to support one agreed-on ICU version if they want to prune
files. Engines would be able to prune every file (if they
support the specific ICU version), or none at all, so there
is more incentive to support a specific version
* If we tag individual files with ICU versions, we gain the big
advantage that engines do not need to agree on a single ICU
version. However, if I understand correctly, this comes at
the cost of “fractured” pruning - engines will only be able
to prune files that were written by themselves (or rather,
with the same ICU version). As a consequence, performance
might not really be interoperable between different engines.
Regardless of the approach we choose, all engines should be able
to read the data - they might just not be able to prune files.
For 2, I’m not sure if I understand the advantages of original
strings yet. As you already pointed out, the collation keys
depend on the ICU version. However, if I understand correctly,
the same limitation would apply to the original strings: the sort
order may (and does) change between different ICU versions, too.
As a consequence, we can’t assume the original lower/upper bound
strings we stored for version X will also be lower/upper bounds
for version Y - at least in the general case. So if I understand
correctly, we would not gain additional pruning opportunities
compared to using collation keys. Or am I missing something here?
At the same time, sort keys do have advantages:
* Iceberg allows the truncation of upper- and lower bounds.
This is trivial for binary collation keys. For original
strings, the task becomes significantly harder: truncating at
a character boundary, for example, would lead to wrong
results, as there are some context-sensitive sequences: e.g.,
with the CLDR root locale, abcเก < abcเ. I think it might be
doable with ICU’s CollationElementIterator, but it will be
tricky to get right.
* Lower/upper bounds are computed once, but will be compared
many times. With original strings, we would need to either
convert to the collation key on the fly, or use ICU’s
collator for a direct comparison. Both options will be slower
than a raw byte-sequence comparison
There is one scenario where original values would shine, though.
I analyzed the order-changes between various ICU versions: in
many cases, only a small range of code points changes/is moved.
If we had additional metadata about which code point ranges a
file contains (e.g., whether it is ASCII only), engines might be
able to prove that the original string bounds for version X are
still valid for version Y.
If I'm not mistaken, this could allow to prune across different
ICU versions in certain situations, which I’d consider a point in
favor of original values (and file-level ICU versions).
> 2 - A provider-qualified identifier (icu.en_US-ci), leaving
room for non-ICU collations like Spark's UTF8_LCASE, rather than
assuming ICU as the sole provider.
Adding a provider-mechanism sounds like a good approach to keep
the spec open for future collations :slightly_smiling_face: I
wonder whether we should restrict the set of allowed providers,
though, similar to how it was done with geo
<https://lists.apache.org/thread/r5x0do8f241bpf565rx8s5s3wc9ogp0f>.
My main motivation for this proposal is interoperability. I worry
that interoperability might suffer or vanish completely if every
engine can come up with their own definitions.
Happy to hear your thoughts on this!
Best, Alex
On 6/27/26 01:37, Szehon Ho wrote:
Very nice direction, left some comments on the spec proposal.
Thanks to you folks for working on it !
Szehon
On Fri, Jun 26, 2026 at 3:29 AM Andrei Tserakhau via dev
<[email protected]> wrote:
Hi all,
I've spend some cycle on the collation discussion and make
something more concrete to react to: a spec-change PR plus
reference implementations (go and java).
- Spec change (apache/iceberg#16972): a "collation"
annotation on string fields, and a
data_file.collation_bounds field so collated columns stay
prunable.
- Reference implementation in iceberg-go
(apache/iceberg-go#1318): the full path end to end - schema
annotation, collation-aware comparison (CLDR/UCA), collation
bounds in the manifest, and version-gated data-file pruning,
with an Avro round-trip and pruning tests.
- A lightweight Java POC (link below): the schema annotation
plus a Collator-backed comparator, to match where the
discussion is. I deliberately left the manifest/bounds side
out of Java for now.
The design follows the original proposal but takes a few
different turns, mostly to adopt what we learned in Delta.
The ones I'd most like input on:
1 - Bounds store original values, not sort keys, tagged with
a per-file collation version. ICU/CLDR sort keys aren't
stable across versions, so storing keys ties every reader to
one exact version; original values plus a per-file version
(readers prune only on an exact match) degrade gracefully
instead of breaking. The schema keeps the collation name
unversioned so anyone can read.
2 - A provider-qualified identifier (icu.en_US-ci), leaving
room for non-ICU collations like Spark's UTF8_LCASE, rather
than assuming ICU as the sole provider.
3 - One structural question I don't have a strong opinion on
yet: I put collation_bounds on data_file as a standalone v3
field, but field id 146 is already the v4 content_stats
struct, and collation bounds might belong inside that
typed-stats framework instead. Worth settling before we fix
field ids.
The full set of differences and the reader/writer rules are
in the PR description and the write-up. Comments very
welcome — both on the calls above and on whether the
standalone-field vs content_stats direction is the right one.
Best, Andrei
- original proposal:
https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.0
- spec change: https://github.com/apache/iceberg/pull/16972
- POC in go: https://github.com/apache/iceberg-go/pull/1318
- java POC:
https://github.com/laskoviymishka/iceberg/tree/prototype/collation-support
On Mon, Mar 30, 2026 at 10:54 PM Alexander Löser
<[email protected]> wrote:
Hi Andrei,
I'm glad you're interested. Looking forward to
collaborate with you!
Thanks for all the feedback here and in the doc. I only
had a quick glance, but I think you raised some good
points. I'll address/respond to your comments as soon
as I get the chance, hopefully tomorrow.
I think you also left some comments in this mail that
are not yet in the doc - I'll move those to a dedicated
section at the end of the doc, so we can use the doc as
a single source of truth/discussion.
> Happy to share our Delta design doc and implementation
learnings in more detail.
Sure, sounds good :)
Best,
Alex
On 3/29/26 01:25, Andrei Tserakhau via dev wrote:
Hi Alexander,
This looks really interesting. We've been working on
collation support in Delta and have shipped it in
production for some time, so this is an area we care
about a lot. If this proposal moves forward we'd be
happy to collaborate on the design and implementation.
The pseudo-field approach for collation metrics is
clean and composes well with existing Iceberg
infrastructure. The specifier coverage is comprehensive.
A few areas worth discussing as this evolves:
1 - Sort key stability and versioning
ICU sort keys are not stable across versions, so a
pinned ICU version bump in a future Iceberg release
would invalidate all existing collation metrics. In
multi-engine environments, requiring all engines to
converge on one ICU version is unrealistic.
We store original string values instead of sort keys
and allow per-file version annotations -- worth
discussing whether something similar could work here.
2 - Provider abstraction
The proposal assumes ICU as the sole provider, but
Spark ships non-ICU collations like UTF8_LCASE that are
widely used. A provider or namespace layer would
prevent name collisions and support engine-specific
collations without future spec changes.
3 - Operational surface
A few things that turned out correctness-critical in
our implementation: partition transforms on collated
columns (collation-equal but byte-distinct values in
different directories), sort order semantics, equality
deletes under collation, and Parquet filter pushdown
(must be disabled since Parquet has no collation concept).
These don't all need to be solved in v1 but would help
to scope them.
4 - Smaller items (nit's)
UTF-8 bounds for the original field id should be "must
write" not "should" -- otherwise backward compat breaks
for non-aware engines. Engine fallback behavior
(case-sensitive vs older ICU vs fail) could use a
recommended preference order to avoid divergent results
across engines. The collation specifier syntax would
benefit from a formal grammar.
---
Happy to share our Delta design doc and implementation
learnings in more detail. Looking forward to the
discussion.
Best,
Andrei
On Sat, Mar 28, 2026 at 11:49 PM Alexander Löser
<[email protected]> wrote:
Hi everyone,
this is my first interaction with the Iceberg
community, so here a few words about myself:
- I'm Alex, a Berlin-based software engineer
- I've been working at Snowflake for 4 years now
- I spend most of my time on data types,
particularly binary, strings and collations.
I'd like to start a discussion about adding
collations to the Iceberg spec.
Conceptually, collations are an annotation on the
string data type. By default, most engines perform
string operations case-sensitively.
Collations allow specifying alternative comparison
rules. This is useful for achieving, e.g., case- or
accent-insensitive string operations, or
language-specific string sorting.
Collations are supported by many engines:
Databricks
<https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-collation>,
Spark
<https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.collate.html>,
Snowflake
<https://docs.snowflake.com/en/sql-reference/collation>,
Oracle
<https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/COLLATION.html>
- to
name just a few - this list is not complete.
In Snowflake, we see heavy use of the collation
feature. Several users have approached us,
mentioning they want to migrate to Iceberg tables,
but are currently blocked by Iceberg's lack of
collation support.
Given the widespread support for collations across
different engines, I believe introducing collations
to Iceberg will increase interoperability and boost
its adoption.
I'd be curious about your thoughts.
*Goal of the proposal*
- Support collation specifications for columns
- Define how collation bounds should be stored -
UTF-8 based bounds are not useful for collated columns
*Required Changes*
- Extend the schema to let (string) fields be
annotated with a collation
More details can be found in this doc
<https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.0#heading=h.y1ant4w2163k>.
I'm also hoping to present the idea in the next
community sync.
Best, Alex
