Hi Alex,
Thanks, these are the right questions. Let me answer
them, but I think all three are really facets of one
decision worth pulling out, so I'll do that at the end.
Original values vs sort keys. I don't think the two
limitations are symmetric. You're right that a bound
stored under version X may not hold under Y for either
representation, and a naive reader prunes only on an
exact version match either way. But they degrade
differently: a sort key from X is incomparable under Y
(nothing a Y reader can do with it) while an original
value is the actual string, so a Y reader can
re-interpret it. In the case you describe at the end of
your mail, where only a small code-point range moved and
a file's values fall outside it, that reader can prove
the X bound still holds and prune across versions. Sort
keys foreclose that; original values keep it open. So
original values are a superset: worst case they match
sort keys, best case they prune across versions.
Your two sort-key advantages are real, I just don't
think they belong in the format. Truncation: agreed it's
hard for collated strings (your abcเก contraction case
is exactly the trap), so I sidestepped it, collation
bounds must be tight, a writer that can't store the
exact min/max omits the bound. Collation-aware
truncation with CollationElementIterator is a possible
later optimization. Compare cost: pruning is per-file at
planning time, not per-row, so collator vs byte compare
is in the noise; and an engine that wants the byte path
can derive and cache the sort key from the stored value.
Original values don't block that, they just don't bake a
version-specific encoding into the format.
Column vs file-level version. As you say, every engine
can read regardless, so this is a pruning-performance
choice, not correctness. In the
schema-registered-metrics design a file carries bounds
under a declared (collation, version), and a reader
prunes any file with a metric for a version it can
produce, not only files it wrote. So convergence on one
ICU version gives full cross-engine pruning, same as
column-level, and a writer or compaction can populate
several versions at once. It gives the column-level
benefit by convention without making a version bump a
format-breaking change.
[One data point from the engine side, since I'm coming
at this from the Databricks runtime: our runtime is
effectively versionless (customers don't pin an ICU
version, and upgrades happen under them) so "the same
table read by clients on different ICU versions" isn't a
corner case for us, it's the default. That's what pushes
me toward per-file versioning: pinning one version per
table or column means either forcing the whole fleet to
upgrade in lockstep or breaking pruning on every bump,
and neither survives a versionless fleet. And in
practice most version bumps we've gone through don't
reorder the data in a given column at all, which is
exactly why keeping original values, and eventually your
code-point-range idea, lets a reader keep pruning across
a bump instead of falling back to a scan.]
Providers. Agreed. I'll tighten the spec to a registered
set like geo, icu to start, utf8 reserved, non-ICU
collations added by spec change rather than ad hoc.
Interop is the whole point and an open namespace
undercuts it.
-----------
Stepping back: I think the three above collapse into one
question that needs broader alignment than the two of us:
> how much cross-engine pruning interoperability should
the format guarantee, versus leave to convention?
Original-vs-sortkey, column-vs-file version, and
open-vs-restricted providers are all that same tradeoff
from different angles. It's a values call more than a
correctness one, and it binds every engine that hasn't
weighed in yet: Trino, Flink, Spark, PyIceberg, rust.
From the DBR side I can say the multi-version case is
real rather than theoretical, but that's one engine's
vantage point. I'd like to get the interop question in
front of the other implementers before we fix field IDs,
the dev list is probably enough for now, and a community
sync is there if it needs more than async. No rush on
that; I'd rather let the thread settle the mechanics first.
Best,
Andrei
On Fri, Jul 3, 2026 at 12:22 AM Alexander Löser
<[email protected]> wrote:
Hi Andrei,
Thanks for putting together the spec PR and the
detailed write-up! The approach mostly looks solid
to me. I have a few questions/initial thoughts
regarding the changes you proposed (compared to the
original proposal):
> 1 - Bounds store original values, not sort keys,
tagged with a per-file collation version. ICU/CLDR
sort keys aren't stable across versions, so storing
keys ties every reader to one exact version;
original values plus a per-file version (readers
prune only on an exact match) degrade gracefully
instead of breaking. The schema keeps the collation
name unversioned so anyone can read
If I understand correctly, we’re talking about two
separate things here:
1. Tagging a column vs a single file with a certain
ICU version
2. Using collation keys vs original strings (the
ones that will produce the min/max collation keys)
For 1, it comes down to a tradeoff:
* If we tag the column with the ICU version, we’d
force engines to support one agreed-on ICU
version if they want to prune files. Engines
would be able to prune every file (if they
support the specific ICU version), or none at
all, so there is more incentive to support a
specific version
* If we tag individual files with ICU versions, we
gain the big advantage that engines do not need
to agree on a single ICU version. However, if I
understand correctly, this comes at the cost of
“fractured” pruning - engines will only be able
to prune files that were written by themselves
(or rather, with the same ICU version). As a
consequence, performance might not really be
interoperable between different engines.
Regardless of the approach we choose, all engines
should be able to read the data - they might just
not be able to prune files.
For 2, I’m not sure if I understand the advantages
of original strings yet. As you already pointed out,
the collation keys depend on the ICU version.
However, if I understand correctly, the same
limitation would apply to the original strings: the
sort order may (and does) change between different
ICU versions, too. As a consequence, we can’t assume
the original lower/upper bound strings we stored for
version X will also be lower/upper bounds for
version Y - at least in the general case. So if I
understand correctly, we would not gain additional
pruning opportunities compared to using collation
keys. Or am I missing something here?
At the same time, sort keys do have advantages:
* Iceberg allows the truncation of upper- and
lower bounds. This is trivial for binary
collation keys. For original strings, the task
becomes significantly harder: truncating at a
character boundary, for example, would lead to
wrong results, as there are some
context-sensitive sequences: e.g., with the CLDR
root locale, abcเก < abcเ. I think it might be
doable with ICU’s CollationElementIterator, but
it will be tricky to get right.
* Lower/upper bounds are computed once, but will
be compared many times. With original strings,
we would need to either convert to the collation
key on the fly, or use ICU’s collator for a
direct comparison. Both options will be slower
than a raw byte-sequence comparison
There is one scenario where original values would
shine, though. I analyzed the order-changes between
various ICU versions: in many cases, only a small
range of code points changes/is moved. If we had
additional metadata about which code point ranges a
file contains (e.g., whether it is ASCII only),
engines might be able to prove that the original
string bounds for version X are still valid for
version Y.
If I'm not mistaken, this could allow to prune
across different ICU versions in certain situations,
which I’d consider a point in favor of original
values (and file-level ICU versions).
> 2 - A provider-qualified identifier
(icu.en_US-ci), leaving room for non-ICU collations
like Spark's UTF8_LCASE, rather than assuming ICU as
the sole provider.
Adding a provider-mechanism sounds like a good
approach to keep the spec open for future collations
:slightly_smiling_face: I wonder whether we should
restrict the set of allowed providers, though,
similar to how it was done with geo
<https://lists.apache.org/thread/r5x0do8f241bpf565rx8s5s3wc9ogp0f>.
My main motivation for this proposal is
interoperability. I worry that interoperability
might suffer or vanish completely if every engine
can come up with their own definitions.
Happy to hear your thoughts on this!
Best, Alex
On 6/27/26 01:37, Szehon Ho wrote:
Very nice direction, left some comments on the spec
proposal.
Thanks to you folks for working on it !
Szehon
On Fri, Jun 26, 2026 at 3:29 AM Andrei Tserakhau
via dev <[email protected]> wrote:
Hi all,
I've spend some cycle on the collation
discussion and make something more concrete to
react to: a spec-change PR plus reference
implementations (go and java).
- Spec change (apache/iceberg#16972): a
"collation" annotation on string fields, and a
data_file.collation_bounds field so collated
columns stay prunable.
- Reference implementation in iceberg-go
(apache/iceberg-go#1318): the full path end to
end - schema annotation, collation-aware
comparison (CLDR/UCA), collation bounds in the
manifest, and version-gated data-file pruning,
with an Avro round-trip and pruning tests.
- A lightweight Java POC (link below): the
schema annotation plus a Collator-backed
comparator, to match where the discussion is. I
deliberately left the manifest/bounds side out
of Java for now.
The design follows the original proposal but
takes a few different turns, mostly to adopt
what we learned in Delta. The ones I'd most
like input on:
1 - Bounds store original values, not sort
keys, tagged with a per-file collation version.
ICU/CLDR sort keys aren't stable across
versions, so storing keys ties every reader to
one exact version; original values plus a
per-file version (readers prune only on an
exact match) degrade gracefully instead of
breaking. The schema keeps the collation name
unversioned so anyone can read.
2 - A provider-qualified identifier
(icu.en_US-ci), leaving room for non-ICU
collations like Spark's UTF8_LCASE, rather than
assuming ICU as the sole provider.
3 - One structural question I don't have a
strong opinion on yet: I put collation_bounds
on data_file as a standalone v3 field, but
field id 146 is already the v4 content_stats
struct, and collation bounds might belong
inside that typed-stats framework instead.
Worth settling before we fix field ids.
The full set of differences and the
reader/writer rules are in the PR description
and the write-up. Comments very welcome — both
on the calls above and on whether the
standalone-field vs content_stats direction is
the right one.
Best, Andrei
- original proposal:
https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.0
- spec change:
https://github.com/apache/iceberg/pull/16972
- POC in go:
https://github.com/apache/iceberg-go/pull/1318
- java POC:
https://github.com/laskoviymishka/iceberg/tree/prototype/collation-support
On Mon, Mar 30, 2026 at 10:54 PM Alexander
Löser <[email protected]> wrote:
Hi Andrei,
I'm glad you're interested. Looking forward
to collaborate with you!
Thanks for all the feedback here and in the
doc. I only had a quick glance, but I think
you raised some good points. I'll
address/respond to your comments as soon
as I get the chance, hopefully tomorrow.
I think you also left some comments in this
mail that are not yet in the doc - I'll
move those to a dedicated section at the
end of the doc, so we can use the doc as a
single source of truth/discussion.
> Happy to share our Delta design doc and
implementation learnings in more detail.
Sure, sounds good :)
Best,
Alex
On 3/29/26 01:25, Andrei Tserakhau via dev
wrote:
Hi Alexander,
This looks really interesting. We've been
working on collation support in Delta and
have shipped it in production for some
time, so this is an area we care about a
lot. If this proposal moves forward we'd
be happy to collaborate on the design and
implementation.
The pseudo-field approach for collation
metrics is clean and composes well with
existing Iceberg infrastructure. The
specifier coverage is comprehensive.
A few areas worth discussing as this evolves:
1 - Sort key stability and versioning
ICU sort keys are not stable across
versions, so a pinned ICU version bump in
a future Iceberg release would invalidate
all existing collation metrics. In
multi-engine environments, requiring all
engines to converge on one ICU version is
unrealistic.
We store original string values instead of
sort keys and allow per-file version
annotations -- worth discussing whether
something similar could work here.
2 - Provider abstraction
The proposal assumes ICU as the sole
provider, but Spark ships non-ICU
collations like UTF8_LCASE that are widely
used. A provider or namespace layer would
prevent name collisions and support
engine-specific collations without future
spec changes.
3 - Operational surface
A few things that turned out
correctness-critical in our
implementation: partition transforms on
collated columns (collation-equal but
byte-distinct values in different
directories), sort order semantics,
equality deletes under collation, and
Parquet filter pushdown (must be disabled
since Parquet has no collation concept).
These don't all need to be solved in v1
but would help to scope them.
4 - Smaller items (nit's)
UTF-8 bounds for the original field id
should be "must write" not "should" --
otherwise backward compat breaks for
non-aware engines. Engine fallback
behavior (case-sensitive vs older ICU vs
fail) could use a recommended preference
order to avoid divergent results across
engines. The collation specifier syntax
would benefit from a formal grammar.
---
Happy to share our Delta design doc and
implementation learnings in more detail.
Looking forward to the discussion.
Best,
Andrei
On Sat, Mar 28, 2026 at 11:49 PM Alexander
Löser <[email protected]> wrote:
Hi everyone,
this is my first interaction with the
Iceberg community, so here a few words
about myself:
- I'm Alex, a Berlin-based software
engineer
- I've been working at Snowflake for 4
years now
- I spend most of my time on data
types, particularly binary, strings
and collations.
I'd like to start a discussion about
adding collations to the Iceberg spec.
Conceptually, collations are an
annotation on the string data type. By
default, most engines perform string
operations case-sensitively.
Collations allow specifying
alternative comparison rules. This is
useful for achieving, e.g., case- or
accent-insensitive string operations,
or language-specific string sorting.
Collations are supported by many
engines: Databricks
<https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-collation>,
Spark
<https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.collate.html>,
Snowflake
<https://docs.snowflake.com/en/sql-reference/collation>,
Oracle
<https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/COLLATION.html>
- to
name just a few - this list is not
complete.
In Snowflake, we see heavy use of the
collation feature. Several users have
approached us, mentioning they want to
migrate to Iceberg tables, but are
currently blocked by Iceberg's lack of
collation support.
Given the widespread support for
collations across different engines, I
believe introducing collations to
Iceberg will increase interoperability
and boost its adoption.
I'd be curious about your thoughts.
*Goal of the proposal*
- Support collation specifications for
columns
- Define how collation bounds should
be stored - UTF-8 based bounds are not
useful for collated columns
*Required Changes*
- Extend the schema to let (string)
fields be annotated with a collation
More details can be found in this doc
<https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.0#heading=h.y1ant4w2163k>.
I'm also hoping to present the idea in
the next community sync.
Best, Alex
