nssalian commented on code in PR #17705:
URL: https://github.com/apache/iceberg/pull/17705#discussion_r3806409033
##########
site/docs/blog/posts/2026-08-12-variant-in-apache-iceberg.md:
##########
@@ -68,10 +72,32 @@ optional group payload (VARIANT(1)) {
}
```
-- `metadata` holds a dictionary of the field names used in the value, so the
`value` bytes reference each name by an integer id instead of repeating the
name string.
+- `metadata` holds the dictionary of field names used in the value, so those
names are not repeated inline with the data.
- `value` holds the encoded data: a scalar, an array, or an object. Arrays and
objects store a `field_offset` per element (the byte offset where that
element's value starts), and objects also store a `field_id` per field (an
index into the metadata dictionary).
-The Variant column itself is addressed by field ID like any other Iceberg
column, but its `metadata` and `value` subfields are accessed by name, which
matters for shredding.
+The Variant column itself is addressed by its Iceberg field ID like any other
column, but its `metadata` and `value` subfields are accessed by name, which
matters for shredding.
+
+The same Variant maps into every file format Iceberg supports: a Parquet
`group`, an Avro `record`, or an ORC `struct`, each holding the `metadata` and
`value` pair. In Avro and ORC, a Variant is always the single unshredded pair.
+
+### One column, many layouts
+
+A Variant's structure is not consistent across rows or files, but the column's
Iceberg type is always `variant`, whatever shapes flow through it. Adding or
removing a field inside the data changes only the bytes in each row.
+
+That one logical column can be laid out differently in each data file. In
Parquet, one file may store it unshredded as the `metadata` + `value` pair
while another shreds its hot fields into dedicated typed columns. Both files
carry the same Variant field ID, and a reader reconciles whichever layout it
finds:
+
+```text
+payload (one Variant column, one field ID)
+├─ data file A, unshredded: metadata + value
+└─ data file B, shredded: metadata + value + typed_value.event,
typed_value.country
+```
+
+Snapshots do not change this. Each snapshot records the schema that was
current when it was written, and because the Variant column keeps its field ID
across schema changes, time travel reads every file back through the same
column.
+
+### Statistics and data skipping
+
+Because Variant is a column in the Iceberg schema, the table's manifests carry
statistics for it, and that is what lets Iceberg skip files during planning.
Iceberg records value and null counts for a Variant column. When a field is
shredded into its own typed column, Iceberg also records lower and upper bounds
for it, stored as a Variant object whose keys are normalized JSON paths to each
field. An unshredded `value` blob is opaque, so it contributes counts but no
bounds.
Review Comment:
You're right, the "Bounds for Variant" mechanism is general and optional,
not shredded-only. Reworded the paragraph accordingly: per-field bounds are
optional, and the shredding framing now describes the current writer rather
than the spec.
For more context: in Iceberg Java, ParquetMetrics derives bounds only from
shredded typed_value columns; the unshredded value returns counts only, since
bounds there would mean reading the raw value bytes. So, the spec permits
unshredded bounds, the writer just records counts today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
