kevinjqliu commented on code in PR #17705:
URL: https://github.com/apache/iceberg/pull/17705#discussion_r3845612765
##########
site/docs/blog/posts/2026-08-12-variant-in-apache-iceberg.md:
##########
@@ -68,10 +72,32 @@ optional group payload (VARIANT(1)) {
}
```
-- `metadata` holds a dictionary of the field names used in the value, so the
`value` bytes reference each name by an integer id instead of repeating the
name string.
+- `metadata` holds the dictionary of field names used in the value, so those
names are not repeated inline with the data.
- `value` holds the encoded data: a scalar, an array, or an object. Arrays and
objects store a `field_offset` per element (the byte offset where that
element's value starts), and objects also store a `field_id` per field (an
index into the metadata dictionary).
-The Variant column itself is addressed by field ID like any other Iceberg
column, but its `metadata` and `value` subfields are accessed by name, which
matters for shredding.
+The Variant column itself is addressed by its Iceberg field ID like any other
column, but its `metadata` and `value` subfields are accessed by name, which
matters for shredding.
+
+The same Variant maps into every file format Iceberg supports: a Parquet
`group`, an Avro `record`, or an ORC `struct`, each holding the `metadata` and
`value` pair. In Avro and ORC, a Variant is always the single unshredded pair.
+
+### One column, many layouts
+
+A Variant's structure is not consistent across rows or files, but the column's
Iceberg type is always `variant`, whatever shapes flow through it. Adding or
removing a field inside the data changes only the bytes in each row.
Review Comment:
```suggestion
A Variant's structure is not consistent across rows or files, but the
column's Iceberg type is always `variant`, whatever shapes flow through it.
Adding or removing a field inside the data does not require an Iceberg schema
change; it only changes the Variant values and, potentially, their per-file
physical layout.
```
"only the bytes in each row" is slightly misleading with shredding
##########
site/docs/blog/posts/2026-08-12-variant-in-apache-iceberg.md:
##########
@@ -41,12 +43,12 @@ Consider event data whose shape changes over time:
{"event": "view", "page": "/docs", "session": "s-3c07", "referrer": {"source":
"search", "term": "iceberg variant"}}
```
-Two traditional approaches handle this, and both have drawbacks:
+Without Variant, an Iceberg table has two ways to store data like this, and
both have drawbacks:
- **JSON stored as a string.** This is flexible, but reading a single field
means parsing the whole text. JSON's type system is also thin: a timestamp is
just a string, and a number's precision is ambiguous.
- **A rigid, flattened schema.** This is fast to query, but every new field is
a schema migration, and sparse or one-off fields waste space.
-Variant is as flexible as JSON but stores data in a compact, typed binary
form. Values keep their native types: a timestamp stays a timestamp and a
decimal stays an exact decimal, instead of collapsing to JSON's strings and
numbers. Within a value, field names are collected into a dictionary and
referenced by id, so a name is not written out in full each time it appears. No
schema is declared up front, so documents of different shapes coexist in one
column and a new field needs no migration.
+Variant is as flexible as JSON but encodes data as compact, typed binary.
Values keep their native types: a timestamp stays a timestamp and a decimal
stays an exact decimal, instead of collapsing to JSON's strings and numbers. No
schema is declared up front, so documents of different shapes coexist in one
column and a new field needs no migration.
Review Comment:
```suggestion
Variant is as flexible as JSON but encodes data as compact, typed binary.
Values keep their native types: a timestamp stays a timestamp and a decimal
stays an exact decimal, instead of collapsing to JSON's strings and numbers.
The Iceberg schema declares the column as `variant`, but not the fields or
types inside it, so values can differ in structure from row to row and adding a
new field requires no table schema migration.
```
nit: distinguish iceberg schema vs variant schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
