RussellSpitzer commented on code in PR #17537: URL: https://github.com/apache/iceberg/pull/17537#discussion_r3769131652
########## site/docs/blog/posts/2026-08-12-variant-in-apache-iceberg.md: ########## @@ -0,0 +1,141 @@ +--- +date: 2026-08-12 +title: "Semi-Structured Data in Apache Iceberg: Meet the Variant Type" +slug: variant-in-apache-iceberg +authors: + - nssalian +categories: + - blog +--- + +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +Semi-structured data, such as JSON-like documents whose fields differ from row to row, has always been a poor fit for table formats built around fixed schemas. Iceberg v3 adds the Variant type for exactly this data: a single column can hold values of arbitrary, evolving shape, stored in a compact binary form that engines read and write consistently. + +This is the first post in a series on Variant in Apache Iceberg. It covers what Variant is, why it exists, and how it fits into an Iceberg table. The next post covers shredding, the technique that stores frequently accessed Variant fields as typed, columnar data. Variant is stored in Parquet, Avro, and ORC; shredding is currently available only in Parquet. + +<!-- more --> + +## The problem Variant solves + +Consider event data whose shape changes over time: + +```json +{"event": "view", "page": "/pricing", "session": "s-8f21"} +{"event": "signup", "session": "s-8f21", "plan": "pro", "price": 29.00} +{"event": "view", "page": "/docs", "session": "s-3c07", "referrer": {"source": "search", "term": "iceberg variant"}} +``` + +Two traditional approaches handle this, and both have drawbacks: + +- **JSON stored as a string.** This is flexible, but reading a single field means parsing the whole text. JSON's type system is also thin: a timestamp is just a string, and a number's precision is ambiguous. +- **A rigid, flattened schema.** This is fast to query, but every new field is a schema migration, and sparse or one-off fields waste space. + +Variant is as flexible as JSON but stores data in a compact, typed binary form. Values keep their native types: a timestamp stays a timestamp and a decimal stays an exact decimal, instead of collapsing to JSON's strings and numbers. Within a value, field names are collected into a dictionary and referenced by id, so a name is not written out in full each time it appears. No schema is declared up front, so documents of different shapes coexist in one column and a new field needs no migration. + +## Variant in Apache Iceberg + +Variant was added to the Iceberg type system in the v3 spec. The spec places it in its own category: `variant` is "neither a primitive type nor a nested type." It is richer than a primitive, yet has no fixed, declared shape the way a struct or list does. + +A Variant value is similar to JSON, but with a wider set of primitives including `date`, `timestamp`, `timestamptz`, `binary`, and `decimal`. It can also nest: + +1. A **Variant array** is an ordered collection of Variant values. Unlike an Iceberg list, its elements are not constrained to a single element type. +2. A **Variant object** is a collection of string-keyed fields whose values are themselves Variant values. Unlike a struct column in an Iceberg schema, its fields are not a fixed, named set of typed columns. + +### How it is stored + +Iceberg does not define its own binary encoding for Variant. The type and its encoding come from the [Apache Parquet project](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md), and Iceberg uses that encoding as-is, so a `variant` column maps to a Parquet `group` with two binary fields: + +```parquet +optional group payload (VARIANT(1)) { + required binary metadata; + required binary value; +} +``` + +- `metadata` holds a dictionary of the field names used in the value, so the `value` bytes reference each name by an integer id instead of repeating the name string. Review Comment: I'm not sure I understand the "so" here. The reverse lookup of name within the value I don't think is super important. Maybe instead "so the `value` bytes can store just the offset of where each field_id begins without repeating the name."? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
