This is an automated email from the ASF dual-hosted git repository.
kevinjqliu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new 44bf484e89 Spec: Clarify variant type classification and primitive
type scoping (#16836)
44bf484e89 is described below
commit 44bf484e895f21d385bbf0c32363e2fd6b0f0db9
Author: Junwang Zhao <[email protected]>
AuthorDate: Sun Aug 2 00:05:19 2026 +0800
Spec: Clarify variant type classification and primitive type scoping
(#16836)
This PR clarifies how the Iceberg specification classifies and references
the `variant` type.
- Clarifies that `variant` is a semi-structured type and is neither
primitive nor nested.
- Uses primitive-type scoping consistently when describing supported source
types for identity and bucket transforms.
- Applies the same terminology to the availability of `tight_bounds`.
The previous wording explicitly excluded `variant` from constructs already
scoped to primitive types, which could create ambiguity about how `variant` is
classified. This is a specification clarification only and does not change
format semantics or supported behavior.
---
format/expressions-spec.md | 4 ++--
format/spec.md | 8 +++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/format/expressions-spec.md b/format/expressions-spec.md
index 01aa7d8b35..2975648a25 100644
--- a/format/expressions-spec.md
+++ b/format/expressions-spec.md
@@ -223,7 +223,7 @@ All partition transforms produce `null` for a `null` input
value.
| Function name | Description
| Source types |
Result type |
|-------------------|--------------------------------------------------------------|----------------------------------------------------------------------|-------------|
-| `identity(value)` | Source value, unmodified
| Any primitive except for `geometry`, `geography`, and `variant` |
Source type |
+| `identity(value)` | Source value, unmodified
| Any primitive except for `geometry` and `geography` | Source type |
| `year(value)` | Extract a date or timestamp year, as years from 1970
| `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` |
`int` |
| `month(value)` | Extract a date or timestamp month, as months from
1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`,
`timestamptz_ns` | `int` |
| `day(value)` | Extract a date or timestamp day, as days from 1970-01-01
| `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` |
`date` |
@@ -235,7 +235,7 @@ Note that `year`, `month`, and `hour` transforms produce
ordinal values and not
| Parameterized function name | Description
| Source types
| Result type |
|-----------------------------|-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------|-------------|
-| `bucket(N, value)` | Hash of value, mod `N` (see [table spec
details][bucket-ref]) | Any primitive except for `geometry`,
`geography`, `variant`, `boolean`, `float`, or `double` | `int` |
+| `bucket(N, value)` | Hash of value, mod `N` (see [table spec
details][bucket-ref]) | Any primitive except for `geometry`,
`geography`, `boolean`, `float`, or `double` | `int` |
| `truncate(W, value)` | Value truncated to width `W` (see [table spec
details][truncate-ref]) | `int`, `long`, `decimal`, `string`, `binary`
| Source type |
[bucket-ref]: spec/#bucket-transform-details
diff --git a/format/spec.md b/format/spec.md
index ebb0125989..42be123524 100644
--- a/format/spec.md
+++ b/format/spec.md
@@ -228,7 +228,7 @@ When the `location` field is present in table metadata, it
is used directly as t
### Schemas and Data Types
-A table's **schema** is a list of named columns. All data types are either
primitives or nested types, which are maps, lists, or structs. A table schema
is also a struct type.
+A table's **schema** is a list of named columns. Data types are primitive,
nested, or semi-structured. Nested types are maps, lists, or structs. A table
schema is also a struct type.
For the representations of these types in Avro, ORC, and Parquet file formats,
see Appendix A.
@@ -244,6 +244,8 @@ A **`map`** is a collection of key-value pairs with a key
type and a value type.
A **`variant`** is a value that stores semi-structured data. The structure and
data types in a variant are not necessarily consistent across rows in a table
or data file. The variant type and binary encoding are defined in the [Parquet
project](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md),
with support currently available for V1. Support for Variant is added in
Iceberg v3.
+As a semi-structured type, `variant` is neither a primitive type nor a nested
type.
+
Variants are similar to JSON with a wider set of primitive values including
date, timestamp, timestamptz, binary, and decimals.
Variant values may contain nested types:
@@ -567,7 +569,7 @@ Partition field IDs must be reused if an existing partition
spec contains an equ
| Transform name | Description
| Source types
| Result type |
|-------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------|
-| **`identity`** | Source value, unmodified
| Any except for `geometry`, `geography`, and `variant`
| Source type |
+| **`identity`** | Source value, unmodified
| Any primitive except for `geometry` and `geography` | Source type |
| **`bucket[N]`** | Hash of value, mod `N` (see below)
| `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`,
`timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int`
|
| **`truncate[W]`** | Value truncated to width `W` (see below)
| `int`, `long`, `decimal`, `string`, `binary`
| Source type |
| **`year`** | Extract a date or timestamp year, as years from 1970
| `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`
| `int` |
@@ -818,7 +820,7 @@ Each stats struct holds statistics for one table field. It
may contain the follo
|-------------|--------|---------------------------|---------------------------|-----------------------------------------------|-------------|
| _optional_ | 1 | `lower_bound` | Field type or `geo_lower`
| all primitives or `variant` | Lower bound stored as the
field's type, or `geo_lower` for geo types |
| _optional_ | 2 | `upper_bound` | Field type or `geo_upper`
| all primitives or `variant` | Upper bound stored as the
field's type, or `geo_upper` for geo types |
-| _optional_ | 3 | `tight_bounds` | `boolean`
| all except `geometry`, `geography`, `variant` | When true, `lower_bound` and
`upper_bound` must be equal to the min and max values |
+| _optional_ | 3 | `tight_bounds` | `boolean`
| all primitives except for `geometry` and `geography` | When true,
`lower_bound` and `upper_bound` must be equal to the min and max values |
| _optional_ | 4 | `value_count` | `long`
| all | Number of values in the
column (including null and NaN values) |
| _optional_ | 5 | `null_value_count` | `long`
| optional fields | Number of null values in the
column |
| _optional_ | 6 | `nan_value_count` | `long`
| `float`, `double` | Number of NaN values in the
column |
