This is an automated email from the ASF dual-hosted git repository.
laskoviymishka pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-go.git
The following commit(s) were added to refs/heads/main by this push:
new 21e85b611 docs: document variant type, field-extract filtering, and
shredding (#2045)
21e85b611 is described below
commit 21e85b611b13625a340cfe29f0344fbd1cc0d54f
Author: Neelesh Salian <[email protected]>
AuthorDate: Wed Sep 23 12:29:28 2026 -0700
docs: document variant type, field-extract filtering, and shredding (#2045)
* docs: document variant type, field-extract filtering, and shredding
* PR comment fixes and cleanup
---
website/src/SUMMARY.md | 1 +
website/src/api.md | 2 +-
website/src/expression-dsl.md | 2 +
website/src/feature-status.md | 3 +-
website/src/row-filter-syntax.md | 20 +++++++
website/src/variant.md | 110 +++++++++++++++++++++++++++++++++++++++
6 files changed, 136 insertions(+), 2 deletions(-)
diff --git a/website/src/SUMMARY.md b/website/src/SUMMARY.md
index 01d9ce92f..3506e9aa9 100644
--- a/website/src/SUMMARY.md
+++ b/website/src/SUMMARY.md
@@ -26,6 +26,7 @@
- [Configuration](./configuration.md)
- [CLI](./cli.md)
- [API](./api.md)
+ - [Variant Type](./variant.md)
- [Row Filter Syntax](./row-filter-syntax.md)
- [Expression DSL](./expression-dsl.md)
- [Concurrent Writes](./concurrent-writes.md)
diff --git a/website/src/api.md b/website/src/api.md
index 127517c9b..83747beb6 100644
--- a/website/src/api.md
+++ b/website/src/api.md
@@ -589,7 +589,7 @@ When iceberg-go converts an Iceberg schema to Arrow (e.g.
for the scanner output
| `struct<...>` | `arrow.StructOf(...)` |
| `list<E>` | `arrow.ListOf(E)` (or `LargeListOf` if `useLargeTypes`) |
| `map<K, V>` | `arrow.MapOf(K, V)` |
-| `variant` | `arrow.ExtensionType` for Variant |
+| `variant` | `arrow.ExtensionType` for Variant (see [Variant
Type](./variant.md)) |
Helpers in
[`table/arrow_utils.go`](https://github.com/apache/iceberg-go/blob/main/table/arrow_utils.go):
diff --git a/website/src/expression-dsl.md b/website/src/expression-dsl.md
index e5f50c066..f482b66d1 100644
--- a/website/src/expression-dsl.md
+++ b/website/src/expression-dsl.md
@@ -56,6 +56,7 @@ A *term* is the left-hand side of a predicate. Iceberg-go has
two flavors:
- `Reference("column_name")` - an unbound term that names a column
(`exprs.go:373`). Typing happens at bind time, when the expression is matched
against a schema. This is what you almost always want.
- `BoundReference` - the resolved form, produced by `Reference.Bind(schema,
caseSensitive)` (`exprs.go:389`). You only encounter these when writing custom
expression visitors.
+- `Extract(ref, path, typ)` - a variant sub-path term that navigates a
member-selector path (`$.a.b` or `$['a']['b']`; no array indexing or wildcards)
into a `variant` column and casts the leaf to `typ` (`variant_extract.go`). Use
it inside any predicate builder to filter on a field within a variant. See [Row
Filter Syntax](./row-filter-syntax.md#variant-extraction).
The interfaces are:
@@ -126,6 +127,7 @@ For projection, evaluation, and visitor patterns, see
`visitors.go` and the scan
| Filter a scan or row-level delete | Convenience builders +
`NewAnd`/`NewOr`/`NewNot` |
| Combine many clauses dynamically | Start from `AlwaysTrue{}` (for AND) or
`AlwaysFalse{}` (for OR), fold with `NewAnd`/`NewOr` |
| Construct a predicate whose operator is chosen at runtime |
`UnaryPredicate(op, term)`, `LiteralPredicate(op, term, lit)`,
`SetPredicate(op, term, lits)` |
+| Filter on a field inside a variant column | `Extract(ref, path, typ)` inside
a predicate builder |
| Walk an expression tree | A custom `BooleanExprVisitor` from `visitors.go` |
For the per-operator cookbook, return to [Row Filter
Syntax](./row-filter-syntax.md).
diff --git a/website/src/feature-status.md b/website/src/feature-status.md
index 87d81986f..800932cd6 100644
--- a/website/src/feature-status.md
+++ b/website/src/feature-status.md
@@ -50,7 +50,8 @@ All V1 features are supported. V1 is the format-version
baseline.
| Row lineage (`_row_id`, `_last_updated_sequence_number`) | Supported |
| Encryption keys in metadata | Supported |
| Variant type, non-shredded | Supported (PR
[#932](https://github.com/apache/iceberg-go/pull/932); umbrella
[#929](https://github.com/apache/iceberg-go/issues/929)) |
-| Variant type, shredded reader / writer | In progress
([#986](https://github.com/apache/iceberg-go/issues/986),
[#987](https://github.com/apache/iceberg-go/issues/987)) |
+| Variant type, shredded reader / writer | Supported (reader
[#1238](https://github.com/apache/iceberg-go/pull/1238), writer
[#1351](https://github.com/apache/iceberg-go/pull/1351)) |
+| Variant predicate pushdown / field extract | Supported (PR
[#1607](https://github.com/apache/iceberg-go/pull/1607)) |
| Deletion vectors, read | Supported |
| Deletion vectors, write (unpartitioned) | Supported |
| Deletion vectors, write (partitioned) | In progress
([#1135](https://github.com/apache/iceberg-go/issues/1135), PR
[#1151](https://github.com/apache/iceberg-go/pull/1151)) |
diff --git a/website/src/row-filter-syntax.md b/website/src/row-filter-syntax.md
index ccb2490f0..83857ce98 100644
--- a/website/src/row-filter-syntax.md
+++ b/website/src/row-filter-syntax.md
@@ -88,6 +88,26 @@ iceberg.NotStartsWith(iceberg.Reference("name"), "tmp_")
Operators `OpStartsWith` and `OpNotStartsWith` (`exprs.go:53-54`). The value
must be a `string`.
+## Variant extraction
+
+`iceberg.Extract(ref, path, typ)` builds a term that navigates a
member-selector path into a `variant` column and casts the leaf to `typ`. The
path uses dot (`$.user.age`) or quoted-bracket (`$['user']['age']`) notation;
array indexing (`$.items[0]`), wildcards, filters, and recursive descent are
rejected at bind time. It is an `UnboundTerm`, so it plugs into the same
predicate builders as `iceberg.Reference`:
+
+```go
+filter := iceberg.EqualTo(
+ iceberg.Extract(iceberg.Reference("payload"), "$.user.age",
iceberg.PrimitiveTypes.Int64),
+ int64(30),
+)
+```
+
+The referenced column must be a `variant`, and a target type is required.
Supported target types: `bool`, `int32`, `int64`, `float32`, `float64`, `date`,
`time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`,
`fixed`, `binary`, `decimal`, `uuid`.
+
+Caveats:
+
+- Extract predicates prune whole data files by shredded-field bounds
(best-effort, and only for shredded fields), then filter row by row. Row-group
statistics and bloom filters do not apply to them.
+- Extract terms are DSL-only (no string row-filter form) and are not
REST-serializable. Under local scan planning (the default) they run as local
residuals; with remote scan planning, a filter containing an Extract term fails
to serialize rather than being silently dropped.
+
+See [Variant Type](./variant.md) for shredding.
+
## Constants
```go
diff --git a/website/src/variant.md b/website/src/variant.md
new file mode 100644
index 000000000..56cf52830
--- /dev/null
+++ b/website/src/variant.md
@@ -0,0 +1,110 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# Variant Type
+
+`variant` is the Iceberg v3 semi-structured type: a single self-describing
column
+that holds arbitrary JSON-like values (objects, arrays, and scalars) without a
+fixed schema. iceberg-go supports both non-shredded and shredded variants, and
can
+push predicates down into individual variant fields.
+
+## Reading and writing
+
+A variant column is declared with `iceberg.VariantType{}` in the table schema
and
+maps to the Arrow `variant` extension type (see [API](./api.md)). Non-shredded
+variant values need no extra configuration: write an Arrow record whose variant
+column uses the extension type, and read it back the same way.
+
+Build the column with a variant builder and write it like any other Arrow
column
+(the table must be format version 3):
+
+```go
+import (
+ "fmt"
+
+ "github.com/apache/arrow-go/v18/arrow"
+ "github.com/apache/arrow-go/v18/arrow/array"
+ "github.com/apache/arrow-go/v18/arrow/extensions"
+ "github.com/apache/arrow-go/v18/arrow/memory"
+ "github.com/apache/arrow-go/v18/parquet/variant"
+ "github.com/apache/iceberg-go"
+ "github.com/apache/iceberg-go/table"
+)
+
+// tbl was created with table.PropertyFormatVersion = "3" and this schema.
+schema := iceberg.NewSchema(0,
+ iceberg.NestedField{ID: 1, Name: "payload", Type: iceberg.VariantType{}},
+)
+arrowSchema, _ := table.SchemaToArrowSchema(schema, nil, true, false)
+
+bldr := extensions.NewVariantBuilder(memory.DefaultAllocator,
extensions.NewDefaultVariantType())
+defer bldr.Release()
+
+var vb variant.Builder
+_ = vb.Append(map[string]any{"x": int64(320), "target": "button-submit"})
+val, _ := vb.Build()
+bldr.Append(val)
+
+col := bldr.NewArray()
+defer col.Release()
+rec := array.NewRecordBatch(arrowSchema, []arrow.Array{col}, 1)
+defer rec.Release()
+
+tx := tbl.NewTransaction()
+arrTable := array.NewTableFromRecords(arrowSchema, []arrow.Record{rec})
+defer arrTable.Release()
+_ = tx.AppendTable(ctx, arrTable, 1024, nil)
+tbl, _ = tx.Commit(ctx)
+```
+
+A normal scan reads it back; the variant column comes back as an
`*extensions.VariantArray`:
+
+```go
+result, _ := tbl.Scan().ToArrowTable(ctx)
+defer result.Release()
+
+variants := result.Column(0).Data().Chunk(0).(*extensions.VariantArray)
+v, _ := variants.Value(0)
+fmt.Println(v.Value()) // variant.ObjectValue for the object above
+```
+
+## Shredding
+
+Shredding stores the fields of a variant as typed Parquet sub-columns instead
of a
+single opaque blob. This lets scans read and filter a field as a native column
+rather than decoding every value. Shredding is opt-in and configured with two
write
+properties (see [Configuration](./configuration.md#parquet-writer)):
+
+- `write.parquet.shred-variants` - enable shredding of top-level variant
columns
+ (default `false`).
+- `write.parquet.variant-inference-buffer-size` - rows buffered per file to
infer
+ the shredding schema (default `100`).
+
+The reader accepts both shredded and non-shredded data; no read-side
+configuration is required.
+
+## Filtering on variant fields
+
+Filter on a field inside a variant with `iceberg.Extract`, which plugs into the
+usual predicate builders. See [Row Filter
Syntax](./row-filter-syntax.md#variant-extraction)
+for usage, supported target types, and caveats.
+
+## Constraints
+
+- A variant column cannot be a partition source or an identity-transform
source.
