nssalian opened a new pull request, #1351:
URL: https://github.com/apache/iceberg-go/pull/1351
## Summary
Adds opt-in shredding of top-level variant columns on the write path.
Frequently-occurring
variant fields are written as typed Parquet `typed_value` columns (with the
`value` blob kept
as fallback) instead of fully-encoded binary, so engines can read and filter
them directly.
arrow-go performs the physical shredding; this PR adds the inference (which
fields and types to
shred) and the per-file writer wiring.
Disabled by default - no change to existing writes.
## Changes
- `AnalyzeVariantShredding` infers the inner Arrow type from a sample of
variant values,
mirroring Iceberg Java's `VariantShreddingAnalyzer`: most-common-type
selection with explicit
tie-break, integer/decimal widening, a 10% field-frequency floor, and
300-field / depth /
intermediate-field bounds. Inference is deterministic (same values produce
the same schema).
- The `RollingDataWriter` buffers the first N converted batches per file,
infers the shredding
schema, opens the file with it, replays the buffer, and re-infers on roll.
The logical
`FileSchema` is unchanged; only the physical Arrow schema carries the
shredded layout.
- Shredded decimals require the spec's INT32/INT64/FLBA-by-precision
physical types
(VariantShredding.md); arrow-go emits those only with
`StoreDecimalAsInteger`, enabled
alongside shredding so default writes are unaffected.
## Properties (both default to current behavior)
| Key | Default |
|---|---|
| `write.parquet.shred-variants` | `false` |
| `write.parquet.variant-inference-buffer-size` | `100` |
## Testing
- Unit: analyzer inference (widening, frequency floor, field cap,
nested/array, decimal/temporal/
uuid leaves), the shred-transform `(value, typed_value)` truth table, and
round-trips through
`UnshredVariant`.
- Write path: real writer to Parquet and back, including no-leak under a
checked allocator across
roll/re-bootstrap, row conservation, partitioned fanout, scalar
physical-type assertions, a
precision > 18 decimal, and a regular (non-variant) decimal column whose
manifest bounds stay
correct under the file-global `StoreDecimalAsInteger`.
- Cross-engine: Go writes a shredded file and Spark 4 reads it back and
reassembles the values
(integration-tagged). Locally ran both integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
