nssalian opened a new pull request, #2002:
URL: https://github.com/apache/iceberg-go/pull/2002
## Rationale for the change
Bumps arrow-go to v18.8, uses its new `compute.VariantGet` for variant
extract with a zero-copy fast path for shredded fields, and re-enables the
Parquet dictionary cost-fallback that v18.8 disabled for compressed columns.
## Changes
### Arrow-go v18.8
1. Bump `github.com/apache/arrow-go/v18` to `v18.8.0`.
2. Accept both the canonical `arrow.parquet.variant` and legacy
`parquet.variant` extension names (`arrow_utils.go`,
`internal/parquet_files.go`).
3. Force `WithDictionaryCostFallbackFor` on every leaf so zstd columns drop
a dictionary that saves nothing (parquet-mr `shouldFallBack` parity; v18.8
disabled this for compressed columns) (`internal/parquet_files.go`).
### Variant extract
4. Residual extract navigates the path via `compute.VariantGet`
(apache/arrow-go#1206), with the per-row `CastVariantLiteral` walk as fallback;
adds `BoundExtract.VariantPath()` (`variant_residual.go`, `variant_extract.go`).
5. `tryShreddedTypedColumn`: when a field is shredded to exactly the target
type (no field-level residual), return its `typed_value` column directly -
Iceberg's cast is then an identity (mirrors Java
`VariantExpressionUtil.castTo`). Null rows and absent-object ancestors fold
into the validity mask (zero-copy when none, one bitmap-AND per level
otherwise); promotions, field-level residual, and unshredded data fall back
(`variant_residual.go`).
## Performance
`BenchmarkVariantExtract` measures extract-column materialization on a
131,072-row batch (the default read batch size), Apple M4 Max, go1.27.
"Production" is the current code (the fast path when the field is shredded to
the queried type, otherwise the `compute.VariantGet` fallback); "per-row" is
the previous walk.
| int64 field, 131,072 rows | production | per-row (previous) |
|---|--:|--:|
| clean, fully shredded (fast path) | 47 ns, 1 alloc | 66.9 ms, 2,358,045
allocs |
| 1% null rows (fast path) | 5.0 us, 10 allocs | 66.4 ms, 2,334,461 allocs |
| nested `$.a.b` (fast path) | 78 ns, 2 allocs | 102 ms, 2,751,261 allocs |
| 1% off-type rows, field-level residual (fallback) | 32.3 ms | 68.7 ms |
| unshredded (routed to per-row) | 33.6 ms | 33.5 ms |
String targets show the same pattern.
- When the field is shredded to the queried type - the common case after
#1846 type uniformity, including nullable columns and nested paths - extraction
returns the shredded `typed_value` column directly. This is O(1) in the row
count and effectively allocation-free: 47 ns and 1 allocation on a clean batch,
versus ~67 ms and 2.36M allocations for the per-row walk. Nullable columns add
a single bitmap merge (~5 us); nested paths ~78 ns.
- When the field is not cleanly shredded, extraction falls back.
Partly-shredded data (field-level residual) uses the columnar
`compute.VariantGet`, ~2x faster than the per-row walk (32.3 ms vs 68.7 ms).
Unshredded columns route directly to the per-row walk, matching it (33.6 ms vs
33.5 ms) rather than paying `VariantGet`.
- These figures cover the extract-column materialization step only, not a
full scan, which additionally includes predicate evaluation, IO, and
decompression.
## Testing
- `TestExtractFastPathParity`: fast-path output byte-identical to the
per-row reference across exact/nested/absent/promotion/residual/null shapes;
asserts it is wired in and zero-copy (mutation-checked) under a checked
allocator.
- `TestShreddedVariantExtractResidualNoLeak`: extract through the real
residual filter under a checked allocator (leak check).
- `TestGetWritePropertiesEnablesDictCostFallback`: cost fallback enabled per
leaf.
- `TestVariantExtractScanEndToEnd` /
`TestVariantExtractResidualAndHeterogeneous`: fast path and fallback via the
public `Scan().WithRowFilter(...)`.
- `BenchmarkVariantExtract`: the numbers above (int64/string x
clean/nulls/residual/unshredded/nested).
## AI Disclosure
- Model: Claude Opus 4.8
- Platform/Tool: Claude Code
- Human Oversight: fully reviewed
- Prompt Summary: Adopt arrow-go 18.8 and add a zero-copy fast path for
fully-shredded variant extract
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
