nssalian opened a new pull request, #1478: URL: https://github.com/apache/iceberg-go/pull/1478
Closes [#988](https://github.com/apache/iceberg-go/issues/988). Writes per-field min/max bounds for shredded variant columns. Each shredded `typed_value` field's bounds are serialized as a variant object keyed by its normalized JSON path and stored under the parent variant field id, matching Java's `ParquetMetrics.variant()`. The parent variant column still emits no scalar min/max. Builds on the shredded writer [#1351](https://github.com/apache/iceberg-go/pull/1351). ## Changes - `collectVariantBounds` walks each shredded variant's `typed_value` subtree, reads each primitive leaf's Parquet min/max, and serializes the lower/upper bounds as a variant object (spec "Bounds for Variant"). Keys are RFC-9535 normalized JSON paths, matching Java's `PathUtil`. - A field's bound is dropped when its residual `value` column holds a non-null, non-variant-null value (partial shred / mixed type), and when min or max is NaN. Both match Java. - Arrays contribute no bounds (Java parity). - `DataFileStatsFromMeta` takes the inferred arrow schema to locate shredded leaves; `ToDataFile` merges the bounds into `lower_bounds`/`upper_bounds`. The add-files path passes no schema and does not recompute. ## Notes - Integer bounds use arrow-go's minimal-width `AppendInt`, not Java's fixed INT32/INT64. Both are valid variant encodings and read to the same integer, so bounds are read-equivalent across implementations though not byte-identical. - Java also emits variant field value/null counts; iceberg-go still omits those (pre-dates this PR, out of scope). Tracking as a follow-up. - The residual-value invalidation matches Java's `MetricsVariantVisitor.value()` rule, including the no-usable-stats case hardened in [apache/iceberg#16585](https://github.com/apache/iceberg/pull/16585). ## Testing - Unit: path escaping incl. control chars, leaf enumeration (nested object, narrow int, array-skip, root scalar), bounds serialization, NaN guard. - Write path: byte-exact bounds for all 14 shredded scalar types; nested object; a null-coexisting field keeps its bound; a mixed-type field is dropped. - Spark 4 integration: a Go-written shredded table carries variant child bounds and Spark reads it back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
