huan233usc opened a new pull request, #18072:
URL: https://github.com/apache/iceberg/pull/18072
Computes file-level 2D bounding-box metrics for `geometry` columns written to
Parquet. The bounds are stored in `lower_bounds`/`upper_bounds`, making
spatial
file pruning possible. This PR produces the bounds; expression and
scan-planner
integration that consumes them is separate follow-up work.
This is the first slice of the geo bounds work (Phase 2), scoped to the
clean,
unambiguous planar case.
## Problem
The ordinary Parquet column min/max for WKB is a lexicographic byte bound,
not a
spatial bound. Iceberg therefore diverts geometry/geography columns to
counts-only metrics in its current `ParquetMetrics` path. Although newer
Parquet
metadata can represent geospatial statistics separately, Iceberg does not
currently consume those footer bounds. As a result, geometry data files do
not
carry spatial bounds in Iceberg metadata.
## Approach
Compute the box while values are written, using the existing writer-side
value-scanning metrics channel -- the same path `float` and `double` use to
track
NaN counts that ordinary footer statistics cannot provide
(`ParquetValueWriter.metrics()` -> `ParquetWriter.metrics()`):
- `GeometryFieldMetrics` wraps core `GeometryBoundsBuilder` (#17509) and
emits a
`FieldMetrics<GeospatialBound>` whose lower/upper corners serialize
through the
existing geometry `Conversions` case. It carries no bounds when a
dimension is
absent or a value cannot be parsed.
- The generic Parquet `GeometryWriter` folds each value into the box while
writing
byte-identical WKB.
- `GeometryFieldMetrics` also tracks the average WKB size over non-null
values, so
replacing the counts-only geospatial writer for geometry columns preserves
the
`avg_value_size_in_bytes` metric from #17333.
This does not change `ParquetMetrics`, `ParquetWriter`, or `Conversions`.
NaN is
skipped per coordinate dimension in `GeometryBoundsBuilder`; malformed WKB
suppresses file bounds rather than failing the write.
## Scope
**GEOMETRY only, 2D (XY), planar.** This PR writes file bounds but does not
add a
spatial predicate to the Expression API or wire spatial pruning into scan
planning.
Deliberately left as follow-ups:
- **Spark** geometry writers (the Spark 4.1 typed `GeometryVal` path);
- **GEOGRAPHY** bounds, including longitude periodicity, edge latitude
extrema,
numerical coverage guarantees, pole handling, and coordinate-range policy;
- higher-dimensional Z/M bounds;
- the v4 `content_stats` `geo_lower`/`geo_upper` bridge;
- ORC and Avro geo bounds (Avro geo value I/O is supported separately, but
does
not produce spatial bounds);
- CRS validation; and
- avoiding WKB scanning when the selected `MetricsConfig` will not retain
bounds.
## Tests
- `TestGeometryFieldMetrics` covers cross-value aggregation, counts, empty or
no-value results, and that the average WKB size is still reported.
- `TestMetrics.testMetricsForGeospatialTypes` verifies that generic Parquet
writes
produce the expected geometry bounds while geography remains bounds-less,
and
keeps the existing `avg_value_size_in_bytes` assertion.
- `TestParquetValueWriters.geometryValueSizeMetricsExcludeNulls` checks that
the
geometry writer preserves the average WKB size over non-null values.
WKB walking, per-axis NaN accumulation, and malformed-input behavior live in
`TestGeometryBoundsBuilder` from #17509.
---
**AI Disclosure**
- Model: GPT-5
- Platform/Tool: Codex
- Human Oversight: partially reviewed
- Prompt Summary: Sync the PR with upstream main, resolve conflicts,
preserve geo average-size metrics, and complete bbox regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
