laskoviymishka commented on issue #989: URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-6680086033
>> the parquet writer in arrow-go that iceberg-go uses doesn't write geospatial row group statistics Yep, it’s a gap. arrow-go has the generated thrift struct, but nothing populates it on write or exposes it on read. On the iceberg-go side we worked around that in #1371 by computing bounds from WKB ourselves for the manifest, so pruning works for files we write. The remaining gap is imports: files added through `AddDataFilesByPath` only go through `DataFileStatsFromMeta`, so they end up without geo bounds. On @zeroshade’s question: C++ hand-rolls this with a ~250-line `WKBGeometryBounder` in `parquet/geospatial`, no geo dependency, and only for geometry; geography is skipped. Rust has a separate `parquet-geospatial` crate on top of georust `wkb` / `geo-traits`, behind an optional geospatial feature. So both do it in the writer from the logical type, which also points away from adding an extension hook. My slight preference is to do the same in arrow-go: a small WKB bounder rather than pulling `go-geom` into the parquet module, since Go doesn’t have cargo-style feature gating. For bbox we only need to walk the coordinates, not build a full geometry model. @zeroshade i'll leave a final dependency call on you. If this lands, it would also be useful to expose the stats on read, so iceberg-go can pick up bounds from externally written files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
