C-Loftus commented on issue #2193: URL: https://github.com/apache/iceberg-go/issues/2193#issuecomment-6273717099
Discussed with others in the iceberg slack in the #dev channel. I came to this conclusion: > This scenario is in which there is Parquet file whose column is annotated with the native GEOMETRY or GEOGRAPHY logical type, being read or added into an Iceberg table where that same column is declared as plain binary - Java Iceberg appears to [not allow this](https://github.com/apache/iceberg/blob/6890694bd326648c3ab5930b8c97986118bbc1e5/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/TestSparkSchemaUtil.java#L185) - In the case of a V2 table without the proper GEOMETRY logical type annotation, there wouldn't be spatial statistics on the column. As a result, adding V3 `GEOMETRY` typed data to this column is likely to cause an issue since it would mix data with row group spatial statistics and some without it. It is also possible there could be issues with query planners if there is a mix of BINARY row group stats that simply store the max/min binary value as opposed to the GEOMETRY row group stats which store a bounding box. So this is the binary <-> geo projection status in iceberg-go: | Parquet file column being read or added |Existing Iceberg table column type its being added to | Status | |---|---|---| | Plain binary (BYTE_ARRAY, no geo logical type) added to | geometry column (v3) | Fails (unchanged, intended): arbitrary bytes aren't guaranteed to be valid WKB | | GEOMETRY/GEOGRAPHY logical type added to | geometry column (v3) | Now works (the fix from the geoarrow-go bump in https://github.com/apache/iceberg-go/pull/2125) | | GEOMETRY/GEOGRAPHY logical type added to | binary column (e.g. a v2 table) | Fails with `cannot promote geometry to binary`. While theoretically possible to implement, high chance for issues with mismatched/missing geospatial row group stats propagating to issues in query planners | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
