paleolimbot opened a new issue, #18680: URL: https://github.com/apache/iceberg/issues/18680
### Apache Iceberg version main (development) ### Query engine None ### Please describe the bug 🐞 After https://github.com/apache/iceberg/pull/17578, Parquet types are validated against the Iceberg type. This is great! In an attempt to reduce the number of readers that had to consider CRSes or PROJJSON or axis ordering, arrow-rs and Arrow C++ (i.e., the arrow interfaces to Parquet writing) make an attempt to write a defaulted CRS. Unfortunately, this means those interfaces can't currently write Parquet files with "EPSG:4326" as the Geometry or Geography CRS. arrow-rs: https://github.com/apache/arrow-rs/blob/7dcc1602c460e25a624fd4f095a786352808b884/parquet/src/arrow/schema/extension.rs#L183-L196 arrow C++: https://github.com/apache/arrow-rs/blob/7dcc1602c460e25a624fd4f095a786352808b884/parquet/src/arrow/schema/extension.rs#L183-L196 Notably, iceberg-rust is using this interface. This means that when it writes a table in "EPSG:4326", the Parquet files have Geometry or Geography fields with a defaulted CRS, which iceberg-java reads back as "ogc:crs84", considers un equal, and fails validation. EPSG:4326 is probably the most common thing somebody might type themselves here. Would it be possible to have the Equals implementations be updated to consider EPSG:4326 and OGC:CRS84 equal to each other? https://github.com/apache/iceberg/blob/3239fc629e8b8dba1427dc720caf69ae1d432c7d/api/src/main/java/org/apache/iceberg/types/Types.java#L685-L699 https://github.com/apache/iceberg/blob/3239fc629e8b8dba1427dc720caf69ae1d432c7d/api/src/main/java/org/apache/iceberg/types/Types.java#L608-L622 ..or could this be considered "type promotion" to allow the two concepts to coexist more readily? https://github.com/apache/iceberg/blob/3239fc629e8b8dba1427dc720caf69ae1d432c7d/api/src/main/java/org/apache/iceberg/types/TypeUtil.java#L471-L497 cc @dannycjones who spotted this! ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
