paleolimbot commented on code in PR #2933: URL: https://github.com/apache/iceberg-rust/pull/2933#discussion_r3970696162
########## crates/iceberg/src/spec/datatypes.rs: ########## @@ -43,6 +43,8 @@ pub const MAP_VALUE_FIELD_NAME: &str = "value"; pub(crate) const MAX_DECIMAL_BYTES: u32 = 24; pub(crate) const MAX_DECIMAL_PRECISION: u32 = 38; +const DEFAULT_GEOSPATIAL_CRS: &str = "OGC:CRS84"; +const EQUIVALENT_DEFAULT_GEOSPATIAL_CRS: &str = "EPSG:4326"; Review Comment: It's all good...we've been having the CRS discussion for a long time! No questions are silly. > or you can implement CRS equivalence I am specifically thinking of the situation of inserting into an existing table from arbitrary (GeoArrow extension type) input. I assume there is some check that the data about to be inserted carries the same (or compatible) schema. If there's some kind of check that int8 Arrow data can be inserted into an Iceberg Int32, this is the same kind of check I'm talking about (totally valid to insert EPSG:4326 values into an Iceberg Geometry with the default CRS or one with "OGC:CRS84" or possibly any case-insensitive variant of either). Maybe that doesn't exist today. Most CRSes will arrive from Arrow/GeoArrow as PROJJSON (e.g., GeoPandas, DuckDB, GDAL) because the GeoArrow spec mandated that for a few years before Parquet/Iceberg formalized their spec, or EPSG:4326 because it's more widely used than OGC:CRS84 despite being slightly less correct because of the axis thing. So if you don't allow GeoArrow input with the PROJJSON equivalent of EPSG:4326, the PROJJSON equivalent of OGC:CRS84, or the authority/code equivalents of either of those to be inserted into previously existing tables, there will be a lot of failures. The easiest way to avoid all of that is to aggressively convert GeoArrow input to the default Iceberg CRS so that Iceberg can remain ignorant of CRS equality. Iceberg Java doesn't touch Arrow or GeoArrow, and Spark has its own integer based system, so I'm not sure there's precedent to draw on from there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
