paleolimbot commented on code in PR #8801:
URL: https://github.com/apache/arrow-rs/pull/8801#discussion_r2519378777
##########
parquet/src/arrow/schema/mod.rs:
##########
@@ -76,16 +77,18 @@ pub(crate) fn parquet_to_arrow_schema_and_fields(
key_value_metadata: Option<&Vec<KeyValue>>,
) -> Result<(Schema, Option<ParquetField>)> {
let mut metadata =
parse_key_value_metadata(key_value_metadata).unwrap_or_default();
- let maybe_schema = metadata
+ let mut maybe_schema = metadata
.remove(super::ARROW_SCHEMA_META_KEY)
.map(|value| get_arrow_schema_from_metadata(&value))
.transpose()?;
// Add the Arrow metadata to the Parquet metadata skipping keys that
collide
- if let Some(arrow_schema) = &maybe_schema {
+ if let Some(arrow_schema) = maybe_schema.as_mut() {
arrow_schema.metadata().iter().for_each(|(k, v)| {
metadata.entry(k.clone()).or_insert_with(|| v.clone());
});
+ #[cfg(feature = "geospatial")]
+ parquet_geospatial::crs::replace_keyvalue(arrow_schema, &metadata)
Review Comment:
I think we're on the same page...I may be misunderstanding how this might
get invoked, but I think:
- A user wants to write arrow data with a given schema, and that arrow
schema contains a column with GeoArrow metadata
- The existing implementation serializes that schema, including the GeoArrow
metadata on the column and puts it in this key/value key
- Independently, the Parquet schema converts the Parquet logical types for
readers that can't (or have been requested not to) inspect the embedded Arrow
schema.
- Readers that read the file from Arrow get the type exactly as it was
written
- Readers that read the file from elsewhere still have correct CRS
information as it was translated to Parquet
Is there another situation I haven't considered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
