[
https://issues.apache.org/jira/browse/ARROW-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433403#comment-17433403
]
Micah Kornfield commented on ARROW-14303:
-----------------------------------------
Its been a while since I looked at this code, but I don't remember if all
metadata is copied back into the deserialized schema or if we'd have to modify
the read side as well (or maybe this is already implemented?)
I guess is is a 2X cost where X is the size of the metadata. so depending on
parameters and relative size of the data I suppose this could add up.
> [C++][Parquet] Do not duplicate Schema metadata in Parquet schema metadata
> and serialized ARROW:schema value
> ------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-14303
> URL: https://issues.apache.org/jira/browse/ARROW-14303
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 7.0.0
>
>
> Metadata values are being duplicated in the Parquet file footer — we should
> either only store them in ARROW:schema or the Parquet schema metadata.
> Removing them from the Parquet schema metadata may break applications that
> are expecting that metadata to be there when serialized from Arrow, so
> dropping the keys from ARROW:schema is probably a safer choice
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
