[
https://issues.apache.org/jira/browse/ARROW-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487624#comment-17487624
]
Micah Kornfield commented on ARROW-15548:
-----------------------------------------
I don't think it is a bad idea to have something control for placing metadata
for each column on a per row-group basis. Based on the discussion on this
thread it seems like this should be opt-in though
> [C++][Parquet] Field-level metadata are not supported?
> (ColumnMetadata.key_value_metadata)
> ------------------------------------------------------------------------------------------
>
> Key: ARROW-15548
> URL: https://issues.apache.org/jira/browse/ARROW-15548
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Parquet
> Reporter: Joris Van den Bossche
> Priority: Major
>
> Due to an application where we are considering to use field-level metadata
> (so not schema-level metadata), but also want to be able to save this data to
> Parquet, I was looking into "field-level metadata" for Parquet, which I
> assumed we supported this.
> We can roundtrip Arrow's field-level metadata to/from Parquet, as shown with
> this example:
> {code:python}
> schema = pa.schema([pa.field("column_name", pa.int64(), metadata={"key":
> "value"})])
> table = pa.table({'column_name': [0, 1, 2]}, schema=schema)
> pq.write_table(table, "test_field_metadata.parquet")
> >>> pq.read_table("test_field_metadata.parquet").schema
> column_name: int64
> -- field metadata --
> key: 'value'
> {code}
> However, the reason this is restored is actually because of this being stored
> in the Arrow schema that we (by default) store in the {{ARROW:schema}}
> metadata in the Parquet FileMetaData.key_value_metadata.
> With a small patched version to be able to turn this off (currently this is
> harcoded to be turned on in the python bindings), it is clear this
> field-level metadata is not restored on roundtrip without this stored arrow
> schema:
> {code:python}
> pq.write_table(table, "test_field_metadata_without_schema.parquet",
> store_arrow_schema=False)
> >>> pq.read_table("test_field_metadata_without_schema.parquet").schema
> column_name: int64
> {code}
> So there is currently no mapping from Arrow's field level metadata to
> Parquet's column-level metadata ({{ColumnMetaData.key_value_metadata}} in
> Parquet's thrift structures).
> (which also means that using field-level metadata roundtripping to parquet
> only works as long as you are using Arrow for writing/reading, but not if you
> want to be able to also exchange such data with non-Arrow Parquet
> implementations)
> In addition, it also seems we don't even expose this field in our C++ or
> Python bindings, to just access that data if you would have a Parquet file
> (written by another implementation) that has key_value_metadata in the
> ColumnMetaData.
> cc [~emkornfield]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
