etseidl opened a new issue, #11328: URL: https://github.com/apache/arrow-rs/issues/11328
### Is your feature request related to a problem or challenge? This is related to #7582 and grew out of a discussion in https://github.com/apache/datafusion/issues/24288. To summarize, given a desire to cache parts of the parquet metadata to allow selective population and reuse, somewhere along the line one encounters a need to clone the `ParquetMetaData` (for instance if the metadata is wrapped in an `Arc` and is subsequently shared). This clone can be very expensive as it entails copying the dense vector of `RowGroupMetaData`, which is essentially a `num_row_groups X num_columns` matrix of `ColumnChunkMetaData`s. ### Describe the solution you'd like Wrap the `file_metadata` and `row_groups` members of the `ParquetMetaData` in `Arc` to allow cheap cloning. This leads to few problems because those fields are generally read once and not mutated. The one production exception to this appears to be in `SerializedFileWriter::new_with_options` where a newly created `ParquetMetaData` is converted to a builder, the row groups extracted and filtered via a predicate, and the `ParquetMetaData` reassembled. ### Describe alternatives you've considered Careful management of the ParquetMetaData lifetime may allow clones to be avoided. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
