etseidl opened a new pull request, #11297: URL: https://github.com/apache/arrow-rs/pull/11297
# Which issue does this PR close? None as yet, but related to discussion in https://github.com/apache/datafusion/issues/24288 # Rationale for this change One source of pain with the `ParquetMetaData` is the fact that cloning it is an expensive operation. This makes operations like cloning a cached footer metadata and adding a custom page index provider for the purposes of a single query inefficient. # What changes are included in this PR? Wrap the `file_metadata` and `row_groups` fields of `ParquetMetaData` with `Arc` so they can be easily shared. Modifies the `ParquetMetaDataBuilder` in the same way. Now simply replacing the page index in a shared `ParquetMetaData` can be done cheaply. Modifications to the `row_groups` will be copy-on-write operations that might incur a penalty. Also updates the custom page index provider example to show how a caching `PageIndexProvider` could be implemented. # Are these changes tested? Should be covered by existing tests # Are there any user-facing changes? No API changes, but memory for the metadata increases due to the `Arc` wrappers. Certain operations on `row_groups` may now trigger a `clone`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
