alamb opened a new issue, #11028: URL: https://github.com/apache/arrow-rs/issues/11028
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** - Follow on to https://github.com/apache/arrow-rs/pull/10842 from @etseidl We added the [`PageIndexProvider`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/page_index.rs#L139) trait so users can supply custom page index representations. However there is no way for `PageIndexProvider` to report their memory usage in [`ParquetMetaData::memory_size`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/mod.rs#L231) Since one of the main motivations for the provider trait is more efficient caching of parquet metadata, callers need accurate memory accounting for custom providers as well. See here for some prior attempts at exposing `HeapSize` publicly: - https://github.com/apache/arrow-rs/pull/9138 - https://github.com/apache/arrow-rs/pull/9842 **Describe the solution you'd like** Add a way for the `PageIndexProvider` trait to report its heap size so `ParquetMetaData::memory_size` can include it. Ideally implementors could re-use the [`HeapSize`](https://github.com/apache/arrow-rs/blob/79dbf5a3524cb8077d58eaec21c17f97de4ba4ca/parquet/src/file/metadata/memory.rs#L36) machinery the parquet crate already implements. **Describe alternatives you've considered** - Add a required (or defaulted-to-zero) `heap_size(&self) -> usize` method directly on `PageIndexProvider`, without exposing the `HeapSize` trait publicly. - Make the `HeapSize` trait public (per the prior attempts above) and require `PageIndexProvider: HeapSize`. **Additional context** - Original review discussion: https://github.com/apache/arrow-rs/pull/10842#discussion_r3913858985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
