etseidl opened a new pull request, #11475: URL: https://github.com/apache/arrow-rs/pull/11475
# Which issue does this PR close? - Closes #7582 # Rationale for this change Same as #11159. This is an alternative that is perhaps more flexible. # What changes are included in this PR? This is stacked on #11438 and #11297 so the diff is confusing. Focus on the changes to `page_index.rs` and the new page_index_cache example. This takes a page from the grid work, but this uses a key structure that allows for missing indexes without a need to wrap them in `Option`. This also uses different storage for the `PageIndex` itself vs the `PageIndexBuilder`. The latter uses a `HashMap` to allow for easy insertion to change the index shape, and the former then transforms back to a rigid immutable shape, with (hopefully) easy transitions between the two. The other big change is wrapping each index in `Arc` to allow for easy sharing with a cache. This adds some overhead, but hopefully the benefits outweigh the cost. # Are these changes tested? Yes there are new tests added and existing tests cover a lot of this. # Are there any significant user-facing changes? There are no changes to the public API, but memory use for sparse indexes should drop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
