etseidl opened a new pull request, #11159: URL: https://github.com/apache/arrow-rs/pull/11159
# Which issue does this PR close? - Closes #7582. # Rationale for this change The current `PageIndex` uses a nested vector of vectors to hold the page indexes. This wastes a great deal of memory when only a few columns or row groups are desired. # What changes are included in this PR? Following a [suggestion](https://github.com/apache/arrow-rs/pull/10784#discussion_r3836791268) from @adriangb, this replaces the dense vector of vectors with a new `Grid` structure. Testing has shown that this structure can greatly reduce memory usage, while incurring an acceptable performance hit on index lookup. # Are these changes tested? Yes, new tests are added # Are there any user-facing changes? No, only internal storage structures are changed # AI assistance The original idea is from Claude Code, and I had Codex refine it a bit. My own madness is sprinkled in as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
