asubiotto opened a new pull request, #8953: URL: https://github.com/apache/arrow-rs/pull/8953
Previously, List and LargeList would fall through to the interleave_fallback match arm, which is inefficient. This commit implements interleave_list, which interleaves a list's child arrays and rebuilds the offsets buffer. Running it on production tests reduced memory by 80%. # Which issue does this PR close? - Closes #8952 # Rationale for this change Performance and memory usage when interleaving List/LargeList # Are these changes tested? This PR does not include tests because interleave tests for Lists already exist # Are there any user-facing changes? No, purely performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
