rich7420 opened a new pull request, #6845: URL: https://github.com/apache/datafusion-comet/pull/6845
## Which issue does this PR close? Closes #6525. ## Rationale for this change Map lookups selecting short nested values can retain buffers sized for much larger values that were not selected, increasing memory use and spill pressure. ## What changes are included in this PR? Gather nested values from the selected ranges, using typed copies for primitive children and compaction for oversized owned buffers. Preserve nulls, sliced arrays and first-match lookup semantics. Add retained-capacity regressions, native SQL coverage and benchmarks. This PR is independent of #6498 and leaves list extraction unchanged. ## How are these changes tested? Rebased onto main `bfacc43a5`: - 26 Rust tests passed: 18 map lookup, 4 gather and 4 list extraction tests. - Expression-crate Clippy passed over all targets, including benchmarks; Rust formatting and whitespace checks passed. - Root-reactor Spark 4.1 test compilation, Spotless and Scalastyle passed. - The six-query nested-map SQL fixture passed on Spark 4.1 with native-execution assertions. The JNI library was rebuilt from this checkout and its checksum matched the library used by the JVM tests. Performance benchmarks were not rerun during this rebase. Please apply `run-spark-4.1-tests` for broader coverage; keep Draft pending CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
