viirya opened a new pull request, #58177: URL: https://github.com/apache/spark/pull/58177
### What changes were proposed in this pull request? When reading a projection of an Arrow-cached relation, `ArrowCachedBatchSerializer`'s read path deserialized **every** cached column -- decompressing and loading all of them off-heap -- and then discarded the unselected ones. This change reads only the selected columns out of the cached bytes. A new helper `ArrowCachedBatchSerializer.readProjectedRecordBatch` reads the encapsulated IPC RecordBatch message's metadata (a small flatbuffer that lists every buffer's offset and length within the body) and copies just the byte ranges belonging to the selected columns straight out of the in-memory cached `Array[Byte]`, so the unselected columns are never copied off-heap, loaded, or decompressed. The selected buffers become windows into a single off-heap allocation, mirroring how the standard IPC reader slices one body buffer, so ownership stays a single allocation freed once. It is wired into both read paths -- `convertCachedBatchToColumnarBatch` (columnar) and `convertCachedBatchToInternalRow` (row). A projection whose selected attribute is absent from the cache schema (index `-1`) falls back to reading the full batch. ### Why are the changes needed? The wasted work is proportional to the pruned columns and dominates wide-relation scans, especially with compression. An e2e SQL benchmark (`sum(col0)` over a cached relation, cache materialized outside the timed region, 1M rows x 50 long columns) shows: | | before | after | |---|---|---| | Arrow cache -- uncompressed | 31 ms | 16 ms (~1.9x) | | Arrow cache -- zstd level 1 | 326 ms | 22 ms (~15x) | The compressed case gains most, since the 49 pruned columns are no longer decompressed. ### Does this PR introduce _any_ user-facing change? No. The Arrow cache serializer (SPARK-57268) is unreleased, and this is a read-path performance improvement with identical results. ### How was this patch tested? - A new `ArrowCachedBatchSerializerSuite` test covers projection shapes (reordering, single column at each position, complex-after-var-width, duplicate selection, full projection) on both read paths, and is negative-validated (an off-by-one in the buffer-span arithmetic fails it). - Full `ArrowCachedBatchSerializerSuite` and `ArrowCachedBatchKryoRegistrationSuite` pass (77 tests). - A new `columnPruningWideTable` case in `ArrowCacheBenchmark` measures the read path with the cache built outside the timed region; the committed benchmark result files are regenerated by the benchmark CI job. ### Was this patch authored or co-authored using generative AI tooling? Yes, this pull request and its description were written by Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
