dongjoon-hyun opened a new pull request, #58971: URL: https://github.com/apache/spark/pull/58971
### What changes were proposed in this pull request? `SortMergeAsOfJoinScanner` rescans the buffered right-side group once per left row, and did `bestMatch = rightRow.copy()` for every right row that improved on the match so far, although only the final one is ever used. That copy is only needed when `ExternalAppendOnlyUnsafeRowArray` has switched to its spillable backing store, whose iterator re-points a single `UnsafeRow` on every `next()`. While the buffer is in memory the iterator returns the distinct rows it stores, so a retained match stays valid. This PR exposes that as `ExternalAppendOnlyUnsafeRowArray.isSpillBacked` and copies a match only when it is true. The flag is read per scan rather than cached, because `clear()` drops the spillable backing store between equi-key groups. ### Why are the changes needed? The copies are pure overhead on the common, non-spilled path, and there are `O(matches)` of them per left row. `AsOfJoinBenchmark` on an Apple M3 Max / JDK 21, `Best Time(ms)` of the "Sort-merge AS-OF join" case. Run-to-run variance is high on this workload, so before and after were run alternately in separate JVMs for three rounds: | Case | Before | After | |---|---|---| | `groups=100` | 32, 29, 29 | 24, 27, 26 | | `groups=10` | 119, 95, 93 | 62, 63, 64 | | no equi-key | 944, 707, 691 | 448, 457, 454 | Reusing one `UnsafeRow` via `copyFrom` was measured too and did not help: it removes the allocation but keeps the memory copy, which is what costs here. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
