dongjoon-hyun commented on code in PR #58969:
URL: https://github.com/apache/spark/pull/58969#discussion_r4074280533
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/AsOfJoinBenchmark.scala:
##########
@@ -125,6 +125,9 @@ object AsOfJoinBenchmark extends SqlBasedBenchmark {
runBenchmark("AS-OF Join Benchmark") {
// 10K left x 10K right, 100 groups - both paths feasible
asOfJoinBenchmark(leftRows = 10000, rightRows = 10000, numGroups = 100)
+ // Few large groups: each left row scans ~1K buffered right rows, which
is where the
+ // per-left-row cost of the right-side group scan dominates.
Review Comment:
Thanks for catching this, @peter-toth - you're right, and I applied your
suggestion in ad9e72d.
I confirmed the early return in `findBestBackwardForward`: once `bestMatch`
is set, the first row
failing the as-of condition returns immediately, so a left row walks a
prefix of the buffer rather
than all of it. Simulating the fixture independently, I get the same
magnitudes you did - avg 35.3
comparisons per left row at `groups=100` vs 335.3 at `groups=10`, a 9.5x
increase - so the new
case is still the scan-sensitive shape it is meant to be, just with the
comment now describing
the right quantity.
(Minor: my count of full-buffer scans came out as 99 and 9 rather than 98
and 8 - the left row
whose `ts` equals its group id has no match for every group except `group_id
= 0`. Doesn't affect
the point.)
I left the PR description as is, since "buffers ~1000 right rows per group"
is about the buffer
size and is accurate as written.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
