LukaZdravic commented on PR #59957: URL: https://github.com/apache/spark/pull/59957#issuecomment-6613192265
@cloud-fan FYI: - 9f9960db287 fixes the ANSI issue: the cursor now checks that the key has right rows before it evaluates the left as-of side, like the buffered scan. New tests cover an invalid cast and an overflow, with the cursor scan off and on. It also fixes the three nits. - Benchmark: my first numbers came from a run before SPARK-60582, when the DataFrame path still made both keys DOUBLE. `AsOfJoinBenchmark` now uses BIGINT keys on both sides, with a buffered scan and a cursor scan case. Local best times, buffered -> cursor: 100 groups 37 -> 26 ms, 10 groups 105 -> 22 ms, no equi-key 810 -> 15 ms. - 9b3fa868c5e is new and small: the scan no longer copies every right row it reads, only the cursor's best row, into one reused buffer. On the scan alone it is about 15% faster when most right rows are skipped (10M x 100M), and the same for 1:1. A new random test checks both scans against a plain join. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
