zanmato1984 commented on PR #53816: URL: https://github.com/apache/arrow/pull/53816#issuecomment-6304093509
The linked issue reports aggregation becoming exceptionally slow after a right outer join. This PR attributes that slowdown to repeated table growth, but the reproduction does not support that diagnosis. I tested the original Acero plan using Arrow 26 static libraries, whose relevant baseline sources match this PR, rebuilding the SwissTable translation unit from both the base and head (`815e3d7`). All completed runs validated the resulting groups and counts. - With one probe batch, 32 build batches of 32,768 rows, and four workers, the original plan took 39.67 seconds both before and after this change. - Replaying 524,288 actual join-output rows through aggregation alone on one worker took 8.010 seconds before and 8.035 seconds after. Both versions performed 17 grows and exactly the same 1,219,618,330 slow-path block visits. - In the baseline replay, lookup took approximately 7.27 seconds, while `grow_double()` took approximately 0.096 seconds. Sorting or shuffling the same rows before aggregation reduced aggregation time to approximately 0.011–0.012 seconds, still with 17 grows; preparation time was excluded. These results point to order-dependent probing costs, not resizing as the dominant cause of this workload's slowdown. Also, the implementation still loops over `grow_double()`; it does not perform a single target-sized allocation as described. I do not think this should be merged as a fix for #45847. The root-cause analysis and proposed approach need to be revisited. If pre-growing is intended as a separate optimization, please provide a workload demonstrating a measurable benefit and adjust the description and issue-closing claim accordingly. Separately, this investigation has identified substantial room to improve Fast Grouper's handling of these input orders, and I plan to work on that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
