westonpace commented on issue #35334: URL: https://github.com/apache/arrow/issues/35334#issuecomment-1527921729
This script is helpful, it reproduces the issue quite clearly. It seems the issue is definitely in the join code. A flame chart shows that significantly more time is spent both building and searching the hash table. The size of the key shouldn't really matter in this case since these operations are working on the hash which is always 32-bit. More concretely, the `arrow::compute::SwissTable::search_block` function is called an order of magnitude more times when working with numeric data. Based on my (admittedly rough) understanding I believe this suggests a less even distribution across hash buckets when working with numeric data. This fix is going to require someone motivated to really dig in and understand how the hash table works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
