alamb commented on issue #24768:
URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5462774736
@peterxcli I think @jayzhan211 's writeup is a design proposal. My reading
of it is:
1. Add a dynamic transition from `CollectLeft` --> `Partitioned` mode if the
single collect left hash table is too large
2. Add per-partition spilling to the Partitioned mode
I may have mis read it, and I have some suggestions on how to make this
proposal clearer:
<details><summary>Suggestions</summary>
<p>
1. "What is going on (symptoms)" heading:
1. Add a link or the exact query that is being run explicitly (the text
decribes the scenario in the abstract but is not a concrete reprodicer)
2. Describe the different options available to run the query today a bit
more ('default settings', "Always use Merge Join", "count(DISTINCT payload)
hash aggregate") -- it was not entirely clear to me what those options
Also, what is the difference between "hash join, build side 12M rows" and
"hash join t_probe ⋈ t_build ON k" ? Is the idea that we can't get the join to
run by switching the inputs? If so I suggest explicitly saying so and using the
same format (maybe you could collapse the table entries into a single row)
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
