alamb commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5468121036
I have a few questions of your plan @jayzhan211: 1. What happens if the planner picks CollectLeft but the input can not be buffered in the available memory (will it still OOM)? 2. What happens when one of the initial 16 radix / sub buckets in a partition doesn't fit in memory? Will that bucket be split too? 3. The description mentions the fact that HashJoin today preserves probe side input order at the output. If we bucket the probe side to spill, we will lose the input order. Will the HashJoin recover the input order somehow (e.g. a resort) or just advertise it doesn't preserve the input order anymore? 4. What will happen if one bucket gets very large, but the other buckets aren't out of memory yet (e.g. the case where there is one single key that has 90% of the rows)? > I'll give it a try shortly to see if we can make the HJ implementation simpler first. I think this is a great plan @2010YOUY01 -- especially in this day and age when the bottleneck is our ability to understand the code that LLMs / coding tools generate, keeping the code simpler to review / understand I think is one of the most valuable endeavors -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
