alamb commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5551736957
As a specific suggestion, I wonder if we could do something like: 1. Phase 1: Have the hash join fall back to sort-merge-join on memory pressure 2. Phase 2: Initial hybrid hash join (e.g. spill individual hash partitions, etc), but fallback to sort-merge-join for cases that are not (yet) handled like a single hot key, etc) 3. Phase 3-N: support hybrid hash join in more of the corner cases That would get us the most important feature (no OOM's with default setting) and then adding the additional features would effectively be performance improvements for the spilling case -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
