alamb commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5590296831
> Type 1 can be optimized in a similar way with a dynamic filter, without adding too much extra complexity inside the operator. To implement type 2, we might need to maintain two parallel HJ implementations (order-preserving vs. non-order-preserving) plus extra optimizer rules, which is a lower ROI for the engineering effort involved. I agree we would probably need two implementations (or a flag that said order must be preserved, and then continue to error if the hash table exceeded the memory size). I think it would be hard to remove functionality that DataFusion already has (and make some queries slower) -- the good news is I think it would be fine to keep te existing limitations on HashJoin for the order preserving case (e.g. can't spill) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
