alamb commented on issue #1826: URL: https://github.com/apache/datafusion-ballista/issues/1826#issuecomment-4918528236
I think there are two potental avenues to explore here: 1. If the optimizer came up with a better join plan (a different join order might be able to decrease the intermediate result cardinality and thus avoid the issue) 2. Improve whatever physical operator is generating huge intermediate results I personally think working on 2 (updating physical operators to not generate large single intermediate bathces) is valuable in general As for join ordering, I think that is much more complicated problem (as it relies on having good statistics, cardinality estimates, etc) that I am not as convinced we have a good generic story for -- see prior discussions like - https://github.com/apache/datafusion/issues/17718 - https://github.com/apache/datafusion/issues/18249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
