shrivasshankar commented on issue #27146: URL: https://github.com/apache/datafusion/issues/27146#issuecomment-6670177608
sorry, I looked into it and reproduced it locally. in Q18, HAVING sum(l_quantity) > 300 filters the output of an aggregate, so the sum column has no min/max stats. before my PR, decimals weren't supported, so this filter got the default 20% (the ~120M rows in the 55.1 plan). now it goes through interval analysis, but with no bounds it estimates 100%, so 600M rows. that makes it look like the biggest input, and the semi join builds on the wrong side. I mentioned this "no stats means 100%" case in the PR description but didn't think about aggregate outputs. I reproduced it at SF1: the join flips and peak memory goes from ~430 MB to ~1.25 GB. for the re-land, I'll keep the default selectivity when a decimal column has no min/max stats, and add a Q18 plan test so this can't come back. i've tested that locally: Q18 gets its old plan back and the Q6 improvement stays. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
