shrivasshankar commented on issue #27146: URL: https://github.com/apache/datafusion/issues/27146#issuecomment-6693809440
> > sorry, I looked into it and reproduced it locally. > > in Q18, HAVING sum(l_quantity) > 300 filters the output of an aggregate, so the sum column has no min/max stats. before my PR, decimals weren't supported, so this filter got the default 20% (the ~120M rows in the 55.1 plan). now it goes through interval analysis, but with no bounds it estimates 100%, so 600M rows. that makes it look like the biggest input, and the semi join builds on the wrong side. > > I mentioned this "no stats means 100%" case in the PR description but didn't think about aggregate outputs. I reproduced it at SF1: the join flips and peak memory goes from ~430 MB to ~1.25 GB. > > for the re-land, I'll keep the default selectivity when a decimal column has no min/max stats, and add a Q18 plan test so this can't come back. i've tested that locally: Q18 gets its old plan back and the Q6 improvement stays. > > I took a cursory look when the error was reported and came to the same conclusion and what you propose is a reasonable fix for the degenerate cases with no bounds, happy to take a look when ready to get it to a reviewable state for maintainers to approve (FYI I will be mostly off next week with limited connectivity, fully back after that). Thanks for taking a look that quickly, [@shrivasshankar](https://github.com/shrivasshankar)! Thanks! It’s ready for review in apache/datafusion#27148 no rush. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
