mohitgurav20 commented on issue #25623: URL: https://github.com/apache/datafusion/issues/25623#issuecomment-5821399987
Hi @gabotechs, thanks for opening this! This is a great cardinality estimation refinement. When scalar subqueries are used in equality predicates (e.g. col = (SELECT ...)), the planner currently falls back to a blanket 20% selectivity because the actual scalar value isn't available during planning. As you pointed out, we should instead fall back to the available column statistics—specifically, using 1.0 / NDV (Number of Distinct Values) for equality predicates, which provides a much more accurate estimate than an arbitrary 20%. I've recently been working on optimizing the physical execution layer (specifically memory and buffer tracking in HashJoinExec) and have been spending a lot of time with DataFusion's physical expressions and statistics, so I have the exact context needed to implement this correctly. I plan to update the selectivity estimation logic for binary expressions so that when one side is an unresolved scalar subquery (or any unknown scalar), we gracefully fall back to utilizing the other column's NDV from the available statistics instead of defaulting to 20%. Could you assign this to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
