kumarUjjawal commented on issue #24711: URL: https://github.com/apache/datafusion/issues/24711#issuecomment-5466623131
I opened #24781 for the NOT IN part of this issue. The PR adds compact pruning for large, non-null string NOT IN lists. It also tests mixed containers to ensure that pruning does not remove a container containing both excluded and allowed values. The remaining work is compact pruning for NULL-containing lists, for both IN and NOT IN. This needs separate handling because it must preserve SQL three-valued logic during pruning and in identify_fully_matched_row_groups. In particular: - x IN (..., NULL) can return TRUE or UNKNOWN. - x NOT IN (..., NULL) can return FALSE or UNKNOWN. - Removing the NULL literal can therefore produce an unsafe full-match decision. I plan to address NULL-containing lists in a follow-up. Once that change is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
