SubhamSinghal opened a new pull request, #21730:
URL: https://github.com/apache/datafusion/pull/21730
## Which issue does this PR close?
- Closes #
## Rationale for this change
Large `IN` lists (>20 values) in Parquet row group pruning were silently
returning `true` (no pruning), meaning all row groups were kept even when a
tighter predicate was possible. For a non-negated `IN` list with comparable
literal values, we can derive a coarse but correct range predicate `col >= min
AND col <= max` that prunes row groups whose
entire range falls outside the IN list span.
## What changes are included in this PR?
When an `IN` list exceeds the 20-value threshold, instead of returning
`true`, scan all list elements for the min and max
value and emit `col >= min AND col <= max`. This falls back to `true`
safely if:
- The list is negated (`NOT IN`) — no useful range exists
- Any element is not a literal
- Any two elements are incomparable (e.g. mixed timestamp
units/timezones — `partial_cmp` returns `None`)
## Are these changes tested?
Yes, with UT
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
