txwyy123 opened a new pull request, #27189: URL: https://github.com/apache/datafusion/pull/27189
## Which issue does this PR close? Closes #26748. ## Rationale After #26310 landed, Hive null partitions (`__HIVE_DEFAULT_PARTITION__`) are correctly read as NULL with `Absent` min/max statistics. However, when `Statistics::try_merge_iter` merges file statistics into file-group or table-level stats, `precision_min`/`precision_max` return `Absent` if either input is `Absent`. This means a single null-partition file wipes out the min/max bounds for its entire file group, defeating partition pruning for any table that has null partitions. ## Changes Added `is_all_null_or_empty()` that checks the pre-computed stats counts (no row scanning): it returns `true` when `null_count == num_rows` (both `Exact`) or `num_rows == Exact(0)`. Modified the merge loop in `try_merge_iter_with_ndv_fallback` to skip min/max from all-null or empty inputs instead of letting their `Absent` bounds poison the result. When counts are inexact or absent, the function conservatively returns `false` to preserve the existing behavior. Updated the `FileGroup` doc comment to reflect that all-null partitions are now skipped. ## Tests Added 5 test functions: - `test_is_all_null_or_empty` — unit tests for the helper - `test_try_merge_iter_skips_all_null_partition` — real + null merge in both orders, and null + null - `test_try_merge_iter_inexact_null_count_stays_conservative` — inexact counts degrade to Absent - `test_try_merge_iter_empty_partition_skipped` — empty partition (0 rows) is skipped - `test_try_merge_iter_three_files_with_null_partition` — three-way merge with a null partition in the middle All 82 stats tests pass (including existing tests), zero regressions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
