Omkar-Wagholikar opened a new pull request, #27546: URL: https://github.com/apache/datafusion/pull/27546
## Which issue does this PR close? Closes issue #25253 ## Rationale for this change (After #25242) A missing Parquet null_count is treated as 'unknown'. That is right in general, but parquet-rs before 53.1.0 left out null_count whenever it was zero. For files from those writers, including DataFusion before 42.1.0, queries lost: - IS NULL row-group pruning - COUNT(col) answered from metadata - sort-pushdown Exact - TopK NULLS FIRST runtime pruning ## What changes are included in this PR? - missing_null_counts_are_zero(&FileMetaData) decides once per file from created_by. It returns true for parquet-rs below 53.1 and datafusion below 42.1. Any other writer, custom string or unparseable value returns false, so those files keep the #25242 behavior. - The result feeds StatisticsConverter::with_missing_null_counts_as_zero in row-group pruning (static, inverted and runtime TopK) and in statistics_from_parquet_metadata. - prune_by_statistics, which has no footer, stays conservative. - The existing footer-rewrite test fixture is split into a shared replace_footer helper that the new test uses. ## What is the testing strategy for this PR? - Unit tests cover the created_by parsing, including the 53.0/53.1 and 42.0/42.1 boundaries and malformed strings. They also cover file statistics and row-group pruning for legacy and non-legacy writers. - missing_null_count_is_zero_for_legacy_writers in datafusion/core/tests/parquet/dynamic_row_group_pruning.rs rewrites a real file's footer. For legacy writers WHERE v IS NULL prunes 2 of 3 row groups; for others it prunes none. Results match in every case. The test fails without the fix. - This can't be an .slt test, because it needs a hand-edited footer. ## Are there any user-facing changes? Queries over affected legacy files regain pruning and metadata-only optimizations. No API changes. Page-index null counts are unchanged, and I'd like guidance on whether they should be covered. I haven't updated the 56.0.0 upgrade-guide entry that #25242 added, since that release has an rc1 tag. I'd welcome advice on where a note belongs. I haven't benchmarked this. It restores optimizations rather than adding new ones. I used Claude Code to help find the code, and write the tests I written and reviewed most of the change myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
