Omkar-Wagholikar opened a new pull request, #27546:
URL: https://github.com/apache/datafusion/pull/27546

   ## Which issue does this PR close?
   Closes issue #25253
   
   ## Rationale for this change
   (After #25242) A missing Parquet null_count is treated as 'unknown'. That is 
right in general, but parquet-rs before 53.1.0 left out null_count whenever it 
was zero. For files from those writers, including DataFusion before 42.1.0, 
queries lost:
   - IS NULL row-group pruning
   - COUNT(col) answered from metadata
   - sort-pushdown Exact
   - TopK NULLS FIRST runtime pruning
   
   ## What changes are included in this PR?
   - missing_null_counts_are_zero(&FileMetaData) decides once per file from 
created_by. It returns true for parquet-rs below 53.1 and datafusion below 
42.1. Any other writer, custom string or unparseable value returns false, so 
those files keep the #25242 behavior.
   - The result feeds StatisticsConverter::with_missing_null_counts_as_zero in 
row-group pruning (static, inverted and runtime TopK) and in 
statistics_from_parquet_metadata.
   - prune_by_statistics, which has no footer, stays conservative.
   - The existing footer-rewrite test fixture is split into a shared 
replace_footer helper that the new test uses.
   
   
   ## What is the testing strategy for this PR?
   - Unit tests cover the created_by parsing, including the 53.0/53.1 and 
42.0/42.1 boundaries and malformed strings. They also cover file statistics and 
row-group pruning for legacy and non-legacy writers.
   - missing_null_count_is_zero_for_legacy_writers in 
datafusion/core/tests/parquet/dynamic_row_group_pruning.rs rewrites a real 
file's footer. For legacy writers WHERE v IS NULL prunes 2 of 3 row groups; for 
others it prunes none. Results match in every case. The test fails without the 
fix.
   - This can't be an .slt test, because it needs a hand-edited footer.
   
   ## Are there any user-facing changes?
   
   Queries over affected legacy files regain pruning and metadata-only 
optimizations. No API changes. Page-index null counts are unchanged, and I'd 
like guidance on whether they should be covered. I haven't updated the 56.0.0 
upgrade-guide entry that #25242 added, since that release has an rc1 tag. I'd 
welcome advice on where a note belongs.
   
   I haven't benchmarked this. It restores optimizations rather than adding new 
ones.
   
   I used Claude Code to help find the code, and write the tests  I written and 
 reviewed most of the change myself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] fix: treat missing Parquet ... Omkar-Wagholikar (via GitHub) via github

Reply via email to