GayathriSrividya opened a new pull request, #1022:
URL: https://github.com/apache/hudi-rs/pull/1022

   Closes #859
   
   ## Description
   
   Parquet omits NaN values from float min/max statistics, which can make Spark 
incorrectly skip files for matching predicates. Mark Float32/Float64 column 
ranges as unknown when they contain NaN, so those incomplete stats are not used 
to prune files. Writer paths inspect original batches rather than decoding 
newly written Parquet again.
   
   <!--- Describe your changes in detail -->
   
   <!--- If it fixes an open issue, please link to the issue here. -->
   
   <!--- Please link any related issues and PRs as well. -->
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
     - [ ] Details are described below
   
   Regression test: `./build-wrapper.sh cargo test -p hudi-core 
metadata::table::column_stats::tests::test_column_ranges_from_parquet_nan_values_are_not_prunable
 --lib` — passed (1 passed, 1708 filtered out).
   
   Spark read check not run: `SPARK_HOME` is unset and `spark-submit` is 
unavailable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to