yihua opened a new pull request, #1002:
URL: https://github.com/apache/hudi-rs/pull/1002

   closes #998
   
   ## Description
   
   `FileStatsEstimator` derived bytes per row from the sampled file's row-group 
compressed bytes but divided each file's whole size by it, so small files, 
where the footer and metadata dominate, got inflated `num_records`. It now uses 
the sample's whole file size, taken from the HEAD the footer read already 
makes, and rounds the estimate. `compression_ratio` (behind `byte_size`) has 
the same overhead skew and is left for a follow-up.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
     - [ ] Details are described below
   
   `test_from_parquet_footer_small_file_estimates_own_row_count` and 
`test_get_file_slices_single_file_estimates_exact_num_records` fail without the 
fix (33 and 802 estimated rows for files holding 5 and 4).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to