yihua opened a new pull request, #1002:
URL: https://github.com/apache/hudi-rs/pull/1002
closes #998
## Description
`FileStatsEstimator` derived bytes per row from the sampled file's row-group
compressed bytes but divided each file's whole size by it, so small files,
where the footer and metadata dominate, got inflated `num_records`. It now uses
the sample's whole file size, taken from the HEAD the footer read already
makes, and rounds the estimate. `compression_ratio` (behind `byte_size`) has
the same overhead skew and is left for a follow-up.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_from_parquet_footer_small_file_estimates_own_row_count` and
`test_get_file_slices_single_file_estimates_exact_num_records` fail without the
fix (33 and 802 estimated rows for files holding 5 and 4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]