yihua opened a new issue, #998: URL: https://github.com/apache/hudi-rs/issues/998
### Description of the bug `FileStatsEstimator` samples one base file's footer, computes bytes per row from that file's row-group compressed sizes, and estimates each file's `num_records` as the file's whole size divided by that figure. The whole size also counts the footer and other file-level metadata, which the per-row figure leaves out. In small files that metadata dominates, so the estimate can be many times the real row count, which misleads anything that plans from these statistics. ### Steps To Reproduce Open a table with a single small base file, such as the `V6Nonpartitioned` COW sample table, and call `get_file_slices`. The base file's `file_metadata.num_records` is far above the 4 rows the table reads back. ### Expected behavior A file like the sample estimates to about its own row count, and the sample itself estimates to exactly its row count. ### Software information hudi-rs `main`. ### Additional context `compression_ratio`, which feeds `byte_size`, has the same overhead skew and can be fixed in a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
