yihua opened a new issue, #998:
URL: https://github.com/apache/hudi-rs/issues/998

   ### Description of the bug
   
   `FileStatsEstimator` samples one base file's footer, computes bytes per row 
from that file's row-group compressed sizes, and estimates each file's 
`num_records` as the file's whole size divided by that figure. The whole size 
also counts the footer and other file-level metadata, which the per-row figure 
leaves out. In small files that metadata dominates, so the estimate can be many 
times the real row count, which misleads anything that plans from these 
statistics.
   
   ### Steps To Reproduce
   
   Open a table with a single small base file, such as the `V6Nonpartitioned` 
COW sample table, and call `get_file_slices`. The base file's 
`file_metadata.num_records` is far above the 4 rows the table reads back.
   
   ### Expected behavior
   
   A file like the sample estimates to about its own row count, and the sample 
itself estimates to exactly its row count.
   
   ### Software information
   
   hudi-rs `main`.
   
   ### Additional context
   
   `compression_ratio`, which feeds `byte_size`, has the same overhead skew and 
can be fixed in a follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to