HaoXuAI commented on issue #17743: URL: https://github.com/apache/iceberg/issues/17743#issuecomment-5392727456
Thanks @RussellSpitzer — I checked against main (`f0e7de3c2f`) and I don't think v4 gives us this, for a stronger reason than I first thought. `V4Metadata.MANIFEST_LIST_SCHEMA` (core/src/main/java/org/apache/iceberg/V4Metadata.java:32-49) is field-for-field the v3 set — three `*_files_count`, three `*_rows_count`, no byte total. So the v4 writer as it stands today doesn't summarize bytes. And `content_stats` can't be rolled up into one: `FieldStats` exposes `lowerBound`, `upperBound`, `valueCount`, `nullValueCount`, `nanValueCount` and `avgValueSizeInBytes`, and that last one is documented as the average **in-memory, uncompressed** size for memory estimation. There's no on-disk byte stat in there. On-disk size lives in `DataFile.FILE_SIZE` (104) on the entry, inside the manifest body — so answering "how big is this table" is still a full manifest scan, which is the IO this issue is about. #16025 is also widening the gap rather than closing it: it adds `replaced_files_count`, `modified_files_count`, `replaced_rows_count` and `modified_rows_count` to the summary, still with no `*_files_size_in_bytes`. Since that PR is already reshaping this struct, it seems like the natural place to add them — and the write-side cost is three longs the writer is already summing per entry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
