HaoXuAI opened a new issue, #17743: URL: https://github.com/apache/iceberg/issues/17743
### Feature Request / Improvement `manifest_list` entries summarize file counts and row counts, but not bytes: ``` added_files_count existing_files_count deleted_files_count added_rows_count existing_rows_count deleted_rows_count manifest_length <- the manifest file's own size, not the data it describes ``` Every `data_file` already carries `file_size_in_bytes`, so the information exists — it just isn't rolled up to where it can be read cheaply. ## Proposal Three optional fields on `manifest_file`, mirroring the existing triplets: ``` added_files_size_in_bytes existing_files_size_in_bytes deleted_files_size_in_bytes ``` Zero write cost (the writer already holds every `file_size_in_bytes`), and backward compatible — absent means readers fall back to reading bodies as today. ## Why snapshot summary `total-files-size` doesn't cover it That gives one snapshot's total. Summing it across live snapshots double counts every file shared between them. Manifests are the natural dedup unit, since a manifest is inherited wholesale rather than rewritten — so dedup by `manifest_path`, then sum. That composes; per-snapshot totals don't. ## Cost today One production table (Iceberg 1.9, Spark, S3): 344 live snapshots, 18,425 manifests, 8.1 MB average, **146 GB of manifest bodies read per run**, ~24 minutes. Thread dumps show 6 of 8 reader threads inside `java.util.zip.Inflater`, so decompression dominates, not S3. The table gains **~2 snapshots a day**. Manifests are immutable, so >99% of that 146 GB is re-decompressed each run to recover numbers that never changed. With a byte summary, the same result comes from 344 manifest lists instead of 18,425 bodies. ## Beyond this use case Anything needing size rather than contents: storage cost attribution, compaction and retention planning, catalog-level dashboards over thousands of tables. #14803 and #14820 added delete-file size totals to `PartitionsTable`; this is the same idea in the manifest list, where it's readable without a scan. I couldn't find prior discussion (searched `added_files_size`, `manifest_list` + bytes, spec-labelled size proposals) — happy to be pointed at it. As a spec change this presumably needs a dev@ thread and spec PR; filing here first to gauge support. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
