grantatspothero commented on PR #16207: URL: https://github.com/apache/iceberg/pull/16207#issuecomment-6348051057
A similar issue with excessive memory consumption also occurs here when table metadata has many snapshots and each snapshot contains a `StatisticsFile` and each `StatisticsFile` contains many `BlobMetadata` (ie: a theta sketch per column on a wide table) ``` 414MB metadata.json file size Statistics take up about 334MB (80%) of the file size. Theta sketch stored per each column (25 column table) per snapshot ~90K snapshots ┌──────────────────────────────────┬──────────────┬───────┐ │ Section │ Size │ Share │ ├──────────────────────────────────┼──────────────┼───────┤ │ statistics │ 334.1 MB │ 81% │ ├──────────────────────────────────┼──────────────┼───────┤ │ snapshots │ 74.2 MB │ 18% │ ├──────────────────────────────────┼──────────────┼───────┤ │ snapshot-log │ 5.8 MB │ 1.4% │ ├──────────────────────────────────┼──────────────┼───────┤ │ metadata-log and everything else │ under 0.1 MB │ ~0% │ └──────────────────────────────────┴──────────────┴───────┘ ``` This is the same problem as `snapshots` and `snapshot-log` growing per snapshot but worse because `statistics` can be stored per column. @nastra is it worth revisiting the idea of lazy loading? Statistics over wide tables show that the excessive metadata size occurs at even lower snapshot counts than I originally showed in the original PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
