[
https://issues.apache.org/jira/browse/IMPALA-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091432#comment-18091432
]
Csaba Ringhofer commented on IMPALA-13794:
------------------------------------------
>IcebergFileDescriptor sizes are larger than plain FileDescriptor sizes,
>especially when the table is partitioned
Note that this was recently changed in IMPALA-14564, which deduplicates
partition info.
> Calculate memory usage estimates more precisely for Iceberg tables
> ------------------------------------------------------------------
>
> Key: IMPALA-13794
> URL: https://issues.apache.org/jira/browse/IMPALA-13794
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Zoltán Borók-Nagy
> Assignee: Jason Fehr
> Priority: Major
> Labels: impala-iceberg, ramp-up
>
> Currently we use HdfsTable's memory usage estimates for Iceberg tables, even
> though they have different characteristics.
> We should come up with a calculation that is better suited for Iceberg tables.
> The current estimate doesn't take some things into account, e.g.
> * IcebergFileDescriptor sizes are larger than plain FileDescriptor sizes,
> especially when the table is partitioned
> * IcebergContentFileStore's data structures overhead
> * V3: Deletion Vectors stored in IcebergContentFileStore.dataFileToDV_
> * the internal Iceberg BaseTable object
> Also need to cross-check the estimation with the Jamm weigher (that we use in
> our caches).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
