[
https://issues.apache.org/jira/browse/IMPALA-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109782#comment-18109782
]
Arnab Karmakar edited comment on IMPALA-13449 at 8/31/26 10:21 AM:
-------------------------------------------------------------------
Assigning this ticket to myself and I have a few open questions before starting
working on it:
1. Should we use different memory-estimation constants per table type? For
example, Iceberg FDs are larger than Hive FDs, as they carry extra
FbFileMetadata on top of base FileDescriptor. Using the same 500 bytes/file as
Hive likely underestimates the mem footprint.
Iceberg has structures Hive does not. Delete files, deletion vectors, partition
stats add memory that the current Iceberg formula does not account and separate
constants make it easier to model these.
2. If yes, how should we pick the new constant values?
3. Which components should the Iceberg estimate include? Currently we only
count files and blocks. We could add:
a. Iceberg Partitions
b. Data + delete files
c. Deletion Vectors
d. partition stats, column/puffin stats
Would really appreciate some pointers regarding these and if there is anything
else that needs to be taken into account.
Thanks
cc: [~arawat]
was (Author: JIRAUSER311137):
Assigning this ticket to myself and I have a few open questions before starting
working on it:
1. Should we use different memory-estimation constants per table type? For
example, Iceberg FDs are larger than Hive FDs, as they carry extra
FbFileMetadata on top of base FileDescriptor. Using the same 500 bytes/file as
Hive likely underestimates the mem footprint.
Iceberg has structures Hive does not. Delete files, deletion vectors, partition
stats add memory that the current Iceberg formula does not account and separate
constants make it easier to model these.
2. If yes, how should we pick the new constant values?
3. Which components should the Iceberg estimate include? Currently we only
count files and blocks. We could add:
a. Iceberg Partitions
b. Data + delete files
c. Deletion Vectors
d. partition stats, column/puffin stats
Would really appreciate some pointers regarding these and if there is anything
else that needs to be taken into account.
Thanks
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-13449
> URL: https://issues.apache.org/jira/browse/IMPALA-13449
> Project: IMPALA
> Issue Type: New Feature
> Components: Catalog
> Reporter: Manish Maheshwari
> Assignee: Arnab Karmakar
> Priority: Critical
>
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables. The current size shows tables extremely small and the logic needs to
> be refreshed for Iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
