[
https://issues.apache.org/jira/browse/IMPALA-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18087078#comment-18087078
]
ASF subversion and git services commented on IMPALA-14564:
----------------------------------------------------------
Commit f0f4fec03717c03cde782155641e46b9f18620b2 in impala's branch
refs/heads/master from Noemi Pap-Takacs
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f0f4fec03 ]
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
This change normalizes partition-related metadata out of the
file descriptors in the Catalog and the Frontend.
The partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we denormalize the file and partition metadata: we look up the
relevant partitions by their ids in the Planner, put them in the
file descriptors that are sent to the executors.
Metadata handling during scheduling and execution remains the same.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing:
- ran existing e2e tests
- extended FileMetadataLoaderTest to check the partitions
Assisted by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Reviewed-on: http://gerrit.cloudera.org:8080/23838
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Remove redundant partition information from Iceberg file descriptors
> ---------------------------------------------------------------------
>
> Key: IMPALA-14564
> URL: https://issues.apache.org/jira/browse/IMPALA-14564
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Noémi Pap-Takács
> Assignee: Noémi Pap-Takács
> Priority: Critical
> Labels: impala-iceberg
>
> File descriptors store the partition information (spec id and partition
> keys). Depending on the partitioning, partition keys can consist of many
> string fields corresponding to the partition values. Storing these keys
> redundantly for each file descriptor object adds a large overhead both to
> catalogd's memory and to the serialized data
> (TIcebergTable.TIcebergContentFileStore) that the Catalog sends to the
> Coordinator.
> Removing the partition info from file descriptors could significantly reduce
> their size.
> The partition keys could be stored in a map (id - partition info) that gets
> sent along with the file descriptors and the values could be looked up using
> an id for each partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
