Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24512
to look at the new patch set (#4).
Change subject: POC/slop: decrease per file memory usage in catalog
......................................................................
POC/slop: decrease per file memory usage in catalog
This patch contains several experimantal changes, some
may be too drastic to merge. Changes some flatbuffer
definitions.
Tested with Iceberg table with ~1M files, 25K partititons.
mem need on coordiantor decraseed around 55%:
665MB->300MB
The most important change is avoiding referencing the
original buffer from Thrift in
IcebergContentFileStore.partition_list_, which almost
doubled the mem usage.
Other changes that save areound 140 byte per file
based on measurements:
Main changes with estimated saved bytes:
- FbFileDesc:
- added optimized block info for the common case of
1 block and <=3 replicas (~40 byte)
- merged booleans to a flag (~2 byte)
- merged relative_path and absolute_path (~2 byte)
- rearranged members to avoid padding (~8 byte?)
- IcebergContentFileStore:
- removed EncodedFileDescriptor and store desc +
metadata as a single array (~48 byte)
- reordered FbIcebergMetadata to avoid padding (~8 byte)
- FbFileMetadata removed (~14 byte)
40+2+2+8+48+8+14=122
This is not too far from the measued 140
Testing:
- ran Iceberg tests
Change-Id: Id48c7785ec88e16a9b79ac910add2c0a240869e3
Assisted-by: Claude Sonnet 4.6
---
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler.cc
M common/fbs/CatalogObjects.fbs
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/FileBlock.java
M fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java
M
fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsFileMetadataTest.java
16 files changed, 596 insertions(+), 246 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/24512/4
--
To view, visit http://gerrit.cloudera.org:8080/24512
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id48c7785ec88e16a9b79ac910add2c0a240869e3
Gerrit-Change-Number: 24512
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
