Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24512
to look at the new patch set (#2).
Change subject: POC/slop: decrease per file memory usage in catalog
......................................................................
POC/slop: decrease per file memory usage in catalog
This patch contains several experimantal changes, some
may be too drastic to merge. Changes some flatbuffer
definitions.
Tested with Iceberg table with ~1M files, 25K partititons.
mem need on coordiantor decraseed around 21%:
665MB->520MB, meaaning ~140 byte per file
Main changes with estimated saved bytes:
- FbFileDesc:
- added optimized block info for the common case of
1 block and <=3 replicas (~40 byte)
- merged booleans to a flag (~2 byte)
- merged relative_path and absolute_path (~2 byte)
- rearranged members to avoid padding (~8 byte?)
- IcebergContentFileStore:
- removed EncodedFileDescriptor and store desc +
metadata as a single array (~48 byte)
- reordered FbIcebergMetadata to avoid padding (~8 byte)
- FbFileMetadata removed (~14 byte)
40+2+2+8+48+8+14=122
This is not too far from the measured 140.
Testing:
- ran Iceberg tests
Change-Id: Id48c7785ec88e16a9b79ac910add2c0a240869e3
Assisted-by: Claude Sonnet 4.6
---
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler.cc
M common/fbs/CatalogObjects.fbs
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/FileBlock.java
M fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java
M
fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsFileMetadataTest.java
16 files changed, 560 insertions(+), 245 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/24512/2
--
To view, visit http://gerrit.cloudera.org:8080/24512
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id48c7785ec88e16a9b79ac910add2c0a240869e3
Gerrit-Change-Number: 24512
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
