Hello Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23946
to look at the new patch set (#3).
Change subject: IMPALA-14623: Optimize memory usage for Iceberg file path hashes
......................................................................
IMPALA-14623: Optimize memory usage for Iceberg file path hashes
Use THash128 Thrift struct (16 bytes) instead of String (64 bytes)
for storing 128-bit XXH128 hashes of Iceberg file paths,
achieving 4x memory reduction.
Key Changes:
- Added THash128 Thrift struct with two i64 fields (high/low)
- Updated TIcebergContentFileStore to use THash128 as map keys
- Created Hash128 Java class with Thrift serialization support
- Migrated from Murmur3 to XXH128 for better performance
- Added C++ comparison operators for THash128
Testing:
- Added comprehensive JUnit tests for Hash128 class
Change-Id: Ie0de793de2434dae3b60c3aa4f87dba203eee3c1
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog.cc
M be/src/rpc/hs2-http-test.cc
M be/src/runtime/descriptors.cc
M be/src/service/frontend.cc
M be/src/util/container-util.h
A be/src/util/thash128-util.h
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
A fe/src/main/java/org/apache/impala/util/Hash128.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
A fe/src/test/java/org/apache/impala/util/Hash128Test.java
M fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java
17 files changed, 364 insertions(+), 33 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/23946/3
--
To view, visit http://gerrit.cloudera.org:8080/23946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0de793de2434dae3b60c3aa4f87dba203eee3c1
Gerrit-Change-Number: 23946
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
