Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23946 )
Change subject: IMPALA-14623: Optimize memory usage for Iceberg file path hashes ...................................................................... IMPALA-14623: Optimize memory usage for Iceberg file path hashes Use THash128 Thrift struct (16 bytes) instead of String (64 bytes) for storing 128-bit XXH128 hashes of Iceberg file paths, achieving 4x memory reduction. Key Changes: - Added THash128 Thrift struct with two i64 fields (high/low) - Updated TIcebergContentFileStore to use THash128 as map keys - Created Hash128 Java class with Thrift serialization support - Migrated from Murmur3 to XXH128 for better performance - Added C++ comparison operators for THash128 Testing: - Added comprehensive JUnit tests for Hash128 class Change-Id: Ie0de793de2434dae3b60c3aa4f87dba203eee3c1 Reviewed-on: http://gerrit.cloudera.org:8080/23946 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Zoltan Borok-Nagy <[email protected]> --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog.cc M be/src/rpc/hs2-http-test.cc M be/src/runtime/descriptors.cc M be/src/service/frontend.cc M be/src/util/container-util.h A be/src/util/thash128-util.h M common/thrift/CatalogObjects.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java A fe/src/main/java/org/apache/impala/util/Hash128.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java A fe/src/test/java/org/apache/impala/util/Hash128Test.java M fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java 17 files changed, 364 insertions(+), 33 deletions(-) Approvals: Impala Public Jenkins: Verified Zoltan Borok-Nagy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/23946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie0de793de2434dae3b60c3aa4f87dba203eee3c1 Gerrit-Change-Number: 23946 Gerrit-PatchSet: 4 Gerrit-Owner: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
