[
https://issues.apache.org/jira/browse/IMPALA-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18057328#comment-18057328
]
ASF subversion and git services commented on IMPALA-14623:
----------------------------------------------------------
Commit 49196804f8e74db346001d951b44777d6e0b2bfe in impala's branch
refs/heads/master from Arnab Karmakar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=49196804f ]
IMPALA-14623: Optimize memory usage for Iceberg file path hashes
Use THash128 Thrift struct (16 bytes) instead of String (64 bytes)
for storing 128-bit XXH128 hashes of Iceberg file paths,
achieving 4x memory reduction.
Key Changes:
- Added THash128 Thrift struct with two i64 fields (high/low)
- Updated TIcebergContentFileStore to use THash128 as map keys
- Created Hash128 Java class with Thrift serialization support
- Migrated from Murmur3 to XXH128 for better performance
- Added C++ comparison operators for THash128
Testing:
- Added comprehensive JUnit tests for Hash128 class
Change-Id: Ie0de793de2434dae3b60c3aa4f87dba203eee3c1
Reviewed-on: http://gerrit.cloudera.org:8080/23946
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Zoltan Borok-Nagy <[email protected]>
> Use the raw bytes of the 128-bit Murmur hash of Iceberg file paths
> ------------------------------------------------------------------
>
> Key: IMPALA-14623
> URL: https://issues.apache.org/jira/browse/IMPALA-14623
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Arnab Karmakar
> Priority: Major
> Labels: impala-iceberg
>
> Currently we use the following method to store the hash of an Iceberg file
> path:
>
> In IcebergUtil:
> {noformat}
> public static String getFilePathHash(String path) {
> Hasher hasher = Hashing.murmur3_128().newHasher();
> hasher.putUnencodedChars(path);
> return hasher.hash().toString();
> }{noformat}
> There are 16 raw bytes, but the String representation stores it on 2 * 16 =
> 32 characters. And a character in java is 2 bytes. So it consumes 2 * 32 = 64
> bytes which is 4 times more than needed.
> For tables with large number of files this can cause a significant overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
