[
https://issues.apache.org/jira/browse/IMPALA-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-14623 started by Arnab Karmakar.
-----------------------------------------------
> Use the raw bytes of the 128-bit Murmur hash of Iceberg file paths
> ------------------------------------------------------------------
>
> Key: IMPALA-14623
> URL: https://issues.apache.org/jira/browse/IMPALA-14623
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Arnab Karmakar
> Priority: Major
> Labels: impala-iceberg
>
> Currently we use the following method to store the hash of an Iceberg file
> path:
>
> In IcebergUtil:
> {noformat}
> public static String getFilePathHash(String path) {
> Hasher hasher = Hashing.murmur3_128().newHasher();
> hasher.putUnencodedChars(path);
> return hasher.hash().toString();
> }{noformat}
> There are 16 raw bytes, but the String representation stores it on 2 * 16 =
> 32 characters. And a character in java is 2 bytes. So it consumes 2 * 32 = 64
> bytes which is 4 times more than needed.
> For tables with large number of files this can cause a significant overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
