[
https://issues.apache.org/jira/browse/IMPALA-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18052893#comment-18052893
]
Zoltán Borók-Nagy commented on IMPALA-14623:
--------------------------------------------
I think we can win a lot by changing the keytype of
IcebergContentFileStore.MapListContainer's fileDescMap_, from String to a
custom class e.g. Hash128:
{noformat}
public class Hash128 {
private final long high_;
private final long low_;
public Hash128(long high, long low) {
high_ = high;
low_ = low;
}
public String toLatin1() { ... }
public static Hash128 fromLatin1(String data) { ... }
}{noformat}
TIcebergContentFileStore's fields can remain string to THdfsFileDesc maps, but
we could use Latin-1 (i.e. raw byte) encoding of the hash values.
> Use the raw bytes of the 128-bit Murmur hash of Iceberg file paths
> ------------------------------------------------------------------
>
> Key: IMPALA-14623
> URL: https://issues.apache.org/jira/browse/IMPALA-14623
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Currently we use the following method to store the hash of an Iceberg file
> path:
>
> In IcebergUtil:
> {noformat}
> public static String getFilePathHash(String path) {
> Hasher hasher = Hashing.murmur3_128().newHasher();
> hasher.putUnencodedChars(path);
> return hasher.hash().toString();
> }{noformat}
> There are 16 raw bytes, but the String representation stores it on 2 * 16 =
> 32 characters. And a character in java is 2 bytes. So it consumes 2 * 32 = 64
> bytes which is 4 times more than needed.
> For tables with large number of files this can cause a significant overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
