[
https://issues.apache.org/jira/browse/FLINK-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812085#comment-16812085
]
Liya Fan commented on FLINK-11964:
----------------------------------
Hi [~lzljs3620320], this is a good point, because the selection of the hash
function has great impact on the hash bucket collision, and hence the E2E query
performance. However, selecting a proper hash function is tricky, because if
the hash function is overly computing-intensive, the computing overhead will
outweigh the reduction in hash collision.
I am interested in this topic. Would you please assign this issue to me? Maybe
I can do some systematic research :)
> Avoid hash collision of partition and bucket in HybridHashTable in Blink SQL
> ----------------------------------------------------------------------------
>
> Key: FLINK-11964
> URL: https://issues.apache.org/jira/browse/FLINK-11964
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Operators
> Reporter: Jingsong Lee
> Priority: Major
>
> In HybridHashTable, first select the corresponding partition according to
> hashCode, and then select the bucket in the partition according to hashCode,
> using the same hashCode can easily cause hash collision.
> Consider doing some mix to hashCode when choosing bucket.
> Like JDK HashMap, we can just XOR some shifted bits in the cheapest possible
> way to reduce systematic lossage, as well as to incorporate impact of the
> highest bits that would otherwise never be used in index calculations because
> of table bounds. (bucket use power-of-two masking). Just like: (hash ^
> (hash >>> 16))
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
