[
https://issues.apache.org/jira/browse/FLINK-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu resolved FLINK-11964.
-----------------------------
Resolution: Fixed
1.11.0: 69ed6feef09d36df48b2e849888f9faebdaa2981
1.10.0: 81b18957da8e35b414b6c6017d13720157340d59
> Avoid hash collision of partition and bucket in HybridHashTable in Blink SQL
> ----------------------------------------------------------------------------
>
> Key: FLINK-11964
> URL: https://issues.apache.org/jira/browse/FLINK-11964
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
> Reporter: Jingsong Lee
> Assignee: Jingsong Lee
> Priority: Major
> Fix For: 1.10.0
>
>
> In HybridHashTable, first select the corresponding partition according to
> hashCode, and then select the bucket in the partition according to hashCode,
> using the same hashCode can easily cause hash collision.
> Consider doing some mix to hashCode when choosing bucket.
> Like JDK HashMap, we can just XOR some shifted bits in the cheapest possible
> way to reduce systematic lossage, as well as to incorporate impact of the
> highest bits that would otherwise never be used in index calculations because
> of table bounds. (bucket use power-of-two masking). Just like: (hash ^
> (hash >>> 16))
> In some cases, if a lot of conflicts occurred, this will lead to job hang,
> because hash join will degenerate to nested loop join.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
