[
https://issues.apache.org/jira/browse/DRILL-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889340#comment-15889340
]
ASF GitHub Bot commented on DRILL-5293:
---------------------------------------
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/765#discussion_r103597790
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashPrelUtil.java
---
@@ -89,14 +91,15 @@ public LogicalExpression createCall(String funcName,
List<LogicalExpression> inp
*/
public static <T> T createHashExpression(
List<T> inputExprs,
+ T seed,
HashExpressionCreatorHelper<T> helper,
boolean hashAsDouble) {
assert inputExprs.size() > 0;
final String functionName = hashAsDouble ? HASH32_DOUBLE_FUNCTION_NAME
: HASH32_FUNCTION_NAME;
- T func = helper.createCall(functionName,
ImmutableList.of(inputExprs.get(0)));
+ T func = helper.createCall(functionName,
ImmutableList.of(inputExprs.get(0), seed ));
--- End diff --
Could you confirm if all the function names in function template and data
types that are covered in the Hash32Functions are also present in the
Hash32FunctionsWithSeed ?
> Poor performance of Hash Table due to same hash value as distribution below
> ---------------------------------------------------------------------------
>
> Key: DRILL-5293
> URL: https://issues.apache.org/jira/browse/DRILL-5293
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
> Affects Versions: 1.8.0
> Reporter: Boaz Ben-Zvi
> Assignee: Boaz Ben-Zvi
>
> The computation of the hash value is basically the same whether for the Hash
> Table (used by Hash Agg, and Hash Join), or for distribution of rows at the
> exchange. As a result, a specific Hash Table (in a parallel minor fragment)
> gets only rows "filtered out" by the partition below ("upstream"), so the
> pattern of this filtering leads to a non uniform usage of the hash buckets in
> the table.
> Here is a simplified example: An exchange partitions into TWO (minor
> fragments), each running a Hash Agg. So the partition sends rows of EVEN hash
> values to the first, and rows of ODD hash values to the second. Now the first
> recomputes the _same_ hash value for its Hash table -- and only the even
> buckets get used !! (Or with a partition into EIGHT -- possibly only one
> eighth of the buckets would be used !! )
> This would lead to longer hash chains and thus a _poor performance_ !
> A possible solution -- add a distribution function distFunc (only for
> partitioning) that takes the hash value and "scrambles" it so that the
> entropy in all the bits effects the low bits of the output. This function
> should be applied (in HashPrelUtil) over the generated code that produces the
> hash value, like:
> distFunc( hash32(field1, hash32(field2, hash32(field3, 0))) );
> Tested with a huge hash aggregate (64 M rows) and a parallelism of 8 (
> planner.width.max_per_node = 8 ); minor fragments 0 and 4 used only 1/8 of
> their buckets, the others used 1/4 of their buckets. Maybe the reason for
> this variance is that distribution is using "hash32AsDouble" and hash agg is
> using "hash32".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
