[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508789#comment-15508789 ]

## Advertising

Rui Li edited comment on HIVE-14797 at 9/21/16 5:15 AM: -------------------------------------------------------- Hmm random prime won't work because we need to make sure same rows always have same hash code. I can think of another way: {code} 1. If we have only one field, we can just return the field's hash code. 2. If we have multiple fields, we can compute hash code as: P1*hash(F1)+...+Pn*hash(Fn). Where hash(Fn) is the hash code of the nth field, and {P1,...,Pn} is a deterministic series of prime numbers, e.g. {17,19,...}. Seems {{BigInteger::nextProbablePrime()}} can help generate the series. {code} was (Author: lirui): Hmm random prime won't work because we need to make sure same rows always have same hash code. I can think of another way: 1. If we have only one field, we can just return the field's hash code. 2. If we have multiple fields, we can compute hash code as: P1*hash(F1)+...+Pn*hash(Fn). Where hash(Fn) is the hash code of the nth field, and {P1,...,Pn} is a deterministic series of prime numbers, e.g. {17,19,...}. Seems {{BigInteger::nextProbablePrime()}} can help generate the series. > reducer number estimating may lead to data skew > ----------------------------------------------- > > Key: HIVE-14797 > URL: https://issues.apache.org/jira/browse/HIVE-14797 > Project: Hive > Issue Type: Improvement > Components: Query Processor > Reporter: roncenzhao > Assignee: roncenzhao > Attachments: HIVE-14797.patch > > > HiveKey's hash code is generated by multipling by 31 key by key which is > implemented in method `ObjectInspectorUtils.getBucketHashCode()`: > for (int i = 0; i < bucketFields.length; i++) { > int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], > bucketFieldInspectors[i]); > hashCode = 31 * hashCode + fieldHash; > } > The follow example will lead to data skew: > I hava two table called tbl1 and tbl2 and they have the same column: a int, b > string. The values of column 'a' in both two tables are not skew, but values > of column 'b' in both two tables are skew. > When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and > tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data > skew. > As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. > When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the > result, the job will be skew. -- This message was sent by Atlassian JIRA (v6.3.4#6332)