[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071981#comment-16071981 ]
Hive QA commented on HIVE-14797: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12833869/HIVE-14797.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10830 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed] (batchId=237) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1] (batchId=237) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (batchId=140) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=145) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=232) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=232) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=177) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=177) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=177) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5868/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5868/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5868/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12833869 - PreCommit-HIVE-Build > reducer number estimating may lead to data skew > ----------------------------------------------- > > Key: HIVE-14797 > URL: https://issues.apache.org/jira/browse/HIVE-14797 > Project: Hive > Issue Type: Improvement > Components: Query Processor > Reporter: roncenzhao > Assignee: roncenzhao > Labels: breaking_change > Attachments: HIVE-14797.2.patch, HIVE-14797.3.patch, > HIVE-14797.4.patch, HIVE-14797.patch > > > HiveKey's hash code is generated by multipling by 31 key by key which is > implemented in method `ObjectInspectorUtils.getBucketHashCode()`: > for (int i = 0; i < bucketFields.length; i++) { > int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], > bucketFieldInspectors[i]); > hashCode = 31 * hashCode + fieldHash; > } > The follow example will lead to data skew: > I hava two table called tbl1 and tbl2 and they have the same column: a int, b > string. The values of column 'a' in both two tables are not skew, but values > of column 'b' in both two tables are skew. > When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and > tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data > skew. > As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. > When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the > result, the job will be skew. -- This message was sent by Atlassian JIRA (v6.4.14#64029)