[
https://issues.apache.org/jira/browse/HIVE-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680783#comment-14680783
]
Yongzhi Chen commented on HIVE-11502:
-------------------------------------
[~gopalv], I have confirmed that HIVE-7041 caused the regression. Because the
hadoop bug is there for a long time, after hive switch to use hadoop's
hashcode, we got hadoop's bug. Thanks for find the root cause by pointing the
hadoop bug.
After I add code in
serde/src/java/org/apache/hadoop/hive/serde2/io/DoubleWritable.java
{noformat}
@Override
public int hashCode() {
long v = Double.doubleToLongBits(super.get());
return (int) (v ^ (v >>> 32));
}
{noformat}
The group by query can finish in 15 seconds.
So next step is, how do we fix the issue now?
> Map side aggregation is extremely slow
> --------------------------------------
>
> Key: HIVE-11502
> URL: https://issues.apache.org/jira/browse/HIVE-11502
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Physical Optimizer
> Affects Versions: 1.2.0
> Reporter: Yongzhi Chen
> Assignee: Yongzhi Chen
>
> For the query as following:
> {noformat}
> create table tbl2 as
> select col1, max(col2) as col2
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000) and
> it is in type double, the map side aggregation is very slow. I ran the query
> which took more than 3 hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set hive.map.aggr = false;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
