[
https://issues.apache.org/jira/browse/HIVE-14256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380402#comment-15380402
]
Gopal V commented on HIVE-14256:
--------------------------------
Both the bugs I filed today manage to miss the optimizations already built.
{code}
hive> set hive.optimize.distinct.rewrite;
hive.optimize.distinct.rewrite=true
hive> explain select sum(ss_net_profit), count(distinct ss_customer_sk) from
store_sales;
OK
Plan optimized by CBO.
Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 2 llap
File Output Operator [FS_6]
Group By Operator [GBY_4] (rows=1 width=24)
Output:["_col0","_col1"],aggregations:["sum(VALUE._col0)","count(DISTINCT
KEY._col0:0._col0)"]
<-Map 1 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_9]
Group By Operator [GBY_8] (rows=547946325 width=168)
Output:["_col0","_col1","_col2"],aggregations:["sum(ss_net_profit)","count(DISTINCT
ss_customer_sk)"],keys:ss_customer_sk
Select Operator [SEL_7] (rows=547946325 width=168)
Output:["ss_customer_sk","ss_net_profit"]
TableScan [TS_0] (rows=547946325 width=168)
tpcds_bin_partitioned_orc_200@store_sales,store_sales,Tbl:PARTIAL,Col:NONE,Output:["ss_customer_sk","ss_net_profit"]
Time taken: 3.32 seconds, Fetched: 22 row(s)
hive>
{code}
Single shuffle, no map-side reduction because of the distinct.
> CBO: Rewrite aggregate + distinct as 3-stage DAG
> ------------------------------------------------
>
> Key: HIVE-14256
> URL: https://issues.apache.org/jira/browse/HIVE-14256
> Project: Hive
> Issue Type: Improvement
> Components: CBO
> Affects Versions: 2.2.0
> Reporter: Gopal V
>
> {code}
> select sum(ss_net_profit), count(distinct ss_customer_sk) from store_sales;
> {code}
> is very slow, while manually sub-aggregating this makes it much faster.
> {code}
> select sum(v), count(c) from
> ( select sum(ss_net_profit) as v, ss_customer_sk as k from store_sales group
> by ss_customer_sk);
> {code}
> Query28 in TPC-DS would be an example of whether this would be valuable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
