[
https://issues.apache.org/jira/browse/HIVE-30261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-30261:
----------------------------------
Attachment: partial_aggr_join_transpose.q
partial_aggr_join_transpose.q.out
> Push a partial aggregation below an inner join
> ----------------------------------------------
>
> Key: HIVE-30261
> URL: https://issues.apache.org/jira/browse/HIVE-30261
> Project: Hive
> Issue Type: Sub-task
> Components: Physical Optimizer
> Reporter: Krisztian Kasa
> Assignee: Krisztian Kasa
> Priority: Major
> Labels: pull-request-available
> Attachments: partial_aggr_join_transpose.q,
> partial_aggr_join_transpose.q.out
>
>
> In a star join followed by a GROUP BY, the fact table rows go through the
> shuffle and the join before any aggregation happens. When the join keys and
> the GROUP BY keys have few distinct values on the fact side, the rows could
> be partially aggregated below the join first. Then far fewer rows are
> shuffled, joined and aggregated.
> h4. Criteria to meet for push-down:
> * The join is an inner join. Every input to the join is a ReduceSink
> operator, and for an n-way join, every condition is an inner join.
> * The GROUP BY is above the join with only Select operators between them. It
> has keys, so a full aggregate cannot be pushed down.
> * GROUP BYs with grouping sets and DISTINCT aggregations are not supported.
> * Only SUM, $SUM0, COUNT, MIN, and MAX aggregate functions are supported.
> These are the ones that support partial aggregation.
> * Every key and every argument is deterministic and references at most one
> join input.
> * Every aggregate function argument references the same join input, and the
> GROUP BY is pushed down to that input.
> * If no argument has columns (e.g., COUNT(\*)), the GROUP BY is pushed down
> to the larger join input.
> * The pushed GROUP BY keys are not empty, and there are no more of them than
> the original keys
> h4. Cost based criteria
> * Statistics are available: the row counts of the join and the input to
> which the GROUP BY is pushed down, and the column statistics of every pushed
> key. When no argument has columns (COUNT(\*)), the row counts of every join
> input are needed too.
> * The join outputs at most max.join.expansion × the rows of the input to
> which the GROUP BY is pushed down.
> * The join reduces the input to which the GROUP BY is pushed down by at most
> a factor of max.join.reduction. Hive has no dynamic filter on the probe side
> of a map join, so the pushed GROUP BY would also process every row the join
> discards.
> * The NDV of every pushed key is at most max.key.ndv.ratio × the pushed
> input rows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)