[
https://issues.apache.org/jira/browse/IMPALA-14594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091046#comment-18091046
]
ASF subversion and git services commented on IMPALA-14594:
----------------------------------------------------------
Commit d575796624f9fa14de00d39ea2ee13ccbad153e6 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d57579662 ]
IMPALA-14594: Scale grouping aggregator's input cardinality by the number of
backends
The grouping aggregator uses input cardinality in its decision
about whether to continue expanding for a pre-aggregation.
Currently, each fragment instance is using the full input
cardinality for this calculation, but it will only be processing
a fraction of the input. This makes pre-aggregations very aggressive
about expanding, because they assume very high levels of future
reduction.
This fixes the input cardinality calculation to scale it down
by the number of fragment instances for this fragment. Each
fragment instance now assumes it is processing 1/Nth of the
input cardinality (where N is the number of fragment instances).
This will make the pre-aggregation's expansion calculation
more accurate.
Testing:
- Ran a core job
Change-Id: Iaf66709f12ea9b7328ca4e4c5cd45a05c9dc3032
Reviewed-on: http://gerrit.cloudera.org:8080/23732
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Grouping pre-agg's estimated input cardinality is not scaled by the number of
> threads
> -------------------------------------------------------------------------------------
>
> Key: IMPALA-14594
> URL: https://issues.apache.org/jira/browse/IMPALA-14594
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> Grouping pre-aggs decide whether to keep expanding based on whether the
> aggregation is achieving reduction. It has two measurements: Current
> reduction is (# input rows - # rows returned passthrough) / # hash table
> entries. Predicted total reduction is given by this formula (taken from
> [be/src/exec/grouping-aggregator.cc|https://github.com/apache/impala/blob/4be5fd8896dcd445a6379bdcda4bdcf318f24511/be/src/exec/grouping-aggregator.cc#L376-L380]):
> Extrapolate the current reduction factor (r) using the formula R = 1 + (N/n)
> * (r - 1), where R is the reduction factor over the full input data set, N is
> the number of input rows, excluding passed-through rows, and n is the number
> of rows inserted or merged into the hash tables. This is a very rough
> approximation but is good enough to be useful.
> The predicted total reduction formula is sensitive to the estimated input
> cardinality. A fragment instance is only going to be processing a fraction of
> the input rows, so the estimated input cardinality should be scaled down to
> the number of input rows expected to be processed by this fragment instance
> (i.e. the total value should be divided by the number of fragment instances).
> That is currently not happening, so the estimated reduction is significantly
> higher than it should be. This causes pre-aggs to expand more aggressively.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
