[
https://issues.apache.org/jira/browse/IMPALA-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110468#comment-18110468
]
ASF subversion and git services commented on IMPALA-15278:
----------------------------------------------------------
Commit 7a3cb82b49d845bc5ef597e598f589593dd13549 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7a3cb82b4 ]
IMPALA-15278: Fix incorrect cardinality with OFFSET
OFFSET is used to skip a number of rows before returning results. When
calculating cardinality of a node that has OFFSET, the cardinality
should substract this number before applying the LIMIT.
MERGING-EXCHANGE currently does the reverse which could results in 0
cardinality when OFFSET >= LIMIT. SortNode currently ignores the offset.
This fixes both issue by adding an overload of capCardinalityAtLimit()
that has a parameter for offset. It's used in ExchangeNode and SortNode.
Tests
- Added FE test
- Ran PlannerTest, TpcdsCpuCostPlannerTest and
TpcdsTupleCachePlannerTest locally.
Change-Id: I3b628beabc5c7ec6c4fdda9dff6aaf7a4acae538
Reviewed-on: http://gerrit.cloudera.org:8080/24718
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Incorrect cardinality with OFFSET in MERGING-EXCHANGE and TOP-N
> ---------------------------------------------------------------
>
> Key: IMPALA-15278
> URL: https://issues.apache.org/jira/browse/IMPALA-15278
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> In ExchangeNode.computeStats(), the cardinality is first capped by the limit
> and then decreased by offset:
> {code:java}
> cardinality_ = capCardinalityAtLimit(children_.get(0).getCardinality());
> // Apply the offset correction if there's a valid cardinality
> if (cardinality_ > -1) cardinality_ = Math.max(0, cardinality_ - offset_);
> {code}
> https://github.com/apache/impala/blob/c47f8cce53e3262bde67f881f9591be93b831fe2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java#L155-L157
> This is incorrect. Cardinality should subtract the offset first and then cap
> at the limit.
> Take the following query as an example:
> {code:sql}
> select id, int_col from functional.alltypes order by id limit 1000 offset
> 6200{code}
> It returns 1000 rows but the estimated cardinality is 0 due to this bug:
> {noformat}
> 02:MERGING-EXCHANGE [UNPARTITIONED]
> | offset: 6200
> | order by: id ASC
> | limit: 1000
> | mem-estimate=36.00KB mem-reservation=0B thread-reservation=0
> | tuple-ids=1 row-size=8B cardinality=0
> | in pipelines: 01(GETNEXT){noformat}
> TOP-N SortNode has another issue that the offset is not used in computing
> cardinality.
> {code:java}
> if (isTypeTopN() && includeTies_) {
> cardinality_ =
> MathUtil.smallestValidCardinality(getChild(0).cardinality_,
> limitWithTies_);
> } else {
> cardinality_ = capCardinalityAtLimit(getChild(0).cardinality_);
> }{code}
> https://github.com/apache/impala/blob/c47f8cce53e3262bde67f881f9591be93b831fe2/fe/src/main/java/org/apache/impala/planner/SortNode.java#L306-L311
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
