Quanlong Huang created IMPALA-15278:
---------------------------------------
Summary: Incorrect cardinality with OFFSET in MERGING-EXCHANGE and
TOP-N
Key: IMPALA-15278
URL: https://issues.apache.org/jira/browse/IMPALA-15278
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
In ExchangeNode.computeStats(), the cardinality is first capped by the limit
and then decreased by offset:
{code:java}
cardinality_ = capCardinalityAtLimit(children_.get(0).getCardinality());
// Apply the offset correction if there's a valid cardinality
if (cardinality_ > -1) cardinality_ = Math.max(0, cardinality_ - offset_);
{code}
https://github.com/apache/impala/blob/c47f8cce53e3262bde67f881f9591be93b831fe2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java#L155-L157
This is incorrect. Cardinality should subtract the offset first and then cap at
the limit.
Take the following query as an example:
{code:sql}
select id, int_col from functional.alltypes order by id limit 1000 offset
6200{code}
It returns 1000 rows but the estimated cardinality is 0 due to this bug:
{noformat}
02:MERGING-EXCHANGE [UNPARTITIONED]
| offset: 6200
| order by: id ASC
| limit: 1000
| mem-estimate=36.00KB mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=8B cardinality=0
| in pipelines: 01(GETNEXT){noformat}
TOP-N SortNode has another issue that the offset is not used in computing
cardinality.
{code:java}
if (isTypeTopN() && includeTies_) {
cardinality_ =
MathUtil.smallestValidCardinality(getChild(0).cardinality_,
limitWithTies_);
} else {
cardinality_ = capCardinalityAtLimit(getChild(0).cardinality_);
}{code}
https://github.com/apache/impala/blob/c47f8cce53e3262bde67f881f9591be93b831fe2/fe/src/main/java/org/apache/impala/planner/SortNode.java#L306-L311
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
