[
https://issues.apache.org/jira/browse/IMPALA-15394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15394:
--------------------------------------
Description:
An HdfsScanNode with the count\(*) optimization returns one row per file or row
group, so its cardinality is the file count.
HdfsScanNode.generateHboKeyString() does not include countStarSlot_, so HBO
stores this run under the same key as a regular scan of the same table and
conjuncts. The next regular scan takes the file count as its cardinality.
Seen on a Parquet table with 15 files and 13.66M rows: after {{SELECT count\(*)
FROM t}}, {{SELECT * FROM t}} is planned with {{cardinality=15 (from HBO)}}.
The memory estimates above the scan follow it, and so does the per-query memory
limit when admission control derives it from the estimate. A regular scan that
runs later fixes the record, because the last run wins.
Reproduce: on a Parquet table with more than one file and no HBO record, run
{{SELECT count\(*) FROM t}}, then {{EXPLAIN SELECT * FROM t}}.
was:
An HdfsScanNode with the count(*) optimization returns one row per file or row
group, so its cardinality is the file count.
HdfsScanNode.generateHboKeyString() does not include countStarSlot_, so HBO
stores this run under the same key as a regular scan of the same table and
conjuncts. The next regular scan takes the file count as its cardinality.
Seen on a Parquet table with 15 files and 13.66M rows: after {{SELECT count(*)
FROM t}}, {{SELECT * FROM t}} is planned with {{cardinality=15 (from HBO)}}.
The memory estimates above the scan follow it, and so does the per-query memory
limit when admission control derives it from the estimate. A regular scan that
runs later fixes the record, because the last run wins.
Reproduce: on a Parquet table with more than one file and no HBO record, run
{{SELECT count(*) FROM t}}, then {{EXPLAIN SELECT * FROM t}}.
> HBO stores count(*) scans under the key of a regular scan
> ---------------------------------------------------------
>
> Key: IMPALA-15394
> URL: https://issues.apache.org/jira/browse/IMPALA-15394
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Aleksandr Efimov
> Priority: Major
>
> An HdfsScanNode with the count\(*) optimization returns one row per file or
> row group, so its cardinality is the file count.
> HdfsScanNode.generateHboKeyString() does not include countStarSlot_, so HBO
> stores this run under the same key as a regular scan of the same table and
> conjuncts. The next regular scan takes the file count as its cardinality.
> Seen on a Parquet table with 15 files and 13.66M rows: after {{SELECT
> count\(*) FROM t}}, {{SELECT * FROM t}} is planned with {{cardinality=15
> (from HBO)}}. The memory estimates above the scan follow it, and so does the
> per-query memory limit when admission control derives it from the estimate. A
> regular scan that runs later fixes the record, because the last run wins.
> Reproduce: on a Parquet table with more than one file and no HBO record, run
> {{SELECT count\(*) FROM t}}, then {{EXPLAIN SELECT * FROM t}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
