Xianqing He created IMPALA-15542:
------------------------------------
Summary: Avoid unnecessary Iceberg manifest loading in the
count(*) optimization
Key: IMPALA-15542
URL: https://issues.apache.org/jira/browse/IMPALA-15542
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 5.0.0
Reporter: Xianqing He
h3. Problem
{code:java}
SelectStmt.optimizePlainCountStarQueryForIcebergTable()
{code}
first calls ''Utils.hasDeleteFiles()'' and only afterwards validates the select
list. For time travel queries ({code}FOR SYSTEM_TIME/VERSION AS OF{code}),
''hasDeleteFiles()'' loads the Iceberg manifest files via
''IcebergUtil.getIcebergFiles()'', which can be expensive, but the result is
only needed if the query is eligible for the count(*) optimization.
In addition, for eligible time travel queries on V2 tables the manifest files
are loaded twice: once by ''hasDeleteFiles()'' to select the V1/V2 code path,
and once by ''getRecordCountV2()'' to compute the record count of the data
files without deletes.
h3. Fix
* Validate the select list before ''hasDeleteFiles()'' is called, so ineligible
queries (no count(*) or non-aggregate non-constant select list items, and other
aggregate functions in the V2 case) return without loading any Iceberg
metadata. The validation handles ''IcebergV2CountStarAccumulator'' items of
re-analyzed statements (''alreadyOptimized'').
* For time travel queries load the Iceberg content files only once
(''Utils.getGroupedContentFiles()'') and reuse the result both for the delete
file check and for the V2 record count.
* V1/V2 semantics are unchanged: V1 still allows aggregate functions other than
count(*) but does not rewrite count(*); V2 still only allows count(*),
constants and the accumulator.
h3. Testing
* Extended ''PlannerTest/iceberg-v2-tables.test'' with boundary cases: time
travel queries with an ineligible select list and with other aggregate
functions (V1 and V2 snapshots), plus current-snapshot 'SELECT 1' and
'count(*), sum(i)' cases.
* Ran ''PlannerTest#testIcebergV2TableScans'',
''testIcebergV2TableScansWithResources'', ''testIcebergV2TableScansHashJoin''
and ''AnalyzeStmtsTest#testIcebergTimeTravel''; all pass.
* The new tests also pass on the unmodified code, confirming the change is
behavior-preserving.
h3. Note
Queries without count(*) on V2 tables with delete files were already fixed to
return correct results by IMPALA-14993. This Jira covers the remaining
performance issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
