Ezra Zerihun created IMPALA-15717:
-------------------------------------
Summary: Optimize Impala Incremental Stats estimate gathering
Key: IMPALA-15717
URL: https://issues.apache.org/jira/browse/IMPALA-15717
Project: IMPALA
Issue Type: New Feature
Affects Versions: Impala 4.5.0
Reporter: Ezra Zerihun
This Feature Request is all about optimizing how Impala gathers incremental
stats. So that "COMPUTE INCREMENTAL STATS" estimate is not blocked by
--inc_stats_size_limit_bytes (200 MB default) when the actual size is less.
REQUESTS
# +Fix the estimate in ComputeStatsStmt.analyze() so it reflects what is
actually stored.+ Either count only the columns stats are gathered for (the
validated column list when one is given, otherwise the non-partition,
non-complex columns), or use the actual compressed size of existing partition
stats (HdfsPartition.getPartitionStatsCompressed()), plus an estimate for the
partitions being computed.
## The size check uses hdfsTable.getColumns().size(), which counts every
column of the table, partition columns included:
### long statsSizeEstimate = hdfsTable.getColumns().size() *
numOfAllIncStatsPartitions * HdfsTable.STATS_SIZE_PER_COLUMN_BYTES;
## REF:
###
[https://github.com/apache/impala/blob/4.5.0/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java#L463-L506]
###
[https://github.com/apache/impala/blob/4.5.0/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java#L496-L497]
# +Every run uses the same column list. Even without a column list,
getColumns() overcounts. Stats are never gathered for partition columns or for
complex-type columns, yet both are counted.+
##
[https://github.com/apache/impala/blob/4.5.0/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java#L241-L261]
##
[https://github.com/apache/impala/blob/4.5.0/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java#L960-L963]
*Real World Analysis from a failed compute incremental stats:*
a) Table Cols = 836 (3 partition columns + 37 complex cols + 796 stats eligible
cols). It also ignores the column list.
The partition count is the 1253 partitions that already have incremental stats,
plus the 2 partitions in your statement (1255)
=========================
836 × 1255 × 200 = 209,836,000 bytes (over the 209,715,200-byte limit)
=========================
b)
A code improvement that counts only the 796 eligible columns gives 796 × 1255 ×
200 = 199,796,000 bytes, which is just under the limit and would run.
c)
Or estimate using the actual stored size of the existing partitions' stats,
plus 2 × 61 × 200 bytes for the partitions being computed. With this approach,
the estimate no longer depends on how many columns each existing partition was
last computed with.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
