[
https://issues.apache.org/jira/browse/HIVE-30125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Bereznyakov reassigned HIVE-30125:
---------------------------------------------
Assignee: Konstantin Bereznyakov
> Log missing column stats at WARN instead of ERROR in RelOptHiveTable
> --------------------------------------------------------------------
>
> Key: HIVE-30125
> URL: https://issues.apache.org/jira/browse/HIVE-30125
> Project: Hive
> Issue Type: Improvement
> Components: CBO
> Reporter: Konstantin Bereznyakov
> Assignee: Konstantin Bereznyakov
> Priority: Minor
>
> When CBO needs column statistics that are missing,
> {{RelOptHiveTable.updateColStats}} logs {{{}No Stats for <db>@<table>,
> Columns: <cols>{}}}. Whether this is a WARN or an ERROR depends on the
> {{allowMissingStats}} flag:
> {code:java}
> if (allowMissingStats) {
> LOG.warn(logMsg);
> ...
> } else {
> LOG.error(logMsg);
> throw new RuntimeException(logMsg);
> }
> {code}
> {{allowMissingStats}} is false when {{{}hive.stats.estimate=false{}}}, and it
> is always false in {{HiveReduceExpressionsWithStatsRule}} (enabled by
> {{{}hive.optimize.filter.stats.reduction{}}}). In these cases, the same
> missing stats that are normally a WARN are logged as an ERROR.
> The thrown exception is the real signal, and the callers already log the
> result at the right level:
> * {{CalcitePlanner}} (join reordering), {{HiveAggregateJoinTransposeRule}}
> and {{HiveFilterSortPredicates}} catch the exception, log a WARN ({{{}Missing
> column stats (see previous messages), skipping ...{}}}) and continue planning.
> * If the exception is not caught, {{CalcitePlanner}} logs {{CBO failed,
> skipping CBO.}} at ERROR with the exception, which includes the same table
> and column names. Depending on {{{}hive.cbo.fallback.strategy{}}}, the query
> then fails or is retried without CBO.
> So this ERROR never tells us anything new. If planning recovers, a normal
> WARN-level situation shows up as an ERROR. If planning fails, it repeats the
> ERROR that {{CalcitePlanner}} already logs. Tools that alert on ERROR logs
> then report normal, recovered planning as a failure. We see a large number of
> these in a deployment that runs with {{{}hive.stats.estimate=false{}}}.
> History: the ERROR and the throw come from the original CBO work (HIVE-7395,
> HIVE-7918), when missing stats always stopped CBO. HIVE-10315 split the code
> into a WARN branch and a strict branch. Later, HIVE-12017 and HIVE-22163
> added callers that catch the exception and continue planning. The log level
> in the strict branch was not changed when the exception became recoverable.
> *Proposed change:*
> * {{{}RelOptHiveTable.updateColStats{}}}: log the message at WARN in both
> branches. The exception and the control flow stay the same.
> * {{{}CalcitePlanner{}}}: change {{(see previous errors)}} to {{(see
> previous messages)}} in the missing-stats CBO failure message, to match the
> other three call sites.
> There is no behavior change. The same exception is thrown and handled in the
> same way, and the message with the table and column names is still logged at
> WARN.
> Checked against master at 744271738e.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
