Konstantin Bereznyakov created HIVE-30125:
---------------------------------------------
Summary: Log missing column stats at WARN instead of ERROR in
RelOptHiveTable
Key: HIVE-30125
URL: https://issues.apache.org/jira/browse/HIVE-30125
Project: Hive
Issue Type: Improvement
Components: CBO
Reporter: Konstantin Bereznyakov
When CBO needs column statistics that are missing,
{{RelOptHiveTable.updateColStats}} logs {{{}No Stats for <db>@<table>, Columns:
<cols>{}}}. Whether this is a WARN or an ERROR depends on the
{{allowMissingStats}} flag:
{code:java}
if (allowMissingStats) {
LOG.warn(logMsg);
...
} else {
LOG.error(logMsg);
throw new RuntimeException(logMsg);
}
{code}
{{allowMissingStats}} is false when {{{}hive.stats.estimate=false{}}}, and it
is always false in {{HiveReduceExpressionsWithStatsRule}} (enabled by
{{{}hive.optimize.filter.stats.reduction{}}}). In these cases, the same missing
stats that are normally a WARN are logged as an ERROR.
The thrown exception is the real signal, and the callers already log the result
at the right level:
* {{CalcitePlanner}} (join reordering), {{HiveAggregateJoinTransposeRule}} and
{{HiveFilterSortPredicates}} catch the exception, log a WARN ({{{}Missing
column stats (see previous messages), skipping ...{}}}) and continue planning.
* If the exception is not caught, {{CalcitePlanner}} logs {{CBO failed,
skipping CBO.}} at ERROR with the exception, which includes the same table and
column names. Depending on {{{}hive.cbo.fallback.strategy{}}}, the query then
fails or is retried without CBO.
So this ERROR never tells us anything new. If planning recovers, a normal
WARN-level situation shows up as an ERROR. If planning fails, it repeats the
ERROR that {{CalcitePlanner}} already logs. Tools that alert on ERROR logs then
report normal, recovered planning as a failure. We see a large number of these
in a deployment that runs with {{{}hive.stats.estimate=false{}}}.
History: the ERROR and the throw come from the original CBO work (HIVE-7395,
HIVE-7918), when missing stats always stopped CBO. HIVE-10315 split the code
into a WARN branch and a strict branch. Later, HIVE-12017 and HIVE-22163 added
callers that catch the exception and continue planning. The log level in the
strict branch was not changed when the exception became recoverable.
*Proposed change:*
* {{{}RelOptHiveTable.updateColStats{}}}: log the message at WARN in both
branches. The exception and the control flow stay the same.
* {{{}CalcitePlanner{}}}: change {{(see previous errors)}} to {{(see previous
messages)}} in the missing-stats CBO failure message, to match the other three
call sites.
There is no behavior change. The same exception is thrown and handled in the
same way, and the message with the table and column names is still logged at
WARN.
Checked against master at 744271738e.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
