Zoltán Borók-Nagy created IMPALA-15651:
------------------------------------------
Summary: Calcite: an exception in an optional rewrite stage
discards the whole Calcite plan
Key: IMPALA-15651
URL: https://issues.apache.org/jira/browse/IMPALA-15651
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Zoltán Borók-Nagy
An exception in any CalciteOptimizer.createPreImpalaConvertPlan() program fails
the whole Calcite plan (master 15cc045e2d).
Frontend.getTExecRequestWithFallback() then replans with the original planner.
Optional stages such as CTE substitution (runCTEProgram(), active with
CTE_THRESHOLD > 0) could be skipped safely, yet the query loses every Calcite
optimization. The profile records only the exception class (IMPALA-?).
Seen twice on TPC-DS:
* q78 with CTE_THRESHOLD=1 and the pending IMPALA-15489 change
(https://gerrit.cloudera.org/#/c/25030/ patch set 1): the CTE stage throws in
MaterializedViewJoinRule(Filter) ("Type mismatch: the field sizes are not
equal").
* q23_1 and q23_2: an in-development aggregate-join transpose program
(IMPALA-?) throws an AssertionError. At TPC-DS SF10000 on 12 executors (best of
3), a prototype that skips only the failed stage keeps the Calcite plan: q23_1
368 -> 108 s, q23_2 231 -> 91 s.
Skipping is not always faster than the original planner: q78's Calcite plan
without CTEs was 9 s slower (154 s vs 144 s).
*Proposal:*
* Wrap each optional stage: on an exception, log a WARN, record the stage and
message in the Frontend profile, and continue with its input plan.
* Drop cached metadata (RelOptCluster.invalidateMetadataQuery()) after a failed
stage.
* Rethrow OutOfMemoryError; essential stages (node expansion, coercion, final
convert programs) keep failing.
* Add a flag or query option to disable skipping in planner tests.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
