[
https://issues.apache.org/jira/browse/CALCITE-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128786#comment-18128786
]
Arnaud Jegou edited comment on CALCITE-5365 at 10/19/26 2:23 PM:
-----------------------------------------------------------------
_edit: reframed_
Splitting the condition makes sense to me. Part of the blowup comes from
something else though: above a union, lineage combines fields from different
union inputs, which can never happen (CALCITE-7869). Here only 3 x 3 = 9 of the
3^14 combinations are valid. Fixing that keeps the result small in most cases,
and splitting still helps on top of it when the conjuncts relate different
unions. A limit still makes sense as a safeguard for plans that really have
many valid combinations.
note: the query in the description uses left joins, and {{getAllPredicates}}
returns null for an outer join, so as written it does not reach
{{createAllPossibleExpressions}}. It does with inner joins and
{{getAllPredicates}} does build the 3^14 combinations
was (Author: JIRAUSER310786):
The query in the description uses left joins, and `getAllPredicates` returns
null for an outer join, in 1.30.0 as on main, so as written it does not reach
`createAllPossibleExpressions`. With inner joins it does: `getAllPredicates`
builds 3^14 combinations for the seven-key join condition.
Most of those combinations cannot occur, which is now tracked in CALCITE-7869:
above a Union, lineage combines fields from different inputs. A row of the join
pairs one input of t1 with one input of t2, so only 3 x 3 = 9 combinations are
valid. On main, the same shape with two keys instead of seven returns 81
predicates instead of 9.
Fixing CALCITE-7869 would also address the concern above that a limit loses
predicate information: the valid combinations would stay small, so a limit
would only apply to plans that really have many source combinations, such as
joins of several wide unions. Splitting the condition into conjuncts, as
suggested above, would also avoid the product here, and stays useful after
CALCITE-7869: when the conjuncts relate different unions, as in a chain of
joins over several unions, the count becomes a sum instead of a product. The
two changes complement each other: CALCITE-7869 removes the combinations that
cannot occur, including within a single expression, and splitting keeps the
valid ones small.
> RelMdExpressionLineage#createAllPossibleExpressions will create many RexCall
> lead to OOM
> ------------------------------------------------------------------------------------------
>
> Key: CALCITE-5365
> URL: https://issues.apache.org/jira/browse/CALCITE-5365
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.30.0
> Reporter: hqbhoho
> Priority: Major
>
> Execute query like:
> {code:java}
> select
> *
> from
> (
> select *
> from
> (
> select * from A1
> )
> union all
> (
> select * from A2
> )
> union all
> (
> select * from A3
> )
> ) t1
> left join
> (
> select *
> from
> (
> select * from B1
> )
> union all
> (
> select * from B2
> )
> union all
> (
> select * from B3
> )
> )t2
> on t1.a1 = t2.b1
> and t1.a2 = t2.b2
> and t1.a3 = t2.b3
> and t1.a4 = t2.b4
> and t1.a5 = t2.b5
> and t1.a6 = t2.b6
> and t1.a7 = t2.b7
> left join C1 name
> on t2.b8=name.c1 {code}
> I will get all Predicates via RelMetadataQuery.getAllPredicates()，but will
> occur oom.
> RelMdExpressionLineage#createAllPossibleExpressions will retrun 3^14 RexCall
> when t1 join t2 with 7 condition? (t1.a1 will have 3 replacement)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
