[
https://issues.apache.org/jira/browse/CALCITE-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035034#comment-17035034
]
Ravi Kapoor edited comment on CALCITE-3784 at 2/12/20 10:50 AM:
----------------------------------------------------------------
[~julianhyde] [~hyuan] It seems shocking to me as well unless I monitored the
memory consumption and the thread dump.
There is no multiple conditions as there is only one RexCall is created which
is used in filter expression but this RexCall further uses similar RexCalls
multiple times as an expression and that uses further RexCalls and son on upto
a depth of around 60. Actually I cant have a variable to hold the result but to
evaluate at runtime so I had to embed the repeated expression as I am creating
a sql query at the end to run it on BigQuery.
Let me show you how my RexCall looks like. Take a look at !OOM Screenshot.png!
The last variable StgValidationFlag is used in the filter which uses all the
previous evaluated RexCalls
blows out of memory as the string formation is too large to load even in the
IDE.
Moreover, these RexCalls are all if else conditions exprs.
Example:
Suppose there is an expression e1 which is used in e2 5 times and we have
expression e3 which uses e2 5 times and so on upto e12 as there are twelve
expressions.
So. Yes there are exponential factors involved
was (Author: ravi_kapoor):
[~julianhyde] [~hyuan] It seems shocking to me as well unless I monitored the
memory consumption and the thread dump.
There is no multiple conditions as there is only one RexCall is created which
is used in filter expression but this RexCall further uses similar RexCalls
multiple times as an expression and that uses further RexCalls and son on upto
a depth of around 60. Actually I cant have a variable to hold the result but to
evaluate at runtime so I had to embed the repeated expression as I am creating
a sql query at the end to run it on BigQuery.
Let me show you how my RexCall looks like. Take a look at !OOM Screenshot.png!
The last variable StgValidationFlag is used in the filter which uses all the
previous evaluated RexCalls
blows out of memory as the string formation is too large to load even in the
IDE.
Moreover, these RexCalls are all if else conditions exprs.
> RexCall toString digest gives OOM while huge expression is evaluated
> --------------------------------------------------------------------
>
> Key: CALCITE-3784
> URL: https://issues.apache.org/jira/browse/CALCITE-3784
> Project: Calcite
> Issue Type: Bug
> Components: core
> Reporter: Ravi Kapoor
> Priority: Critical
> Attachments: OOM Screenshot.png
>
>
> I have some complex query which has tens of thousands of rex expressions and
> this expression is used in filter expression in the query.
> On creating a filter below code gets called:
> {code}
> public RelBuilder filter(Iterable<CorrelationId> variablesSet,
> Iterable<? extends RexNode> predicates) {
> final RexNode simplifiedPredicates =
> simplifier.simplifyFilterPredicates(predicates);
> if (simplifiedPredicates == null) {
> return empty();
> }
> {code}
> RexSimplify further adds the rexnode in the Set<RexNode> calling hashcode()
> internally which calls toString()
> Is there any way to avoid this computeDigest Call which creates complex
> string object and blows up the RAM about 14GB?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
