[
https://issues.apache.org/jira/browse/CALCITE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165996#comment-17165996
]
Stamatis Zampetakis edited comment on CALCITE-4144 at 7/27/20, 10:26 PM:
-------------------------------------------------------------------------
I think I understand the problem. [~zhangchenghui] is using both interpreter
mode (aka. {{BindableConvention}}) and compiler mode (aka.
{{EnumerableConvention}}) as I think it is the case for many people using the
default ruleset. Thus, I assume that some operators are in
{{BindableConvention}} and some on {{EnumerableConvention}}.
The {{BINDABLE_CACHE}} is taking care of class generation coming from the
compiler but there is no such cache for the interpreter. In my mind, I was
thinking that interpreter should never generate and compile code however, as it
is observed in this case some part of the interpreter ({{JaninoRexCompiler}})
does it.
I see 3 ways to address this issue:
* Implement a {{ScalarCompiler}} that does not generate code (such as
{{JaninoRexCompiler}}) but interprets expressions.
[FooCompiler|https://github.com/apache/calcite/blob/b306668d796a19ade127dce7b392f970cffd4b39/core/src/main/java/org/apache/calcite/interpreter/Interpreter.java#L141]
outlines how the code would look like. However, I don't know anybody pushing
in this direction so this might never actually happen.
* Make the {{JaninoRexCompiler}} to also use the BINDABLE_CACHE. This is
mostly code refactoring since all the pieces are there.
* Use exclusively {{Enumerable}} operators during planning which already
exploit the BINDABLE_CACHE.
was (Author: zabetak):
I think I understand the problem. [~zhangchenghui] is using both interpreter
mode (aka. {{BindableConvention}}) and compiler mode (aka.
{{EnumerableConvention}}) as I think it is the case for many people using the
default ruleset. Thus, I assume that some operators are in
{{BindableConvention}} and some on {{EnumerableConvention}}.
The {{BINDABLE_CACHE}} is taking care of class generation coming from the
compiler but not there is no such cache for the interpreter. In my mind, I was
thinking that interpreter should never generate and compile code however, as it
is observed in this case some part of the interpreter ({{JaninoRexCompiler}})
does it.
I see 3 ways to address this issue:
* Implement a {{ScalarCompiler}} that does not generate code (such as
{{JaninoRexCompiler}}) but interprets expressions.
[FooCompiler|https://github.com/apache/calcite/blob/b306668d796a19ade127dce7b392f970cffd4b39/core/src/main/java/org/apache/calcite/interpreter/Interpreter.java#L141]
outlines how the code would look like. However, I don't know anybody pushing
in this direction so this might never actually happen.
* Make the {{JaninoRexCompiler}} to also use the BINDABLE_CACHE. This is
mostly code refactoring since all the pieces are there.
* Use exclusively {{Enumerable}} operators during planning which already
exploit the BINDABLE_CACHE.
> Reduce code generation and class loading overhead when getScalar in
> JaninoRexCompiler.
> --------------------------------------------------------------------------------------
>
> Key: CALCITE-4144
> URL: https://issues.apache.org/jira/browse/CALCITE-4144
> Project: Calcite
> Issue Type: Improvement
> Components: core
> Affects Versions: 1.21.0
> Environment: version: calcite-core 1.21
> model: filterableTable
> Reporter: zhangchenghui
> Priority: Major
> Labels: cache, scalar
> Fix For: 1.25.0
>
> Attachments: image-2020-07-27-22-44-44-455.png,
> image-2020-07-27-22-45-25-350.png, image-2020-07-27-22-45-54-346.png,
> image-2020-07-27-22-46-18-306.png
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> I used the FilterableTable mode in the project, but I found that the query
> was particularly slow. I used the JProfile tool to troubleshoot the thread
> time-consuming place, and then through the debug, I found that each request
> took two places:
> 1、org.apache.calcite.adapter.enumerable.EnumerableInterpretable#getBindable
> !image-2020-07-27-22-44-44-455.png!
> This place optimizes the cache settings by setting the cache size.
> 2、org.apache.calcite.interpreter.JaninoRexCompiler#baz
> !image-2020-07-27-22-45-25-350.png!
> But this place is not cached, and a new expression string is used every time
> to create it through reflection.
> JProfile tool time consumption:
> !image-2020-07-27-22-45-54-346.png!
> I originally wanted to add a layer of cache here, but found that the
> expressions generated each time are different, as follows:
> !image-2020-07-27-22-46-18-306.png!
> So you can't use the getBindable method to directly add cache.
> Is there anything you can optimize here? For example, can you generate a
> template class in advance, and then generate different objects through
> different parameter values?
> Bring tea to the boss!
> Looking forward to your reply!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
