Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-06-13 Thread Vladimir Ozerov
Please disregard this email. I sent it to the wrong thread, sorry. вс, 13 июн. 2021 г. в 19:56, Vladimir Ozerov : > Hi, > > I tried to apply different approaches, but eventually, I failed to achieve > my goals. It seems that the current implementation cannot handle the > required scenario, as exp

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-06-13 Thread Vladimir Ozerov
Hi, I tried to apply different approaches, but eventually, I failed to achieve my goals. It seems that the current implementation cannot handle the required scenario, as explained below. Consider the following tree: LogicalAggregate1[group=[b,c]] LogicalAggregate2[group=[a,b,c]] LogicalInpu

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-30 Thread Jinpeng Wu
Hi, Vladimir. I mean the error part only. I am comfortable with the other changes. If changing isTransformationRule could have unexpected consequences, one possible way is reserving current logic and only adding a newline checking the "implements TransformationRule''. Even though we remove the or

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-30 Thread Vladimir Ozerov
Hi Jinpeng, Do you mean the whole change or the error part only? My concern is that if we change the implementation of VolcanoPlanner.isTransformationRule, then some transformation rules that are not marked as "implements TransformationRule" will be treated as implementation rules, which may lead

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-30 Thread Jinpeng Wu
Hi. Warnings and Fixing isTransformationRule are good solutions. But I am still concerned about reporting an error. It will force users to do a large refactoring on existing codes before they can migrate to the new rule driver. Any refactoring can be risky, especially for those critical services.

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-29 Thread Vladimir Ozerov
Great. Does the community have any objections to the following fix? 1. Add a flag to a rule call instance with the expected node type. In the case of a mismatch, we will print a warning once per rule per JVM instance to avoid too many messages. Alternatively, we may print a warning per rule per Vol

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Haisheng Yuan
Great, that is the correct way to change it and that should be the default implementation. On 2021/05/28 17:41:15, Vladimir Ozerov wrote: > BTW, I tried to change the implementation to: > > 1: protected boolean isTransformationRule(VolcanoRuleCall match) { > 2:return match.getRule() inst

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Vladimir Ozerov
BTW, I tried to change the implementation to: 1: protected boolean isTransformationRule(VolcanoRuleCall match) { 2:return match.getRule() instanceof TransformationRule; 3: } It solved my problem - plans returned to normal. In the Apache Calcite repo, only 4 tests in the TopDowOptTest class

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Vladimir Ozerov
Hi Jinpeng, Thank you for the clarification. When I saw the code in question for the first time, my first thought was that it was perhaps designed for gradual migration. The main problem is that the current implementation discards parts of the plan *silently*, which might be difficult to spot. I o

Re: Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Jinpeng Wu
Hi, Vladimir. Good catch! There could be some improvements here. Actually, this problem was discovered early when the top-down rule driver was designed. At that time, no rule was annotated as "TransformationRule". Moreover, it is impossible to ask every calcite user who designed their own rules to

Top-down optimizer cannot explore the search space because physical rule is treated as transformation rule

2021-05-28 Thread Vladimir Ozerov
Hi, I have an optimizer that uses top-down VolcanoPlanner and has a ConverterRule for every LogicalNode. I have a new requirement when one of the physical rules must emit several physical nodes instead of one. I tried to convert a ConverterRule to a normal rule that emits physical nodes. Even thou