[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287949754
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkRewriteSubQueryRule.scala
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, operandJ}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelOptRuleOperand}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Aggregate, Filter, RelFactories}
+import org.apache.calcite.rex.{RexShuttle, _}
+import org.apache.calcite.sql.SqlKind
+import org.apache.calcite.sql.`type`.SqlTypeFamily
+import org.apache.calcite.sql.fun.SqlCountAggFunction
+import org.apache.calcite.tools.RelBuilderFactory
+
+import scala.collection.JavaConversions._
+
+/**
+ * Planner rule that rewrites filter condition like:
+ * `(select count(*) from T) > 0` to `exists(select * from T)`,
Review comment:
the original query is a scalar query, which will be converted to aggregate +
join by Calcite rules.
for example, full sql like `SELECT * FROM x WHERE (SELECT COUNT(*) FROM y
WHERE d > 10) > 0` ，the logical plan converted by Calcite rules is:
```
LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalFilter(condition=[>($3, 0)])
+- LogicalJoin(condition=[true], joinType=[left])
:- LogicalTableScan(table=[[x]])
+- LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
+- LogicalProject($f0=[0])
+- LogicalFilter(condition=[>($0, 10)])
+- LogicalTableScan(table=[[y]])
```
i will update the class comments to make it more clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287949754
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkRewriteSubQueryRule.scala
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, operandJ}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelOptRuleOperand}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Aggregate, Filter, RelFactories}
+import org.apache.calcite.rex.{RexShuttle, _}
+import org.apache.calcite.sql.SqlKind
+import org.apache.calcite.sql.`type`.SqlTypeFamily
+import org.apache.calcite.sql.fun.SqlCountAggFunction
+import org.apache.calcite.tools.RelBuilderFactory
+
+import scala.collection.JavaConversions._
+
+/**
+ * Planner rule that rewrites filter condition like:
+ * `(select count(*) from T) > 0` to `exists(select * from T)`,
Review comment:
the original query is a scalar query, which will be converted to aggregate +
join by Calcite rules.
for example, full sql like `SELECT * FROM x WHERE (SELECT COUNT(*) FROM y
WHERE d > 10) > 0` ，the logical plan converted by Calcite rules is:
```
LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalFilter(condition=[>($3, 0)])
+- LogicalJoin(condition=[true], joinType=[left])
:- LogicalTableScan(table=[[x]])
+- LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
+- LogicalProject($f0=[0])
+- LogicalFilter(condition=[>($0, 10)])
+- LogicalTableScan(table=[[y]])
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287949754
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkRewriteSubQueryRule.scala
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, operandJ}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelOptRuleOperand}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Aggregate, Filter, RelFactories}
+import org.apache.calcite.rex.{RexShuttle, _}
+import org.apache.calcite.sql.SqlKind
+import org.apache.calcite.sql.`type`.SqlTypeFamily
+import org.apache.calcite.sql.fun.SqlCountAggFunction
+import org.apache.calcite.tools.RelBuilderFactory
+
+import scala.collection.JavaConversions._
+
+/**
+ * Planner rule that rewrites filter condition like:
+ * `(select count(*) from T) > 0` to `exists(select * from T)`,
Review comment:
the original query is a scalar query, which will be converted to aggregate +
join by Calcite rules.
for example, full sql like `SELECT * FROM x WHERE (SELECT COUNT(*) FROM y
WHERE d > 10) > 0` ，the logical plan converted by Calcite rules is:
`LogicalProject(a=[$0], b=[$1], c=[$2])`
`+- LogicalProject(a=[$0], b=[$1], c=[$2])`
` +- LogicalFilter(condition=[>($3, 0)])`
` +- LogicalJoin(condition=[true], joinType=[left])`
` :- LogicalTableScan(table=[[x]])`
` +- LogicalAggregate(group=[{}], EXPR$0=[COUNT()])`
`+- LogicalProject($f0=[0])`
` +- LogicalFilter(condition=[>($0, 10)])`
` +- LogicalTableScan(table=[[y]])`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287949754
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkRewriteSubQueryRule.scala
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, operandJ}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelOptRuleOperand}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Aggregate, Filter, RelFactories}
+import org.apache.calcite.rex.{RexShuttle, _}
+import org.apache.calcite.sql.SqlKind
+import org.apache.calcite.sql.`type`.SqlTypeFamily
+import org.apache.calcite.sql.fun.SqlCountAggFunction
+import org.apache.calcite.tools.RelBuilderFactory
+
+import scala.collection.JavaConversions._
+
+/**
+ * Planner rule that rewrites filter condition like:
+ * `(select count(*) from T) > 0` to `exists(select * from T)`,
Review comment:
the original query is a scalar query, which will be converted to aggregate +
join by Calcite rules.
for example, full sql like `SELECT * FROM x WHERE (SELECT COUNT(*) FROM y
WHERE d > 10) > 0` ，the logical plan converted by Calcite rules is:
LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalFilter(condition=[>($3, 0)])
+- LogicalJoin(condition=[true], joinType=[left])
:- LogicalTableScan(table=[[x]])
+- LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
+- LogicalProject($f0=[0])
+- LogicalFilter(condition=[>($0, 10)])
+- LogicalTableScan(table=[[y]])
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287949754
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkRewriteSubQueryRule.scala
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, operandJ}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelOptRuleOperand}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Aggregate, Filter, RelFactories}
+import org.apache.calcite.rex.{RexShuttle, _}
+import org.apache.calcite.sql.SqlKind
+import org.apache.calcite.sql.`type`.SqlTypeFamily
+import org.apache.calcite.sql.fun.SqlCountAggFunction
+import org.apache.calcite.tools.RelBuilderFactory
+
+import scala.collection.JavaConversions._
+
+/**
+ * Planner rule that rewrites filter condition like:
+ * `(select count(*) from T) > 0` to `exists(select * from T)`,
Review comment:
the original query is a scalar query, which will be converted to aggregate +
join by Calcite rules.
for example, full sql like `SELECT * FROM x WHERE (SELECT COUNT(*) FROM y
WHERE d > 10) > 0` ，the logical plan converted by Calcite rules is:
`
LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalProject(a=[$0], b=[$1], c=[$2])
+- LogicalFilter(condition=[>($3, 0)])
+- LogicalJoin(condition=[true], joinType=[left])
:- LogicalTableScan(table=[[x]])
+- LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
+- LogicalProject($f0=[0])
+- LogicalFilter(condition=[>($0, 10)])
+- LogicalTableScan(table=[[y]])
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287917681
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkRewriteSubQueryRule.scala
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, operandJ}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelOptRuleOperand}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Aggregate, Filter, RelFactories}
+import org.apache.calcite.rex.{RexShuttle, _}
+import org.apache.calcite.sql.SqlKind
+import org.apache.calcite.sql.`type`.SqlTypeFamily
+import org.apache.calcite.sql.fun.SqlCountAggFunction
+import org.apache.calcite.tools.RelBuilderFactory
+
+import scala.collection.JavaConversions._
+
+/**
+ * Planner rule that rewrites filter condition like:
+ * `(select count(*) from T) > 0` to `exists(select * from T)`,
Review comment:
The estimation for SEMI/ANTI join is very inaccurate，so we intend to do
deterministic rewriting on SEMI/ANTI join. And we can put this rule to CBO
after we improve estimation of SEMI/ANTI join.
yes, we can do similarly rewriting for `exists(select * from T limit 1)`
later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287917681
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkRewriteSubQueryRule.scala
##
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, operandJ}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall, RelOptRuleOperand}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Aggregate, Filter, RelFactories}
+import org.apache.calcite.rex.{RexShuttle, _}
+import org.apache.calcite.sql.SqlKind
+import org.apache.calcite.sql.`type`.SqlTypeFamily
+import org.apache.calcite.sql.fun.SqlCountAggFunction
+import org.apache.calcite.tools.RelBuilderFactory
+
+import scala.collection.JavaConversions._
+
+/**
+ * Planner rule that rewrites filter condition like:
+ * `(select count(*) from T) > 0` to `exists(select * from T)`,
Review comment:
The estimation for SEMI/ANTI join is very inaccurate，so we intend to do
deterministic rewriting on SEMI/ANTI join. And we can put this rule to CBO
after we improve estimation of SEMI/ANTI join.
yes, we can do similarly rewriting for `exists(select * from T limit 1)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #8520: [FLINK-12600] [table-planner-blink] Introduce planner rules to do deterministic rewriting on RelNode
godfreyhe commented on a change in pull request #8520: [FLINK-12600]
[table-planner-blink] Introduce planner rules to do deterministic rewriting on
RelNode
URL: https://github.com/apache/flink/pull/8520#discussion_r287914800
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/plan/rules/logical/FlinkPruneEmptyRules.scala
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.logical
+
+import org.apache.calcite.plan.RelOptRule.{any, none, operand, some}
+import org.apache.calcite.plan.{RelOptRule, RelOptRuleCall}
+import org.apache.calcite.rel.RelNode
+import org.apache.calcite.rel.core.{Join, JoinRelType, Values}
+
+object FlinkPruneEmptyRules {
Review comment:
`PruneEmptyRules` in Calcite contains more than one rules, I intend to keep
calcite style for this rule. Maybe later, we need to copy other rules from
`PruneEmptyRules` to this file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
