[jira] [Resolved] (CALCITE-3505) Infinite matching of FilterProjectTransposeRule causes stackoverflow
[
https://issues.apache.org/jira/browse/CALCITE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-3505.
Resolution: Cannot Reproduce
> Infinite matching of FilterProjectTransposeRule causes stackoverflow
>
>
> Key: CALCITE-3505
> URL: https://issues.apache.org/jira/browse/CALCITE-3505
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jin Xing
>Priority: Major
> Attachments: graphviz.svg
>
>
> Run ScannableTableTest#testProjectableFilterableTableJoin with minor change
> to reproduce
> {code:java}
> @Test public void testProjectableFilterableTableJoin() throws Exception {
> final StringBuilder buf = new StringBuilder();
> final String explain = "PLAN="
> + "EnumerableHashJoin(condition=[=($0, $3)], joinType=[inner])\n"
> + " EnumerableInterpreter\n"
> + "BindableTableScan(table=[[s, b1]], filters=[[=($0, 10)]])\n"
> + " EnumerableInterpreter\n"
> + "BindableTableScan(table=[[s, b2]], filters=[[=($0, 10)]])";
> CalciteAssert.that()
> .with(
> newSchema("s",
> Pair.of("b1", new BeatlesProjectableFilterableTable(buf,
> true)),
> Pair.of("b2", new BeatlesProjectableFilterableTable(buf,
> true
> .query("select * from \"s\".\"b1\", \"s\".\"b2\" "
> + "where \"s\".\"b1\".\"i\" = 10 and \"s\".\"b2\".\"i\" =
> 10 "
> + "and \"s\".\"b1\".\"i\" = \"s\".\"b2\".\"i\"")
> .withHook(Hook.PLANNER, (Consumer) planner -> {
> planner.removeRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
> })
> .explainContains(explain);
> }
> {code}
> This test has nothing to do with ENUMERABLE_MERGE_JOIN_RULE, but if we
> disable it with planner hook, stackoverflow happens;
> I debugged and found that FilterProjectTransposeRule is matched infinitely
> but not sure the root cause.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-4705) Support hints like /*+ skewjoin(a(c0, c1)) */
[
https://issues.apache.org/jira/browse/CALCITE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564604#comment-17564604
]
Haisheng Yuan commented on CALCITE-4705:
I don't know any other databases that support skew join hint besides these two.
> Support hints like /*+ skewjoin(a(c0, c1)) */
> -
>
> Key: CALCITE-4705
> URL: https://issues.apache.org/jira/browse/CALCITE-4705
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: jibiyr
>Assignee: jibiyr
>Priority: Minor
>
> Dear community,
> calcite parse support hint now,but it can not parse hint like follow,
> _*select /*+ skewjoin(a(c0, c1)) */ * from T0 a join T1 b on a.c0 = b.c0,*_
> this is the bnf which is calcite supported now,
> {code:java}
> //
> hint:
> hintName
> | hintName '(' hintOptions ')'
> hintOptions:
> hintKVOption [, hintKVOption ]*
> | optionName [, optionName ]*
> | optionValue [, optionValue ]*
> hintKVOption:
> optionName '=' stringLiteral
> | stringLiteral '=' stringLiteral
> optionValue:
> stringLiteral
> | numericLiteral
> {code}
> I think the hintOptions should support the expression pattern,like a(c0,c1).
> And this pattern can be parsed by spark/presto now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5000) Expand rule of `AGGREGATE_REDUCE_FUNCTIONS`, when arg of agg-call is in the aggregate's group
[
https://issues.apache.org/jira/browse/CALCITE-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-5000.
Fix Version/s: 1.30.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/c8d1cb5a1e1b1dd98966c29308bc6b1aed245501,
thanks for your PR, [~wojustme]!
> Expand rule of `AGGREGATE_REDUCE_FUNCTIONS`, when arg of agg-call is in the
> aggregate's group
> -
>
> Key: CALCITE-5000
> URL: https://issues.apache.org/jira/browse/CALCITE-5000
> Project: Calcite
> Issue Type: Improvement
>Reporter: Xurenhe
>Assignee: Xurenhe
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.30.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Using calcite's SubstitutionVisitor, we need do a lot work about
> canonicalizing relnode.
> Some sql should be simplified.
> {noformat}
> -- origin sql
> select deptno, salary, max(salary), min(commission)
> from emps
> group by deptno, salary
> -- simplified
> select deptno, salary, salary as expr, min(commission)
> from emps
> group by deptno, salary{noformat}
> It should be simplified, but I didn't find any rule to simplify it.
> This issue is related to CALCITE-4857
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (CALCITE-5032) The unexpected behavior on method RelOptUtil#splitJoinCondition
[
https://issues.apache.org/jira/browse/CALCITE-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-5032.
Fix Version/s: 1.30.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/fe9766aaa5739f59f757206c57c1b9cc071cd6f6,
thanks for your work, [~libenchao]!
> The unexpected behavior on method RelOptUtil#splitJoinCondition
> ---
>
> Key: CALCITE-5032
> URL: https://issues.apache.org/jira/browse/CALCITE-5032
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: luoyuxia
>Assignee: Benchao Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.30.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> When I call the method
> {code:java}
> RexNode splitJoinCondition(
> List sysFieldList,
> List inputs,
> RexNode condition,
> List> joinKeys,
> @Nullable List filterNulls,
> @Nullable List rangeOp)
> {code}
> , and the passed `joinKeys` is list with size 2, `condition` is like "1 =
> 1", then I found the `joinKeys` passed dosen't change as expected.
> For the `joinKeys`, the first list will contain two RexNodes, and the second
> list contains no RexNode. I think the expected behavior should be the first
> list contain one RexNode, and the second contain one RexNode too.
> After I dive into the code, I found such code will be invoked:
> {code:java}
> if ((rangeOp == null)
> && ((leftKey == null) || (rightKey == null))) {
> // no equality join keys found yet:
> // try transforming the condition to
> // equality "join" conditions, e.g.
> // f(LHS) > 0 ===> ( f(LHS) > 0 ) = TRUE,
> // and make the RHS produce TRUE, but only if we're strictly
> // looking for equi-joins
> final ImmutableBitSet projRefs = InputFinder.bits(condition);
> leftKey = null;
> rightKey = null;
> boolean foundInput = false;
> for (int i = 0; i < inputs.size() && !foundInput; i++) {
> if (inputsRange[i].contains(projRefs)) {
> leftInput = i;
> leftFields = inputs.get(leftInput).getRowType().getFieldList();
> leftKey = condition.accept(
> new RelOptUtil.RexInputConverter(
> rexBuilder,
> leftFields,
> leftFields,
> adjustments));
> rightKey = rexBuilder.makeLiteral(true);
> // effectively performing an equality comparison
> kind = SqlKind.EQUALS;
> foundInput = true;
> }
> }
> }
> {code}
> It'll set `leftKey` and `righeKey`, but it won't set `rightInput`. So
> `rightInput` will be 0.
> Then it'll try to add `leftKey` and `righeKey` to `joinKeys`,
> {code:java}
> addJoinKey(
> joinKeys.get(leftInput),
> leftKey,
> (rangeOp != null) && !rangeOp.isEmpty());
> addJoinKey(
> joinKeys.get(rightInput),
> rightKey,
> (rangeOp != null) && !rangeOp.isEmpty());
> {code}
> But the `rightInput` is 0, so the first list of `joinKeys` will be also add
> `righeKey`.
> Could anyone please help me? Is it a bug or I misunderstand something? Thanks
> very much.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-4542) Suboptimal plan is chosen when TopDownRuleDriver is enabled
[
https://issues.apache.org/jira/browse/CALCITE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502791#comment-17502791
]
Haisheng Yuan commented on CALCITE-4542:
Hi [~korlov], thank you for providing the detailed test case.
However, the test case doesn't show there is a bug in the optimizer. Instead, I
found something strange in the test case.
In
https://github.com/korlov42/calcite/commit/f4a5c2f01e0ec67156f7e91c6b5839dca1db6776#diff-853e16de41615bab150cd7ecf77a868e4b4ab0396d5c9f283d4247c222eba814R974,
instead of node.copy(), you should use RelOptRule.convert(). As you can see,
the tablescan (MyTableScan(subset=[rel#29:RelSubset#0.MY.any.[0]])) in the
final plan is not even registered in the MEMO structure.
> Suboptimal plan is chosen when TopDownRuleDriver is enabled
>
>
> Key: CALCITE-4542
> URL: https://issues.apache.org/jira/browse/CALCITE-4542
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.26.0
>Reporter: Konstantin Orlov
>Priority: Major
> Attachments: dump.txt
>
>
> When TopDownRuleDriver is enabled, suboptimal plan is chosen for query with
> join.
> We have our own convention and implementation of all necessary relations. A
> distributed join is considered less expensive by our cost system than a
> single-distributed join, and the merge join is considered less expensive than
> nested loop if index over join condition is present. Nevertheless the merge
> join with a single distribution is chosen by optimizer.
> The query is:
> {code:java}
> select e1."empid", e1."deptno" from "emps" e1 join "emps" e2 on e1."empid" =
> e2."empid"
> {code}
> Actual plan is:
> {code:java}
> MyProject(subset=[rel#16:RelSubset#2.MY.single.[]], empid=[$0], deptno=[$1]):
> rowcount = 1500.0, cumulative cost = {1500.0 rows, 3000.0 cpu, 0.0 io}, id =
> 21
> MyMergeJoin(subset=[rel#20:RelSubset#1.MY.single.[]], condition=[=($0,
> $5)], joinType=[inner]): rowcount = 1500.0, cumulative cost = {150.0 rows,
> 0.0 cpu, 0.0 io}, id = 50
> MyExchange(subset=[rel#25:RelSubset#0.MY.single.[]],
> distribution=[single]): rowcount = 100.0, cumulative cost =
> {9210.340371976183 rows, 100.0 cpu, 0.0 io}, id = 30
> MyTableScan(subset=[rel#29:RelSubset#0.MY.any.[0]], table=[[hr,
> emps]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io},
> id = 27
> MyExchange(subset=[rel#25:RelSubset#0.MY.single.[]],
> distribution=[single]): rowcount = 100.0, cumulative cost =
> {9210.340371976183 rows, 100.0 cpu, 0.0 io}, id = 30
> MyTableScan(subset=[rel#29:RelSubset#0.MY.any.[0]], table=[[hr,
> emps]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io},
> id = 27
> {code}
> Expected plan is:
> {code:java}
> MyProject(subset=[rel#16:RelSubset#2.MY.single.[]], empid=[$0], deptno=[$1]):
> rowcount = 1500.0, cumulative cost = {1500.0 rows, 3000.0 cpu, 0.0 io}, id =
> 21
> MyExchange(subset=[rel#20:RelSubset#1.MY.single.[]],
> distribution=[single]): rowcount = 1500.0, cumulative cost =
> {18420.680743952365 rows, 100.0 cpu, 0.0 io}, id = 24
> MyMergeJoin(subset=[rel#23:RelSubset#1.MY.any.[]], condition=[=($0, $5)],
> joinType=[inner]): rowcount = 1500.0, cumulative cost = {0.15 rows, 0.0 cpu,
> 0.0 io}, id = 50
> MyTableScan(subset=[rel#26:RelSubset#0.MY.hash[0].[0]], table=[[hr,
> emps]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io},
> id = 25
> MyTableScan(subset=[rel#26:RelSubset#0.MY.hash[0].[0]], table=[[hr,
> emps]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io},
> id = 25
> {code}
> Planner [^dump.txt] doesn't contain the join with proper distribution.
> Reproducer could be found
> [here|https://github.com/korlov42/calcite/tree/derive-not-being-called-repoducer].
> Please run {{org.apache.calcite.tools.PlannerTest#test}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (CALCITE-4774) When predicate conditions are equivalent, materialized view recognition fails.
[
https://issues.apache.org/jira/browse/CALCITE-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428655#comment-17428655
]
Haisheng Yuan commented on CALCITE-4774:
Yes, sorry for the confusion, I will ask for code comments for similar tests
for future pull requests.
> When predicate conditions are equivalent, materialized view recognition fails.
> --
>
> Key: CALCITE-4774
> URL: https://issues.apache.org/jira/browse/CALCITE-4774
> Project: Calcite
> Issue Type: Improvement
>Reporter: xzh_dz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> // code placeholder
> org.apache.calcite.test.MaterializedViewSubstitutionVisitorTest
> @Test void testRexCondition() {
> final String mv = ""
> + "select \"name\"\n"
> + "from \"emps\"\n"
> + "where \"deptno\" > 100 and \"deptno\" > 50\n"
> + "group by \"name\"";
> final String query = ""
> + "select \"name\"\n"
> + "from \"emps\"\n"
> + "where \"deptno\" > 100"
> + "group by \"name\"";
> sql(mv, query).withChecker(
> resultContains(""
> + "EnumerableTableScan(table=[[hr, MV0]])")).ok();
> }
> {code}
> Materialized view failed to be matched by optimized results:Materialized view
> failed to be matched by optimized results: java.lang.AssertionError:
> Materialized view failed to be matched by optimized results: at
> org.apache.calcite.test.AbstractMaterializedViewTest.checkMaterialize(AbstractMaterializedViewTest.java:116)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest.access$000(AbstractMaterializedViewTest.java:67)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest$Sql.ok(AbstractMaterializedViewTest.java:229)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4817) Expand SubstitutionVisitor of Aggregate with max/min, which column is the group by list of target
[
https://issues.apache.org/jira/browse/CALCITE-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426404#comment-17426404
]
Haisheng Yuan commented on CALCITE-4817:
Reverted the previous commit.
> Expand SubstitutionVisitor of Aggregate with max/min, which column is the
> group by list of target
> -
>
> Key: CALCITE-4817
> URL: https://issues.apache.org/jira/browse/CALCITE-4817
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Xurenhe
>Assignee: Xurenhe
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-10-08-18-48-56-093.png,
> image-2021-10-08-20-14-10-791.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Now, SubstitutionVisitor support `Query with distinct aggregate on the column
> is matched to the group list existing in target`, as CALCITE-4374 describes.
> But. It's fail, when query aggregate with max/min based on the column, which
> is in the group by list of target.
> Materialized view recognition should also success.
> {code:java}
> //代码占位符
> @Test void testQueryNoDistinctOptionalityAggCallColInTargetGroupBy1() {
> final String mv = ""
> + "select \"name\", \"deptno\" "
> + "from \"emps\" group by \"name\", \"deptno\"";
> final String query = ""
> + "select \"name\", min(\"deptno\")\n"
> + "from \"emps\" group by \"name\"";
> sql(mv, query).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (CALCITE-4817) Expand SubstitutionVisitor of Aggregate with max/min, which column is the group by list of target
[
https://issues.apache.org/jira/browse/CALCITE-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan reopened CALCITE-4817:
> Expand SubstitutionVisitor of Aggregate with max/min, which column is the
> group by list of target
> -
>
> Key: CALCITE-4817
> URL: https://issues.apache.org/jira/browse/CALCITE-4817
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Xurenhe
>Assignee: Xurenhe
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Attachments: image-2021-10-08-18-48-56-093.png,
> image-2021-10-08-20-14-10-791.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Now, SubstitutionVisitor support `Query with distinct aggregate on the column
> is matched to the group list existing in target`, as CALCITE-4374 describes.
> But. It's fail, when query aggregate with max/min based on the column, which
> is in the group by list of target.
> Materialized view recognition should also success.
> {code:java}
> //代码占位符
> @Test void testQueryNoDistinctOptionalityAggCallColInTargetGroupBy1() {
> final String mv = ""
> + "select \"name\", \"deptno\" "
> + "from \"emps\" group by \"name\", \"deptno\"";
> final String query = ""
> + "select \"name\", min(\"deptno\")\n"
> + "from \"emps\" group by \"name\"";
> sql(mv, query).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4817) Expand SubstitutionVisitor of Aggregate with max/min, which column is the group by list of target
[
https://issues.apache.org/jira/browse/CALCITE-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-4817:
---
Fix Version/s: (was: 1.28.0)
> Expand SubstitutionVisitor of Aggregate with max/min, which column is the
> group by list of target
> -
>
> Key: CALCITE-4817
> URL: https://issues.apache.org/jira/browse/CALCITE-4817
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Xurenhe
>Assignee: Xurenhe
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-10-08-18-48-56-093.png,
> image-2021-10-08-20-14-10-791.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Now, SubstitutionVisitor support `Query with distinct aggregate on the column
> is matched to the group list existing in target`, as CALCITE-4374 describes.
> But. It's fail, when query aggregate with max/min based on the column, which
> is in the group by list of target.
> Materialized view recognition should also success.
> {code:java}
> //代码占位符
> @Test void testQueryNoDistinctOptionalityAggCallColInTargetGroupBy1() {
> final String mv = ""
> + "select \"name\", \"deptno\" "
> + "from \"emps\" group by \"name\", \"deptno\"";
> final String query = ""
> + "select \"name\", min(\"deptno\")\n"
> + "from \"emps\" group by \"name\"";
> sql(mv, query).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4817) Expand SubstitutionVisitor of Aggregate with max/min, which column is the group by list of target
[
https://issues.apache.org/jira/browse/CALCITE-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-4817:
---
Summary: Expand SubstitutionVisitor of Aggregate with max/min, which column
is the group by list of target (was: Expand SubstitutionVisitor of Aggregate
with max/min, which column is he group by list of target)
> Expand SubstitutionVisitor of Aggregate with max/min, which column is the
> group by list of target
> -
>
> Key: CALCITE-4817
> URL: https://issues.apache.org/jira/browse/CALCITE-4817
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Xurenhe
>Assignee: Xurenhe
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Now, SubstitutionVisitor support `Query with distinct aggregate on the column
> is matched to the group list existing in target`, as CALCITE-4374 describes.
> But. It's fail, when query aggregate with max/min based on the column, which
> is in the group by list of target.
> Materialized view recognition should also success.
> {code:java}
> //代码占位符
> @Test void testQueryNoDistinctOptionalityAggCallColInTargetGroupBy1() {
> final String mv = ""
> + "select \"name\", \"deptno\" "
> + "from \"emps\" group by \"name\", \"deptno\"";
> final String query = ""
> + "select \"name\", min(\"deptno\")\n"
> + "from \"emps\" group by \"name\"";
> sql(mv, query).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4817) Expand SubstitutionVisitor of Aggregate with max/min, which column is he group by list of target
[
https://issues.apache.org/jira/browse/CALCITE-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-4817.
Fix Version/s: 1.28.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/74e97780add051cb71a122075e5bcbceb40e889c,
thanks for the PR, [~wojustme]!
> Expand SubstitutionVisitor of Aggregate with max/min, which column is he
> group by list of target
>
>
> Key: CALCITE-4817
> URL: https://issues.apache.org/jira/browse/CALCITE-4817
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Xurenhe
>Assignee: Xurenhe
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Now, SubstitutionVisitor support `Query with distinct aggregate on the column
> is matched to the group list existing in target`, as CALCITE-4374 describes.
> But. It's fail, when query aggregate with max/min based on the column, which
> is in the group by list of target.
> Materialized view recognition should also success.
> {code:java}
> //代码占位符
> @Test void testQueryNoDistinctOptionalityAggCallColInTargetGroupBy1() {
> final String mv = ""
> + "select \"name\", \"deptno\" "
> + "from \"emps\" group by \"name\", \"deptno\"";
> final String query = ""
> + "select \"name\", min(\"deptno\")\n"
> + "from \"emps\" group by \"name\"";
> sql(mv, query).ok();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4177) Throw exception when deserialize SqlOperator fails, do not return null
[
https://issues.apache.org/jira/browse/CALCITE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-4177.
Fix Version/s: 1.28.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/ce86af83caae335d1b47def55354d42a517f45d0,
thanks for the PR!
> Throw exception when deserialize SqlOperator fails, do not return null
> --
>
> Key: CALCITE-4177
> URL: https://issues.apache.org/jira/browse/CALCITE-4177
> Project: Calcite
> Issue Type: Improvement
>Reporter: Wang Yanlin
>Assignee: Wang Yanlin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Currently, when deserialize a RexNode fails, calcite returns a null value,
> causing NPE in somewhere else. So we cannot get any information about the
> SqlOperator in the stacktrace. This makes it hard to determine the reason
> when deserialize a very long json string.
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.util.Objects.requireNonNull(Objects.java:203)
> at
> org.apache.calcite.rel.core.AggregateCall.(AggregateCall.java:98)
> at
> org.apache.calcite.rel.core.AggregateCall.create(AggregateCall.java:198)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.toAggCall(RelJsonReader.java:289)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.access$500(RelJsonReader.java:59)
> at
> org.apache.calcite.rel.externalize.RelJsonReader$2.getAggregateCalls(RelJsonReader.java:172)
> at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:220)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:105)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.readRel(RelJsonReader.java:264)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.readRels(RelJsonReader.java:91)
> at
> org.apache.calcite.rel.externalize.RelJsonReader.read(RelJsonReader.java:85)
> at
> org.apache.calcite.plan.RelWriterTest.lambda$deserializeAndDumpToTextFormat$6(RelWriterTest.java:894)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
> ... 67 more
> {noformat}
> Maybe it's better to throw exception instead of return null, adding the name,
> kind, and syntax in the error message
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4774) When predicate conditions are equivalent, materialized view recognition fails.
[
https://issues.apache.org/jira/browse/CALCITE-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416969#comment-17416969
]
Haisheng Yuan commented on CALCITE-4774:
Hi [~nobigo], I checked the code again, they are not the same.
> When predicate conditions are equivalent, materialized view recognition fails.
> --
>
> Key: CALCITE-4774
> URL: https://issues.apache.org/jira/browse/CALCITE-4774
> Project: Calcite
> Issue Type: Improvement
>Reporter: xzh_dz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> // code placeholder
> org.apache.calcite.test.MaterializedViewSubstitutionVisitorTest
> @Test void testRexCondition() {
> final String mv = ""
> + "select \"name\"\n"
> + "from \"emps\"\n"
> + "where \"deptno\" > 100 and \"deptno\" > 50\n"
> + "group by \"name\"";
> final String query = ""
> + "select \"name\"\n"
> + "from \"emps\"\n"
> + "where \"deptno\" > 100"
> + "group by \"name\"";
> sql(mv, query).withChecker(
> resultContains(""
> + "EnumerableTableScan(table=[[hr, MV0]])")).ok();
> }
> {code}
> Materialized view failed to be matched by optimized results:Materialized view
> failed to be matched by optimized results: java.lang.AssertionError:
> Materialized view failed to be matched by optimized results: at
> org.apache.calcite.test.AbstractMaterializedViewTest.checkMaterialize(AbstractMaterializedViewTest.java:116)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest.access$000(AbstractMaterializedViewTest.java:67)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest$Sql.ok(AbstractMaterializedViewTest.java:229)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4774) When predicate conditions are equivalent, materialized view recognition fails.
[
https://issues.apache.org/jira/browse/CALCITE-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-4774.
Fix Version/s: 1.28.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/60457b28c890f12cc811528bd4f3b098bc3d00f7,
thanks for the PR, [~xzh_dz].
> When predicate conditions are equivalent, materialized view recognition fails.
> --
>
> Key: CALCITE-4774
> URL: https://issues.apache.org/jira/browse/CALCITE-4774
> Project: Calcite
> Issue Type: Improvement
>Reporter: xzh_dz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> // code placeholder
> org.apache.calcite.test.MaterializedViewSubstitutionVisitorTest
> @Test void testRexCondition() {
> final String mv = ""
> + "select \"name\"\n"
> + "from \"emps\"\n"
> + "where \"deptno\" > 100 and \"deptno\" > 50\n"
> + "group by \"name\"";
> final String query = ""
> + "select \"name\"\n"
> + "from \"emps\"\n"
> + "where \"deptno\" > 100"
> + "group by \"name\"";
> sql(mv, query).withChecker(
> resultContains(""
> + "EnumerableTableScan(table=[[hr, MV0]])")).ok();
> }
> {code}
> Materialized view failed to be matched by optimized results:Materialized view
> failed to be matched by optimized results: java.lang.AssertionError:
> Materialized view failed to be matched by optimized results: at
> org.apache.calcite.test.AbstractMaterializedViewTest.checkMaterialize(AbstractMaterializedViewTest.java:116)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest.access$000(AbstractMaterializedViewTest.java:67)
> at
> org.apache.calcite.test.AbstractMaterializedViewTest$Sql.ok(AbstractMaterializedViewTest.java:229)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Deleted] (CALCITE-4769) Does Dental Care Important? Check This Out!
[ https://issues.apache.org/jira/browse/CALCITE-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan deleted CALCITE-4769: --- > Does Dental Care Important? Check This Out! > --- > > Key: CALCITE-4769 > URL: https://issues.apache.org/jira/browse/CALCITE-4769 > Project: Calcite > Issue Type: Bug >Reporter: dentitox teeth >Priority: Minor > > Despite the fact that one in five adults says that someone’s smile is the > first thing they notice, the large majority of adults completely overlook the > health of their teeth. Plus, believe it or not, the health of your teeth and > mouth plays an important role in your overall health. That’s why it is > important to maintain proper dental hygiene and do what you can to improve > your oral health. > There’s now a supplement on the market that can do both for you. It’s called > Dentiox Pro, and it’s highly recommended by virtually every dental > professional. If you’re looking to improve your dental health and want a > beautiful smile, then look no further – there’s no better product on the > market than Dentiox Teeth. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4769) Does Dental Care Important? Check This Out!
[ https://issues.apache.org/jira/browse/CALCITE-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413596#comment-17413596 ] Haisheng Yuan commented on CALCITE-4769: Yes, I will completely delete it. Nor sure how to ban this spammer. > Does Dental Care Important? Check This Out! > --- > > Key: CALCITE-4769 > URL: https://issues.apache.org/jira/browse/CALCITE-4769 > Project: Calcite > Issue Type: Bug >Reporter: dentitox teeth >Priority: Minor > > Despite the fact that one in five adults says that someone’s smile is the > first thing they notice, the large majority of adults completely overlook the > health of their teeth. Plus, believe it or not, the health of your teeth and > mouth plays an important role in your overall health. That’s why it is > important to maintain proper dental hygiene and do what you can to improve > your oral health. > There’s now a supplement on the market that can do both for you. It’s called > Dentiox Pro, and it’s highly recommended by virtually every dental > professional. If you’re looking to improve your dental health and want a > beautiful smile, then look no further – there’s no better product on the > market than Dentiox Teeth. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CALCITE-4769) Does Dental Care Important? Check This Out!
[ https://issues.apache.org/jira/browse/CALCITE-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan closed CALCITE-4769. -- Resolution: Invalid > Does Dental Care Important? Check This Out! > --- > > Key: CALCITE-4769 > URL: https://issues.apache.org/jira/browse/CALCITE-4769 > Project: Calcite > Issue Type: Bug >Reporter: dentitox teeth >Priority: Minor > > Despite the fact that one in five adults says that someone’s smile is the > first thing they notice, the large majority of adults completely overlook the > health of their teeth. Plus, believe it or not, the health of your teeth and > mouth plays an important role in your overall health. That’s why it is > important to maintain proper dental hygiene and do what you can to improve > your oral health. > There’s now a supplement on the market that can do both for you. It’s called > Dentiox Pro, and it’s highly recommended by virtually every dental > professional. If you’re looking to improve your dental health and want a > beautiful smile, then look no further – there’s no better product on the > market than Dentiox Teeth. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4704) Log produced plan after rule application using explain formatting
[
https://issues.apache.org/jira/browse/CALCITE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412755#comment-17412755
]
Haisheng Yuan commented on CALCITE-4704:
OK, cancelling my -1. Change it to +1. :)
> Log produced plan after rule application using explain formatting
> -
>
> Key: CALCITE-4704
> URL: https://issues.apache.org/jira/browse/CALCITE-4704
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In many cases, we want to identify which rule lead to a certain
> transformation in the plan or need to observe how the query plan evolves by
> applying some rules in order to fix some bug or find the right place to
> introduce another optimization step.
> Currently there are some logs during the application of a rule triggered by
> the
> [HepPlanner|https://github.com/apache/calcite/blob/e04f3b08dcfb6910ff4df3810772c346b25ed424/core/src/main/java/org/apache/calcite/plan/AbstractRelOptPlanner.java#L367]
> and
> [VolcanoPlanner|https://github.com/apache/calcite/blob/e04f3b08dcfb6910ff4df3810772c346b25ed424/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoRuleCall.java#L126]
> but they more or less display only the top operator of the transformation
> and not the whole subtree. Moreover, the {{RelNode#toString}} used in these
> logs, is not self-contained, so in order to check the transformation the log
> entry needs to be matched with other logs.
> It would help if instead of displaying only the top operator we logged the
> equivalent of explain (i.e., {{RelOptUtil.toString}}) on the transformed
> sub-tree.
> You can find below some extracts from the current logs and how the proposed
> log could look like.
> *HepPlanner*
> +Current logging+
> {noformat}
> 2021-07-27 14:37:58,252 [ForkJoinPool-1-worker-9] DEBUG - call#0: Rule
> FilterIntoJoinRule arguments
> [rel#9:LogicalFilter.NONE.[](input=HepRelVertex#8,condition==($7, $8)),
> rel#7:LogicalJoin.NONE.[](left=HepRelVertex#5,right=HepRelVertex#6,condition=true,joinType=left)]
> produced
> rel#14:LogicalProject.NONE.[](input=LogicalJoin#13,inputs=0..7,exprs=[CAST($8):TINYINT,
> $9, $10])
> 2021-07-27 14:37:58,266 [ForkJoinPool-1-worker-9] DEBUG - call#1: Rule
> ProjectJoinTransposeRule arguments
> [rel#16:LogicalProject.NONE.[](input=HepRelVertex#15,inputs=0..7,exprs=[CAST($8):TINYINT,
> $9, $10]),
> rel#13:LogicalJoin.NONE.[](left=HepRelVertex#5,right=HepRelVertex#6,condition==($7,
> $8),joinType=inner)] produced
> rel#21:LogicalProject.NONE.[](input=LogicalJoin#20,inputs=0..7,exprs=[$11,
> $9, $10])
> 2021-07-27 14:37:58,269 [ForkJoinPool-1-worker-9] DEBUG - call#3: Rule
> ProjectMergeRule arguments
> [rel#11:LogicalProject.NONE.[](input=HepRelVertex#27,exprs=[$1]),
> rel#26:LogicalProject.NONE.[](input=HepRelVertex#25,inputs=0..7,exprs=[$11,
> $9, $10])] produced
> rel#28:LogicalProject.NONE.[](input=HepRelVertex#25,exprs=[$1])
> 2021-07-27 14:37:58,272 [ForkJoinPool-1-worker-9] DEBUG - call#4: Rule
> ProjectJoinTransposeRule arguments
> [rel#28:LogicalProject.NONE.[](input=HepRelVertex#25,exprs=[$1]),
> rel#24:LogicalJoin.NONE.[](left=HepRelVertex#22,right=HepRelVertex#23,condition==($7,
> $8),joinType=inner)] produced
> rel#33:LogicalProject.NONE.[](input=LogicalJoin#32,inputs=0)
> 2021-07-27 14:37:58,274 [ForkJoinPool-1-worker-9] DEBUG - call#6: Rule
> ProjectMergeRule arguments
> [rel#30:LogicalProject.NONE.[](input=HepRelVertex#22,exprs=[$1, $7]),
> rel#18:LogicalProject.NONE.[0](input=HepRelVertex#5,inputs=0..7)] produced
> rel#40:LogicalProject.NONE.[](input=HepRelVertex#5,exprs=[$1, $7])
> 2021-07-27 14:37:58,275 [ForkJoinPool-1-worker-9] DEBUG - call#7: Rule
> ProjectMergeRule arguments
> [rel#31:LogicalProject.NONE.[0](input=HepRelVertex#23,inputs=0),
> rel#19:LogicalProject.NONE.[[0],
> [3]](input=HepRelVertex#6,inputs=0..2,exprs=[CAST($0):TINYINT])] produced
> rel#42:LogicalProject.NONE.[0](input=HepRelVertex#6,inputs=0)
> {noformat}
> +Proposed logging+
> {noformat}
> 2021-07-27 14:37:58,260 [ForkJoinPool-1-worker-9] DEBUG - Rule
> FilterIntoJoinRule produced:
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], DEPTNO0=[CAST($8):TINYINT], DNAME=[$9],
> LOC=[$10])
> LogicalJoin(condition=[=($7, $8)], joinType=[inner])
> LogicalTableScan(table=[[scott, EMP]])
> LogicalTableScan(table=[[scott, DEPT]])
> 2021-07-27 14:37:58,267 [ForkJoinPool-1-worker-9] DEBUG - Rule
> ProjectJoinTransposeRule produced:
>
[jira] [Commented] (CALCITE-4741) AbstractRelNode#getId can overflow into a negative value causing CompileException in certain Enumerable implement methods
[
https://issues.apache.org/jira/browse/CALCITE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401885#comment-17401885
]
Haisheng Yuan commented on CALCITE-4741:
Kind of related with CALCITE-3879.
> AbstractRelNode#getId can overflow into a negative value causing
> CompileException in certain Enumerable implement methods
> -
>
> Key: CALCITE-4741
> URL: https://issues.apache.org/jira/browse/CALCITE-4741
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Ruben Q L
>Assignee: Ruben Q L
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{AbstractRelNode#getId}} can overflow into a negative value.
> This id is used by the {{implement}} method of several Enumerable operators
> ({{EnumerableBatchNestedLoopJoin}} and {{EnumerableMergeUnion}}) to guarantee
> a unique variable name in the dynamic code, e.g.:
> {code:java}
> final ParameterExpression corrArgList =
> Expressions.parameter(Modifier.FINAL,
> List.class, "corrList" + this.getId());
> {code}
> However, if the id is negative, this variable name will have a "-", causing a
> {{CompileException}}:
> {noformat}
> ...
> final org.apache.calcite.linq4j.Enumerable _inputEnumerable1 =
> org.apache.calcite.linq4j.EnumerableDefaults.correlateBatchJoin(org.apache.calcite.linq4j.JoinType.LEFT,
> left0, new org.apache.calcite.linq4j.function.Function1() {
> public org.apache.calcite.linq4j.AbstractEnumerable apply(final
> java.util.List corrList-1794407755) {
> ...
> Caused by: java.util.concurrent.ExecutionException:
> org.codehaus.commons.compiler.CompileException: Line 134, Column 103: One of
> ', )' expected instead of '-'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4652) AggregateExpandDistinctAggregatesRule must cast top aggregates to original type
[
https://issues.apache.org/jira/browse/CALCITE-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398286#comment-17398286
]
Haisheng Yuan edited comment on CALCITE-4652 at 8/12/21, 8:49 PM:
--
Fixed in
https://github.com/apache/calcite/commit/8a1535f94aad1e0ce77797eb84d75b4a5b1692c1.
Thanks for the PR, [~tledkov-gridgain].
was (Author: hyuan):
Fixed in
[https://github.com/apache/calcite/commit/8a1535f94aad1e0ce77797eb84d75b4a5b1692c1,]
thanks for the PR, [~tledkov-gridgain].
> AggregateExpandDistinctAggregatesRule must cast top aggregates to original
> type
> ---
>
> Key: CALCITE-4652
> URL: https://issues.apache.org/jira/browse/CALCITE-4652
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Taras Ledkov
>Assignee: Taras Ledkov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The query {{SELECT COUNT(comm), COUNT(DISTINCT comm) FROM emp}} produces the
> plan below:
> {code}
> LogicalAggregate(group=[{}], EXPR$0=[$SUM0($1)], EXPR$1=[COUNT($0)])
> LogicalAggregate(group=[{0}], EXPR$0=[COUNT()])
> LogicalProject(COMM=[$6])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> In case SUM type is expanded by override {{RelDataTypeSystem#deriveSumType}}
> the {{AggregateExpandDistinctAggregatesRule}} fails with assertion below:
> {code}
> Error:
> java.lang.AssertionError: type mismatch:
> aggCall type:
> BIGINT NOT NULL
> inferred type:
> DECIMAL(19, 0) NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2211)
> at
> org.apache.calcite.rel.core.Aggregate.typeMatchesInferred(Aggregate.java:443)
> at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:170)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.copy(LogicalAggregate.java:154)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.copy(LogicalAggregate.java:48)
> at
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule.convertSingletonDistinct(AggregateExpandDistinctAggregatesRule.java:401)
> ...
> {code}
> *Root cause:*
> the type of SUM aggregate doesn't match original type of the COUNT.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4652) AggregateExpandDistinctAggregatesRule must cast top aggregates to original type
[
https://issues.apache.org/jira/browse/CALCITE-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-4652.
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/8a1535f94aad1e0ce77797eb84d75b4a5b1692c1,]
thanks for the PR, [~tledkov-gridgain].
> AggregateExpandDistinctAggregatesRule must cast top aggregates to original
> type
> ---
>
> Key: CALCITE-4652
> URL: https://issues.apache.org/jira/browse/CALCITE-4652
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Taras Ledkov
>Assignee: Taras Ledkov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The query {{SELECT COUNT(comm), COUNT(DISTINCT comm) FROM emp}} produces the
> plan below:
> {code}
> LogicalAggregate(group=[{}], EXPR$0=[$SUM0($1)], EXPR$1=[COUNT($0)])
> LogicalAggregate(group=[{0}], EXPR$0=[COUNT()])
> LogicalProject(COMM=[$6])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> In case SUM type is expanded by override {{RelDataTypeSystem#deriveSumType}}
> the {{AggregateExpandDistinctAggregatesRule}} fails with assertion below:
> {code}
> Error:
> java.lang.AssertionError: type mismatch:
> aggCall type:
> BIGINT NOT NULL
> inferred type:
> DECIMAL(19, 0) NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2211)
> at
> org.apache.calcite.rel.core.Aggregate.typeMatchesInferred(Aggregate.java:443)
> at org.apache.calcite.rel.core.Aggregate.(Aggregate.java:170)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.(LogicalAggregate.java:72)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.copy(LogicalAggregate.java:154)
> at
> org.apache.calcite.rel.logical.LogicalAggregate.copy(LogicalAggregate.java:48)
> at
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule.convertSingletonDistinct(AggregateExpandDistinctAggregatesRule.java:401)
> ...
> {code}
> *Root cause:*
> the type of SUM aggregate doesn't match original type of the COUNT.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (CALCITE-4726) Add support for filters in AggregateExpandWithinDistinctRule
[
https://issues.apache.org/jira/browse/CALCITE-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan reopened CALCITE-4726:
> Add support for filters in AggregateExpandWithinDistinctRule
>
>
> Key: CALCITE-4726
> URL: https://issues.apache.org/jira/browse/CALCITE-4726
> Project: Calcite
> Issue Type: Improvement
>Reporter: Will Noble
>Assignee: Will Noble
>Priority: Blocker
> Fix For: 1.28.0
>
>
> Currently, {{AggregateExpandWithinDistinctRule}} does not fire if any of the
> aggregate calls in the aggregate have a filter. We should support filters.
> For example, the following query will not be handled by the current rule due
> to the {{FILTER}} clause:
> {code:java}
> SELECT deptno,
> SUM(sal),
> SUM(sal) WITHIN DISTINCT (job) FILTER (WHERE ename LIKE 'A%')
> FROM emp
> GROUP BY deptno
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (CALCITE-4726) Add support for filters in AggregateExpandWithinDistinctRule
[
https://issues.apache.org/jira/browse/CALCITE-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-4726:
---
Comment: was deleted
(was: Fixed in
[https://github.com/apache/calcite/commit/8a1535f94aad1e0ce77797eb84d75b4a5b1692c1,]
thanks for the PR.)
> Add support for filters in AggregateExpandWithinDistinctRule
>
>
> Key: CALCITE-4726
> URL: https://issues.apache.org/jira/browse/CALCITE-4726
> Project: Calcite
> Issue Type: Improvement
>Reporter: Will Noble
>Assignee: Will Noble
>Priority: Blocker
>
> Currently, {{AggregateExpandWithinDistinctRule}} does not fire if any of the
> aggregate calls in the aggregate have a filter. We should support filters.
> For example, the following query will not be handled by the current rule due
> to the {{FILTER}} clause:
> {code:java}
> SELECT deptno,
> SUM(sal),
> SUM(sal) WITHIN DISTINCT (job) FILTER (WHERE ename LIKE 'A%')
> FROM emp
> GROUP BY deptno
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4726) Add support for filters in AggregateExpandWithinDistinctRule
[
https://issues.apache.org/jira/browse/CALCITE-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-4726:
---
Fix Version/s: (was: 1.28.0)
> Add support for filters in AggregateExpandWithinDistinctRule
>
>
> Key: CALCITE-4726
> URL: https://issues.apache.org/jira/browse/CALCITE-4726
> Project: Calcite
> Issue Type: Improvement
>Reporter: Will Noble
>Assignee: Will Noble
>Priority: Blocker
>
> Currently, {{AggregateExpandWithinDistinctRule}} does not fire if any of the
> aggregate calls in the aggregate have a filter. We should support filters.
> For example, the following query will not be handled by the current rule due
> to the {{FILTER}} clause:
> {code:java}
> SELECT deptno,
> SUM(sal),
> SUM(sal) WITHIN DISTINCT (job) FILTER (WHERE ename LIKE 'A%')
> FROM emp
> GROUP BY deptno
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4726) Add support for filters in AggregateExpandWithinDistinctRule
[
https://issues.apache.org/jira/browse/CALCITE-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-4726.
Fix Version/s: 1.28.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/8a1535f94aad1e0ce77797eb84d75b4a5b1692c1,]
thanks for the PR.
> Add support for filters in AggregateExpandWithinDistinctRule
>
>
> Key: CALCITE-4726
> URL: https://issues.apache.org/jira/browse/CALCITE-4726
> Project: Calcite
> Issue Type: Improvement
>Reporter: Will Noble
>Assignee: Will Noble
>Priority: Blocker
> Fix For: 1.28.0
>
>
> Currently, {{AggregateExpandWithinDistinctRule}} does not fire if any of the
> aggregate calls in the aggregate have a filter. We should support filters.
> For example, the following query will not be handled by the current rule due
> to the {{FILTER}} clause:
> {code:java}
> SELECT deptno,
> SUM(sal),
> SUM(sal) WITHIN DISTINCT (job) FILTER (WHERE ename LIKE 'A%')
> FROM emp
> GROUP BY deptno
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4729) Column in CTE should be ambiguous
[
https://issues.apache.org/jira/browse/CALCITE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398146#comment-17398146
]
Haisheng Yuan commented on CALCITE-4729:
Yeah, makes sense.
> Column in CTE should be ambiguous
> -
>
> Key: CALCITE-4729
> URL: https://issues.apache.org/jira/browse/CALCITE-4729
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Lantao Jin
>Priority: Major
>
> {code}
> create table test (a int);
> with t as
> (
> select * from test t1 join test t2 on t1.a=t2.a
> )
> select t.a from t;
> {code}
> The above query fails with 't.a' is ambiguous in Spark/Presto/Postgres. But
> it succeeds in Calcite. Currently, many and many companies use Calcite as a
> query federation layer upon multiple query engines such as Spark and Presto.
> But this behavior confused users.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4712) Add RelHashDistribution
[
https://issues.apache.org/jira/browse/CALCITE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391968#comment-17391968
]
Haisheng Yuan commented on CALCITE-4712:
Yes, you are right. It is kind of randomize or round robin null values across
all the workers / computing units. We can use "randomizeNulls" instead.
> Add RelHashDistribution
> ---
>
> Key: CALCITE-4712
> URL: https://issues.apache.org/jira/browse/CALCITE-4712
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Add RelHashDistribution. The hash distribution should have the following
> properties:
> {code:java}
> 1. ImmutableIntList keys; // distribution keys
> 2. ImmutableList equivKeys; // equivalent keys for each
> distribution key
> 3. int bucketNum; // number of buckets or shards
> 4. boolean nullsColocated; // are NULLS colocated?
> 5. String hashFunc; // name or identity of hash function
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4712) Add RelHashDistribution
[
https://issues.apache.org/jira/browse/CALCITE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391838#comment-17391838
]
Haisheng Yuan commented on CALCITE-4712:
(1, null) and (2, null) will have different hash value, so when redistributing,
they will go to different bucket. The flag is useful for outer joins. For
example, "select * from foo outer join bar on foo.a = bar.a", bar.a may have a
lot of null value, a.k.a. skew value, when doing outer join, there will be long
tail on the worker that have a lot of bar.a=null values when null are
colocated. So instead we can request a hash distribution that does't colocate
null value, so that the computation are evenly distributed. For group by, the
hash distribution always requires "nullsColocated" to be true.
> Add RelHashDistribution
> ---
>
> Key: CALCITE-4712
> URL: https://issues.apache.org/jira/browse/CALCITE-4712
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Add RelHashDistribution. The hash distribution should have the following
> properties:
> {code:java}
> 1. ImmutableIntList keys; // distribution keys
> 2. ImmutableList equivKeys; // equivalent keys for each
> distribution key
> 3. int bucketNum; // number of buckets or shards
> 4. boolean nullsColocated; // are NULLS colocated?
> 5. String hashFunc; // name or identity of hash function
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4712) Add RelHashDistribution
[
https://issues.apache.org/jira/browse/CALCITE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391825#comment-17391825
]
Haisheng Yuan commented on CALCITE-4712:
Thanks for commenting, Julian.
* The class will implement interface RelDistribution. We will use
{{Type.HASH_DISTRIBUTED.}}
* "bucketCount" sounds good.
* "hashFunctionName" is also fine.
* ImmutableList is OK. I see RelDistributionImpl uses
ImmutableIntList, should we keep them consistent?
* I will add an example of "equivKeys".
* No, the nullls are in different columns.
> Add RelHashDistribution
> ---
>
> Key: CALCITE-4712
> URL: https://issues.apache.org/jira/browse/CALCITE-4712
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Add RelHashDistribution. The hash distribution should have the following
> properties:
> {code:java}
> 1. ImmutableIntList keys; // distribution keys
> 2. ImmutableList equivKeys; // equivalent keys for each
> distribution key
> 3. int bucketNum; // number of buckets or shards
> 4. boolean nullsColocated; // are NULLS colocated?
> 5. String hashFunc; // name or identity of hash function
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4712) Add RelHashDistribution
[
https://issues.apache.org/jira/browse/CALCITE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-4712:
---
Description:
Add RelHashDistribution. The hash distribution should have the following
properties:
{code:java}
1. ImmutableIntList keys; // distribution keys
2. ImmutableList equivKeys; // equivalent keys for each
distribution key
3. int bucketNum; // number of buckets or shards
4. boolean nullsColocated; // are NULLS colocated?
5. String hashFunc; // name or identity of hash function
{code}
was:
Add RelHashDistribution. The hash distribution should have the following
properties:
1. ImmutableIntList keys; // distribution keys
2. ImmutableList equivKeys; // equivalent keys for each
distribution key
3. int bucketNum; // number of buckets or shards
4. boolean nullsColocated; // are NULLS colocated?
5. String hashFunc; // name or identity of hash function
> Add RelHashDistribution
> ---
>
> Key: CALCITE-4712
> URL: https://issues.apache.org/jira/browse/CALCITE-4712
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Add RelHashDistribution. The hash distribution should have the following
> properties:
> {code:java}
> 1. ImmutableIntList keys; // distribution keys
> 2. ImmutableList equivKeys; // equivalent keys for each
> distribution key
> 3. int bucketNum; // number of buckets or shards
> 4. boolean nullsColocated; // are NULLS colocated?
> 5. String hashFunc; // name or identity of hash function
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4712) Add RelHashDistribution
Haisheng Yuan created CALCITE-4712: -- Summary: Add RelHashDistribution Key: CALCITE-4712 URL: https://issues.apache.org/jira/browse/CALCITE-4712 Project: Calcite Issue Type: Bug Components: core Reporter: Haisheng Yuan Add RelHashDistribution. The hash distribution should have the following properties: 1. ImmutableIntList keys; // distribution keys 2. ImmutableList equivKeys; // equivalent keys for each distribution key 3. int bucketNum; // number of buckets or shards 4. boolean nullsColocated; // are NULLS colocated? 5. String hashFunc; // name or identity of hash function -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4705) calcite should support hint like /*+ skewjoin(a(c0, c1)) */
[
https://issues.apache.org/jira/browse/CALCITE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388412#comment-17388412
]
Haisheng Yuan commented on CALCITE-4705:
Yes, +1 for this hint.
> calcite should support hint like /*+ skewjoin(a(c0, c1)) */
> ---
>
> Key: CALCITE-4705
> URL: https://issues.apache.org/jira/browse/CALCITE-4705
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: jibiyr
>Assignee: jibiyr
>Priority: Minor
> Fix For: 1.28.0
>
>
> Dear community,
> calcite parse support hint now,but it can not parse hint like follow,
> _*select /*+ skewjoin(a(c0, c1)) */ * from T0 a join T1 b on a.c0 = b.c0,*_
> this is the bnf which is calcite supported now,
> {code:java}
> //
> hint:
> hintName
> | hintName '(' hintOptions ')'
> hintOptions:
> hintKVOption [, hintKVOption ]*
> | optionName [, optionName ]*
> | optionValue [, optionValue ]*
> hintKVOption:
> optionName '=' stringLiteral
> | stringLiteral '=' stringLiteral
> optionValue:
> stringLiteral
> | numericLiteral
> {code}
> I think the hintOptions should support the expression pattern,like a(c0,c1).
> And this pattern can be parsed by spark/presto now.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4704) Log produced plan after rule application using explain formatting
[
https://issues.apache.org/jira/browse/CALCITE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388245#comment-17388245
]
Haisheng Yuan commented on CALCITE-4704:
For complicated queries, i.e. tpcds queries, the proposed change to log the
whole tree will just print out much more useless tree nodes. So I am against
the change.
However making the log format and information more readable is worth doing.
> Log produced plan after rule application using explain formatting
> -
>
> Key: CALCITE-4704
> URL: https://issues.apache.org/jira/browse/CALCITE-4704
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In many cases, we want to identify which rule lead to a certain
> transformation in the plan or need to observe how the query plan evolves by
> applying some rules in order to fix some bug or find the right place to
> introduce another optimization step.
> Currently there are some logs during the application of a rule triggered by
> the
> [HepPlanner|https://github.com/apache/calcite/blob/e04f3b08dcfb6910ff4df3810772c346b25ed424/core/src/main/java/org/apache/calcite/plan/AbstractRelOptPlanner.java#L367]
> and
> [VolcanoPlanner|https://github.com/apache/calcite/blob/e04f3b08dcfb6910ff4df3810772c346b25ed424/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoRuleCall.java#L126]
> but they more or less display only the top operator of the transformation
> and not the whole subtree. Moreover, the {{RelNode#toString}} used in these
> logs, is not self-contained, so in order to check the transformation the log
> entry needs to be matched with other logs.
> It would help if instead of displaying only the top operator we logged the
> equivalent of explain (i.e., {{RelOptUtil.toString}}) on the transformed
> sub-tree.
> You can find below some extracts from the current logs and how the proposed
> log could look like.
> *HepPlanner*
> +Current logging+
> {noformat}
> 2021-07-27 14:37:58,252 [ForkJoinPool-1-worker-9] DEBUG - call#0: Rule
> FilterIntoJoinRule arguments
> [rel#9:LogicalFilter.NONE.[](input=HepRelVertex#8,condition==($7, $8)),
> rel#7:LogicalJoin.NONE.[](left=HepRelVertex#5,right=HepRelVertex#6,condition=true,joinType=left)]
> produced
> rel#14:LogicalProject.NONE.[](input=LogicalJoin#13,inputs=0..7,exprs=[CAST($8):TINYINT,
> $9, $10])
> 2021-07-27 14:37:58,266 [ForkJoinPool-1-worker-9] DEBUG - call#1: Rule
> ProjectJoinTransposeRule arguments
> [rel#16:LogicalProject.NONE.[](input=HepRelVertex#15,inputs=0..7,exprs=[CAST($8):TINYINT,
> $9, $10]),
> rel#13:LogicalJoin.NONE.[](left=HepRelVertex#5,right=HepRelVertex#6,condition==($7,
> $8),joinType=inner)] produced
> rel#21:LogicalProject.NONE.[](input=LogicalJoin#20,inputs=0..7,exprs=[$11,
> $9, $10])
> 2021-07-27 14:37:58,269 [ForkJoinPool-1-worker-9] DEBUG - call#3: Rule
> ProjectMergeRule arguments
> [rel#11:LogicalProject.NONE.[](input=HepRelVertex#27,exprs=[$1]),
> rel#26:LogicalProject.NONE.[](input=HepRelVertex#25,inputs=0..7,exprs=[$11,
> $9, $10])] produced
> rel#28:LogicalProject.NONE.[](input=HepRelVertex#25,exprs=[$1])
> 2021-07-27 14:37:58,272 [ForkJoinPool-1-worker-9] DEBUG - call#4: Rule
> ProjectJoinTransposeRule arguments
> [rel#28:LogicalProject.NONE.[](input=HepRelVertex#25,exprs=[$1]),
> rel#24:LogicalJoin.NONE.[](left=HepRelVertex#22,right=HepRelVertex#23,condition==($7,
> $8),joinType=inner)] produced
> rel#33:LogicalProject.NONE.[](input=LogicalJoin#32,inputs=0)
> 2021-07-27 14:37:58,274 [ForkJoinPool-1-worker-9] DEBUG - call#6: Rule
> ProjectMergeRule arguments
> [rel#30:LogicalProject.NONE.[](input=HepRelVertex#22,exprs=[$1, $7]),
> rel#18:LogicalProject.NONE.[0](input=HepRelVertex#5,inputs=0..7)] produced
> rel#40:LogicalProject.NONE.[](input=HepRelVertex#5,exprs=[$1, $7])
> 2021-07-27 14:37:58,275 [ForkJoinPool-1-worker-9] DEBUG - call#7: Rule
> ProjectMergeRule arguments
> [rel#31:LogicalProject.NONE.[0](input=HepRelVertex#23,inputs=0),
> rel#19:LogicalProject.NONE.[[0],
> [3]](input=HepRelVertex#6,inputs=0..2,exprs=[CAST($0):TINYINT])] produced
> rel#42:LogicalProject.NONE.[0](input=HepRelVertex#6,inputs=0)
> {noformat}
> +Proposed logging+
> {noformat}
> 2021-07-27 14:37:58,260 [ForkJoinPool-1-worker-9] DEBUG - Rule
> FilterIntoJoinRule produced:
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], DEPTNO0=[CAST($8):TINYINT], DNAME=[$9],
> LOC=[$10])
> LogicalJoin(condition=[=($7, $8)], joinType=[inner])
>
[jira] [Commented] (CALCITE-4704) Log produced plan after rule application using explain formatting
[
https://issues.apache.org/jira/browse/CALCITE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388241#comment-17388241
]
Haisheng Yuan commented on CALCITE-4704:
I would suggest logging the matched partial tree only, instead of the whole
plan tree or the sub-trees. Otherwise, for each rule, we will see the plan tree
till the leaf node, even the leaf nodes don't participate the rule matching.
FYI, here is Orca optimizer's logging for transformation rule's result:
{code:java}
2021-07-27 11:22:53:888062 PDT,THD000,TRACE,"Xform: CXformGbAgg2StreamAgg
Input:
+--CLogicalGbAgg( Global ) Grp Cols: ["c" (9)][Global], Minimal Grp Cols: ["c"
(9)], Generates Duplicates :[ 1 ]rows:1 width:12 rebinds:1 origin:
[Grp:9, GrpExpr:1]
|--CLogicalGbAgg( Local ) Grp Cols: ["c" (9)][Local], Minimal Grp Cols: ["c"
(9)], Generates Duplicates :[ 1 ]rows:1 width:12 rebinds:1 origin:
[Grp:13, GrpExpr:0]
+--CScalarProjectList origin: [Grp:17, GrpExpr:0]
Output:
Alternatives:
0:
+--CPhysicalStreamAgg( Global, multi-stage ) Grp Cols: ["c" (9)], Minimal Grp
Cols:["c" (9)], Generates Duplicates :[ 1 ]cost:-0.50
|--CLogicalGbAgg( Local ) Grp Cols: ["c" (9)][Local], Minimal Grp Cols: ["c"
(9)], Generates Duplicates :[ 1 ]rows:1 width:12 rebinds:1 origin:
[Grp:13, GrpExpr:0]
+--CScalarProjectList origin: [Grp:17, GrpExpr:0]
{code}
> Log produced plan after rule application using explain formatting
> -
>
> Key: CALCITE-4704
> URL: https://issues.apache.org/jira/browse/CALCITE-4704
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.27.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In many cases, we want to identify which rule lead to a certain
> transformation in the plan or need to observe how the query plan evolves by
> applying some rules in order to fix some bug or find the right place to
> introduce another optimization step.
> Currently there are some logs during the application of a rule triggered by
> the
> [HepPlanner|https://github.com/apache/calcite/blob/e04f3b08dcfb6910ff4df3810772c346b25ed424/core/src/main/java/org/apache/calcite/plan/AbstractRelOptPlanner.java#L367]
> and
> [VolcanoPlanner|https://github.com/apache/calcite/blob/e04f3b08dcfb6910ff4df3810772c346b25ed424/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoRuleCall.java#L126]
> but they more or less display only the top operator of the transformation
> and not the whole subtree. Moreover, the {{RelNode#toString}} used in these
> logs, is not self-contained, so in order to check the transformation the log
> entry needs to be matched with other logs.
> It would help if instead of displaying only the top operator we logged the
> equivalent of explain (i.e., {{RelOptUtil.toString}}) on the transformed
> sub-tree.
> You can find below some extracts from the current logs and how the proposed
> log could look like.
> *HepPlanner*
> +Current logging+
> {noformat}
> 2021-07-27 14:37:58,252 [ForkJoinPool-1-worker-9] DEBUG - call#0: Rule
> FilterIntoJoinRule arguments
> [rel#9:LogicalFilter.NONE.[](input=HepRelVertex#8,condition==($7, $8)),
> rel#7:LogicalJoin.NONE.[](left=HepRelVertex#5,right=HepRelVertex#6,condition=true,joinType=left)]
> produced
> rel#14:LogicalProject.NONE.[](input=LogicalJoin#13,inputs=0..7,exprs=[CAST($8):TINYINT,
> $9, $10])
> 2021-07-27 14:37:58,266 [ForkJoinPool-1-worker-9] DEBUG - call#1: Rule
> ProjectJoinTransposeRule arguments
> [rel#16:LogicalProject.NONE.[](input=HepRelVertex#15,inputs=0..7,exprs=[CAST($8):TINYINT,
> $9, $10]),
> rel#13:LogicalJoin.NONE.[](left=HepRelVertex#5,right=HepRelVertex#6,condition==($7,
> $8),joinType=inner)] produced
> rel#21:LogicalProject.NONE.[](input=LogicalJoin#20,inputs=0..7,exprs=[$11,
> $9, $10])
> 2021-07-27 14:37:58,269 [ForkJoinPool-1-worker-9] DEBUG - call#3: Rule
> ProjectMergeRule arguments
> [rel#11:LogicalProject.NONE.[](input=HepRelVertex#27,exprs=[$1]),
> rel#26:LogicalProject.NONE.[](input=HepRelVertex#25,inputs=0..7,exprs=[$11,
> $9, $10])] produced
> rel#28:LogicalProject.NONE.[](input=HepRelVertex#25,exprs=[$1])
> 2021-07-27 14:37:58,272 [ForkJoinPool-1-worker-9] DEBUG - call#4: Rule
> ProjectJoinTransposeRule arguments
> [rel#28:LogicalProject.NONE.[](input=HepRelVertex#25,exprs=[$1]),
> rel#24:LogicalJoin.NONE.[](left=HepRelVertex#22,right=HepRelVertex#23,condition==($7,
> $8),joinType=inner)] produced
> rel#33:LogicalProject.NONE.[](input=LogicalJoin#32,inputs=0)
> 2021-07-27 14:37:58,274 [ForkJoinPool-1-worker-9] DEBUG - call#6: Rule
>
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387545#comment-17387545 ] Haisheng Yuan commented on CALCITE-4677: [~chenz...@mail2.sysu.edu.com] sorry to reply you late. Is the new physical node generated by passThrough or by derive? That makes a difference. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When we enable top-down optimization, the VolcanoPlanner couldn't be > completely compatible with the old one. > Especially when we using rules matching RelNodes which convention is not > NONE, like rules that want to convert different conventions between physical > node or some rules that want to do some optimization for serval physical > nodes. It could be found that rules can't fire, because in top-down > optimization physical nodes can only be created by implementation > rules(logical node -> physical node) but not rules matching physical > node(physical node -> physical node). Try to use enforcer instead of rules to > convert node between different conventions. > > Example: > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3181) Support limit per group in Window
[
https://issues.apache.org/jira/browse/CALCITE-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387518#comment-17387518
]
Haisheng Yuan commented on CALCITE-3181:
Yeah, with "row()" the transformation should work. Thanks for the example.
> Support limit per group in Window
> -
>
> Key: CALCITE-3181
> URL: https://issues.apache.org/jira/browse/CALCITE-3181
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> We have a lot of queries like the following to retrieve top N tuples per
> group:
> {code:java}
> SELECT x, y FROM
> (SELECT x, y, ROW_NUMBER() OVER (PARTITION BY x ORDER BY y)
> AS rn FROM t1) t2 WHERE rn <= 3;
> {code}
> The performance is not good if each group has a lot more tuples than wanted,
> because we will retrieve and sort all the tuples, instead of just doing a
> top-N heap sort.
> In order to do optimization for this kind of query, we need to extend window
> to support limit, if and only if there is only 1 window function, and it is
> {{row_number()}}. We also need a substitute rule to push the limit into
> window. Of course, we also need to modify executor to support this
> optimization (can be later).
> {code:java}
> Filter (rn <= 3)
> +- Window (window#0={Partition by x order by y ROW_NUMBER()})
> {code}
> to
> {code:java}
> Filter (rn <= 3)
> +- Window (window#0={Partition by x order by y limit 3 ROW_NUMBER()})
> {code}
> Thoughts? Objections?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4681) Rule operand match specific shape with or without RelSubset.class/RelNode.class
[ https://issues.apache.org/jira/browse/CALCITE-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377858#comment-17377858 ] Haisheng Yuan commented on CALCITE-4681: RelSubset can just work as a replacement. It almost equals placeholder, at least for ordinary logical operators. > Rule operand match specific shape with or without > RelSubset.class/RelNode.class > --- > > Key: CALCITE-4681 > URL: https://issues.apache.org/jira/browse/CALCITE-4681 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: ZiLin Chen >Priority: Major > > If we want to match such a pattern, LogicalJoin1 with left Input XXX(original > LogicalJoin1 left input) and with right input (LogicalJoin.class). we wound > find that XXX we should use RelNode.class or RelSubset.class. However both > RelNode.class and RelSubset.class will match all kind of traitSet which is > inefficient（rule may fires multi time just because of some kind of XXX with > different trait）, especially when this is a JoinReorder Rule. > The XXX operand we want to match is exact what LogicalJoin1.getLeft() return. > LogicalJoin1 > - XXX(original LogicalJoin1 left input) > - LogicalJoin2 > > Is that any way we can provide to solve this problem? > One way maybe changing what RelSubset.class operand match meaning? (Now, > there is no rule in calcite with RelSubset.class as match operand. Before > JoinAssociateRule is the only one match RelSubset.class, but change later ) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3181) Support limit per group in Window
[
https://issues.apache.org/jira/browse/CALCITE-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377857#comment-17377857
]
Haisheng Yuan commented on CALCITE-3181:
Yeah, that makes more sense.
How does it work for the following case?
SELECT a, b, ROW_NUMBER() OVER (PARTITION BY x ORDER BY y)...
here the value of "a" and "b" may vary row by row.
> Support limit per group in Window
> -
>
> Key: CALCITE-3181
> URL: https://issues.apache.org/jira/browse/CALCITE-3181
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> We have a lot of queries like the following to retrieve top N tuples per
> group:
> {code:java}
> SELECT x, y FROM
> (SELECT x, y, ROW_NUMBER() OVER (PARTITION BY x ORDER BY y)
> AS rn FROM t1) t2 WHERE rn <= 3;
> {code}
> The performance is not good if each group has a lot more tuples than wanted,
> because we will retrieve and sort all the tuples, instead of just doing a
> top-N heap sort.
> In order to do optimization for this kind of query, we need to extend window
> to support limit, if and only if there is only 1 window function, and it is
> {{row_number()}}. We also need a substitute rule to push the limit into
> window. Of course, we also need to modify executor to support this
> optimization (can be later).
> {code:java}
> Filter (rn <= 3)
> +- Window (window#0={Partition by x order by y ROW_NUMBER()})
> {code}
> to
> {code:java}
> Filter (rn <= 3)
> +- Window (window#0={Partition by x order by y limit 3 ROW_NUMBER()})
> {code}
> Thoughts? Objections?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377705#comment-17377705 ] Haisheng Yuan commented on CALCITE-4677: Rule matches on physical operators are prohibited. You said you have some rules to only match physical nodes, but that is not the reason that we should support. You didn't provide any rationale to justify that the problem you are dealing with can only be solved by rules to match physical relnodes. Completely compatible with old volcano optimizer is not the goal. > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3181) Support limit per group in Window
[
https://issues.apache.org/jira/browse/CALCITE-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377683#comment-17377683
]
Haisheng Yuan commented on CALCITE-3181:
Yes, they are quite similar with each other.
There are still some slight difference, the window keep the row count to the
limit, but the aggregate in CALCITE-4687 only generate 1 row per group, no
matter with or without the limit. So maybe not exchangable?
> Support limit per group in Window
> -
>
> Key: CALCITE-3181
> URL: https://issues.apache.org/jira/browse/CALCITE-3181
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> We have a lot of queries like the following to retrieve top N tuples per
> group:
> {code:java}
> SELECT x, y FROM
> (SELECT x, y, ROW_NUMBER() OVER (PARTITION BY x ORDER BY y)
> AS rn FROM t1) t2 WHERE rn <= 3;
> {code}
> The performance is not good if each group has a lot more tuples than wanted,
> because we will retrieve and sort all the tuples, instead of just doing a
> top-N heap sort.
> In order to do optimization for this kind of query, we need to extend window
> to support limit, if and only if there is only 1 window function, and it is
> {{row_number()}}. We also need a substitute rule to push the limit into
> window. Of course, we also need to modify executor to support this
> optimization (can be later).
> {code:java}
> Filter (rn <= 3)
> +- Window (window#0={Partition by x order by y ROW_NUMBER()})
> {code}
> to
> {code:java}
> Filter (rn <= 3)
> +- Window (window#0={Partition by x order by y limit 3 ROW_NUMBER()})
> {code}
> Thoughts? Objections?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4681) Rule operand match specific shape with or without RelSubset.class/RelNode.class
[ https://issues.apache.org/jira/browse/CALCITE-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377666#comment-17377666 ] Haisheng Yuan commented on CALCITE-4681: The problem of matching based on traits is that it will match all the operators matching the traits, hence cause more rule matches. I guess what zilin wants is a placeholer, any logical operator is fine, but just match once. I agree that place holder operand is nice to have. > Rule operand match specific shape with or without > RelSubset.class/RelNode.class > --- > > Key: CALCITE-4681 > URL: https://issues.apache.org/jira/browse/CALCITE-4681 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: ZiLin Chen >Priority: Major > > If we want to match such a pattern, LogicalJoin1 with left Input XXX(original > LogicalJoin1 left input) and with right input (LogicalJoin.class). we wound > find that XXX we should use RelNode.class or RelSubset.class. However both > RelNode.class and RelSubset.class will match all kind of traitSet which is > inefficient（rule may fires multi time just because of some kind of XXX with > different trait）, especially when this is a JoinReorder Rule. > The XXX operand we want to match is exact what LogicalJoin1.getLeft() return. > LogicalJoin1 > - XXX(original LogicalJoin1 left input) > - LogicalJoin2 > > Is that any way we can provide to solve this problem? > One way maybe changing what RelSubset.class operand match meaning? (Now, > there is no rule in calcite with RelSubset.class as match operand. Before > JoinAssociateRule is the only one match RelSubset.class, but change later ) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4681) Rule operand match specific shape with or without RelSubset.class/RelNode.class
[ https://issues.apache.org/jira/browse/CALCITE-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377558#comment-17377558 ] Haisheng Yuan commented on CALCITE-4681: [~chenz...@mail2.sysu.edu.com] Use RelSubset with the predicate of Convention == NONE, so that it matches logical operator. In case the child is a logical sort, you are fine, it won't increase the rule match a lot. > Rule operand match specific shape with or without > RelSubset.class/RelNode.class > --- > > Key: CALCITE-4681 > URL: https://issues.apache.org/jira/browse/CALCITE-4681 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: ZiLin Chen >Priority: Major > > If we want to match such a pattern, LogicalJoin1 with left Input XXX(original > LogicalJoin1 left input) and with right input (LogicalJoin.class). we wound > find that XXX we should use RelNode.class or RelSubset.class. However both > RelNode.class and RelSubset.class will match all kind of traitSet which is > inefficient（rule may fires multi time just because of some kind of XXX with > different trait）, especially when this is a JoinReorder Rule. > The XXX operand we want to match is exact what LogicalJoin1.getLeft() return. > LogicalJoin1 > - XXX(original LogicalJoin1 left input) > - LogicalJoin2 > > Is that any way we can provide to solve this problem? > One way maybe changing what RelSubset.class operand match meaning? (Now, > there is no rule in calcite with RelSubset.class as match operand. Before > JoinAssociateRule is the only one match RelSubset.class, but change later ) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4650) TopDownRuleDriver.DeriveTrait.derive produces wrong AssertionError when registering new nodes
[
https://issues.apache.org/jira/browse/CALCITE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-4650:
---
Description:
{{TopDownRuleDriver.DeriveTrait.derive}} asserts that a derived node belongs to
the same {{RelSet}} as the original node.
If the original node {{N1}} refers to set {{S1}}, the derived node {{N2}} might
be registered in the set {{S2}}, if {{S1}} was merged to {{S2}} between the
registration of {{N1}} and derivation of {{N2}}.
Currently, we compare the original set of the original node {{N1}} with the
canonical set of the derived node {{N2}}. This leads to false assertions.
Dicussion:
https://lists.apache.org/thread.html/rd26b7a0f55906cebf5abcaf349ab2bec414e164fa4d9f63789bf34c8%40%3Cdev.calcite.apache.org%3E
was:
{{TopDownRuleDriver.DeriveTrait.derive}} asserts that a derived node belongs to
the same {{RelSet}} as the original node.
If the original node {{N1}} refers to set {{S1}}, the derived node {{N2}} might
be registered in the set {{S2}}, if {{S1}} was merged to {{S2}} between the
registration of {{N1}} and derivation of {{N2}}.
Currently, we compare the original set of the original node {{N1}} with the
canonical set of the derived node {{N2}}. This leads to false assertions.
> TopDownRuleDriver.DeriveTrait.derive produces wrong AssertionError when
> registering new nodes
> -
>
> Key: CALCITE-4650
> URL: https://issues.apache.org/jira/browse/CALCITE-4650
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.28.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{TopDownRuleDriver.DeriveTrait.derive}} asserts that a derived node belongs
> to the same {{RelSet}} as the original node.
> If the original node {{N1}} refers to set {{S1}}, the derived node {{N2}}
> might be registered in the set {{S2}}, if {{S1}} was merged to {{S2}} between
> the registration of {{N1}} and derivation of {{N2}}.
> Currently, we compare the original set of the original node {{N1}} with the
> canonical set of the derived node {{N2}}. This leads to false assertions.
> Dicussion:
> https://lists.apache.org/thread.html/rd26b7a0f55906cebf5abcaf349ab2bec414e164fa4d9f63789bf34c8%40%3Cdev.calcite.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-4677: --- Description: Whenever new physical nodes are on produce, we need to fire rules in case of different physical conventions conversion. DeriveTrait Task deal with this case, while some physical node maybe missing. There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. 1. LogicalSortToTopNRule 2. LogicalSortToLimitMemSortRule 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) LogicalAggreate -> 1. RelSet1 - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and LogicalSortToLimitMemSortRule - OtherNode -> 3. RelSet3 ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. ApplyRuleTask LogicalSortToTopNRule still on the stack. when RelSet3 on Merge (Merge with other RelSets), then RelSet2 clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): subset.resetTaskState() because RelSet1 is root so OptimizeGroup will be scheduled again Now the task stack Task Stack: Task about RelSet1 Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule Task about RelSet3 Task about RelSet1 root schedule again Task about RelSet2 Task about RelSet3 (current task) next Task Stack: Task about RelSet1 Task about RelSet2 Task about RelSet3 Task about RelSet1 Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on RelSet (LogicalSort and LimitMemSort) next... Task Stack: Task about RelSet1 Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. Here we found when TopN is on produce, every RelSubSet can not fire OptimizeInputTask. so TopNToSomeRelNodeRule can not fire was: Whenever new physical nodes are on produce, we need to fire rules in case of different physical conventions conversion. DeriveTrait Task deal with this case, while some physical node maybe missing. There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. 1. LogicalSortToTopNRuleLogicalSortToTopNRule 2. LogicalSortToLimitMemSortRule 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) LogicalAggreate -> 1. RelSet1 - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and LogicalSortToLimitMemSortRule - OtherNode -> 3. RelSet3 ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. ApplyRuleTask LogicalSortToTopNRule still on the stack. when RelSet3 on Merge (Merge with other RelSets), then RelSet2 clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): subset.resetTaskState() because RelSet1 is root so OptimizeGroup will be scheduled again Now the task stack Task Stack: Task about RelSet1 Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule Task about RelSet3 Task about RelSet1 root schedule again Task about RelSet2 Task about RelSet3 (current task) next Task Stack: Task about RelSet1 Task about RelSet2 Task about RelSet3 Task about RelSet1 Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on RelSet (LogicalSort and LimitMemSort) next... Task Stack: Task about RelSet1 Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. Here we found when TopN is on produce, every RelSubSet can not fire OptimizeInputTask. so TopNToSomeRelNodeRule can not fire > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState()
[jira] [Commented] (CALCITE-4677) Different physical conventions conversion on new physical node producing
[ https://issues.apache.org/jira/browse/CALCITE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376239#comment-17376239 ] Haisheng Yuan commented on CALCITE-4677: We prohibited rule matching on physical operators on purpose. If you want to convert between physical conventions, try enforcer instead of rules. Is there any specific case that justify your rationale? > Different physical conventions conversion on new physical node producing > - > > Key: CALCITE-4677 > URL: https://issues.apache.org/jira/browse/CALCITE-4677 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.28.0 >Reporter: ZiLin Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Whenever new physical nodes are on produce, we need to fire rules in case of > different physical conventions conversion. > DeriveTrait Task deal with this case, while some physical node maybe missing. > > There 3 rules, but in TopDownRuleDriver rule 3 can not be applied. > 1. LogicalSortToTopNRuleLogicalSortToTopNRule > 2. LogicalSortToLimitMemSortRule > 3. TopNToSomeRelNodeRule (Need Match Physical Convention Node like TopN) > > LogicalAggreate -> 1. RelSet1 > - LogicalSort -> 2. RelSet2 fires LogicalSortToTopNRule and > LogicalSortToLimitMemSortRule > - OtherNode -> 3. RelSet3 > > ApplyRuleTask LogicalSortToLimitMemSortRule is already scheduled. > ApplyRuleTask LogicalSortToTopNRule still on the stack. > > when RelSet3 on Merge (Merge with other RelSets), then RelSet2 > clearProcessed(): subset.resetTaskState() and then RelSet1 clearProcessed(): > subset.resetTaskState() > because RelSet1 is root so OptimizeGroup will be scheduled again > > Now the task stack > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule > Task about RelSet3 > Task about RelSet1 root schedule again > Task about RelSet2 > Task about RelSet3 (current task) > > next > Task Stack: > Task about RelSet1 > Task about RelSet2 > Task about RelSet3 > Task about RelSet1 > Task about RelSet2 RelSubSet setOptimized so taskState -> Completed. rels on > RelSet (LogicalSort and LimitMemSort) > > next... > Task Stack: > Task about RelSet1 > Task about RelSet2 ApplyRuleTask LogicalSortToTopNRule. > Here we found when TopN is on produce, every RelSubSet can not fire > OptimizeInputTask. so TopNToSomeRelNodeRule can not fire -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4638) Volcano top-down optimizer failed to recognize transformation rule correctly
[ https://issues.apache.org/jira/browse/CALCITE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-4638. Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/40baa0056980ddbd4fcd4e957f59d3b12d86e739. Thanks for the PR, [~vozerov]. > Volcano top-down optimizer failed to recognize transformation rule correctly > - > > Key: CALCITE-4638 > URL: https://issues.apache.org/jira/browse/CALCITE-4638 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.27.0 >Reporter: Haisheng Yuan >Assignee: Vladimir Ozerov >Priority: Major > Labels: pull-request-available > Fix For: 1.28.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > In VolcanoPlanner.isTransformationRule, if the rule is a physical > implementation rule, it may be recognized as a transformation rule > incorrectly. > See discussion in mailing list: > https://lists.apache.org/thread.html/r362da53656b80b727c72cc6e4b807c5b1eccb15a0aa6fec5f6bbe91e%40%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4638) Volcano top-down optimizer failed to recognize transformation rule correctly
[ https://issues.apache.org/jira/browse/CALCITE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357728#comment-17357728 ] Haisheng Yuan commented on CALCITE-4638: Yes, I agree, we need to improve that too. > Volcano top-down optimizer failed to recognize transformation rule correctly > - > > Key: CALCITE-4638 > URL: https://issues.apache.org/jira/browse/CALCITE-4638 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Haisheng Yuan >Priority: Major > > In VolcanoPlanner.isTransformationRule, if the rule is a physical > implementation rule, it may be recognized as a transformation rule > incorrectly. > See discussion in mailing list: > https://lists.apache.org/thread.html/r362da53656b80b727c72cc6e4b807c5b1eccb15a0aa6fec5f6bbe91e%40%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4638) Volcano top-down optimizer failed to recognize transformation rule correctly
Haisheng Yuan created CALCITE-4638: -- Summary: Volcano top-down optimizer failed to recognize transformation rule correctly Key: CALCITE-4638 URL: https://issues.apache.org/jira/browse/CALCITE-4638 Project: Calcite Issue Type: Bug Components: core Reporter: Haisheng Yuan In VolcanoPlanner.isTransformationRule, if the rule is a physical implementation rule, it may be recognized as a transformation rule incorrectly. See discussion in mailing list: https://lists.apache.org/thread.html/r362da53656b80b727c72cc6e4b807c5b1eccb15a0aa6fec5f6bbe91e%40%3Cdev.calcite.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4634) Improve AggregateProjectPullUpConstantsRule to remove all constant keys
[
https://issues.apache.org/jira/browse/CALCITE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356914#comment-17356914
]
Haisheng Yuan commented on CALCITE-4634:
I doubt whether it is worth the extra optimization for all constant keys.
> Improve AggregateProjectPullUpConstantsRule to remove all constant keys
> ---
>
> Key: CALCITE-4634
> URL: https://issues.apache.org/jira/browse/CALCITE-4634
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.16.0
>Reporter: Yingyu
>Priority: Major
>
> Optimizer rule *AggregateProjectPullUpConstantsRule* removes constant keys
> from an Aggregate. However, this rules never removes the last column, because
> Aggregate([]) returns 1 row even if its input is empty.
> An enhancement could be done to AggregateProjectPullUpConstantsRule so that
> it removes all constant keys:
> - Remove all constant keys from an Aggregate
> - (Only)When no key is left from previous step (Aggregate becomes
> Aggregate([])), add a filter to the Aggregate([]) so that it returns empty
> dataset if its input is empty.
> E.g.
> {noformat}
> SELECT 'abc' FROM table_name GROUP BY ('abc')
> {noformat}
> will be translated to:
> {noformat}
> SELECT 'abc' FROM table_name GROUP BY () having count(*)>0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-853) EnumerableAggregate should take advantage of input collation
[ https://issues.apache.org/jira/browse/CALCITE-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-853. --- Fix Version/s: 1.23.0 Resolution: Fixed We already have it by adding EnumerableSortedAggregate in v1.23.0. > EnumerableAggregate should take advantage of input collation > > > Key: CALCITE-853 > URL: https://issues.apache.org/jira/browse/CALCITE-853 > Project: Calcite > Issue Type: Improvement >Reporter: liyang >Priority: Major > Fix For: 1.23.0 > > > Li Yang > Aug 20 (2 days ago) > > I encountered Out Of Mem exception when a huge result set is passed into > EnumerableAggregate and get aggregated in memory. I'm thinking if the input > is sorted by the group-by key, then the groupBy() don't have to hold all data > in memory any more. > Julian Hyde > 2:20 PM (16 hours ago) > > Yes, that would be useful. Please log a jira. > Enumerable.groupBy doesn't know its input's collation so can't make that > decision, but EnumerableAggregate does. I think that EnumerableAggregate > should have a "trigger key", a subset of its group key, and if the trigger > key changes it will emit and flush its hash table. > As well as for your use case, it will be useful for streaming queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4549) IndexOutOfBoundsException when group view by a sub query
[
https://issues.apache.org/jira/browse/CALCITE-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326185#comment-17326185
]
Haisheng Yuan commented on CALCITE-4549:
I don't remember we had seen issue with IN query like this. Possibly a
regression introduced by CALCITE-3456. [~danny0405] Can you help with this?
> IndexOutOfBoundsException when group view by a sub query
>
>
> Key: CALCITE-4549
> URL: https://issues.apache.org/jira/browse/CALCITE-4549
> Project: Calcite
> Issue Type: Bug
>Reporter: jinfeng
>Priority: Major
>
> It can be reproduced by the following code
> Add test in `SqlToConverterTest`
> {code:java}
> //代码占位符
> @Test void testGroupView() {
> final String sql = "SELECT " +
> "case when ENAME in( 'a', 'b') then 'c' else 'd' end " +
> "from EMP_20 " +
> "group by " +
> "case when ENAME in( 'a', 'b') then 'c' else 'd' end ";
> sql(sql).ok();
> }
> {code}
>
> The error is
> {code:java}
> //代码占位符
> Index: 0, Size: 0Index: 0, Size: 0
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at
> java.util.ArrayList.rangeCheck(ArrayList.java:659) at
> java.util.ArrayList.get(ArrayList.java:435) at
> org.apache.calcite.sql2rel.SqlToRelConverter$LookupContext.findRel(SqlToRelConverter.java:5849)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookup(SqlToRelConverter.java:4847)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookupExp(SqlToRelConverter.java:4791)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:4099)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$2500(SqlToRelConverter.java:224)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:5240)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4485)
> at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:325) at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:5087)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.substituteSubQuery(SqlToRelConverter.java:1138)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.replaceSubQueries(SqlToRelConverter.java:1091)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:3273)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:3140)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:702)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:664)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3537)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:589)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:618)
> at
> org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:778)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:4259)
> at
> org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:4251)
> at
> org.apache.calcite.test.SqlToRelConverterTest.testGroupView(SqlToRelConverterTest.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
> at
> org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
> at
>
[jira] [Commented] (CALCITE-4561) Wrong results for plan with EnumerableHashJoin (semi) on nullable colunms
[
https://issues.apache.org/jira/browse/CALCITE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326183#comment-17326183
]
Haisheng Yuan commented on CALCITE-4561:
Nice finding. it is better to generate not_null filter operator on both sides.
> Wrong results for plan with EnumerableHashJoin (semi) on nullable colunms
> -
>
> Key: CALCITE-4561
> URL: https://issues.apache.org/jira/browse/CALCITE-4561
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.27.0
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> When the query plan contains an {{EnumerableHashJoin}} with type {{SEMI}}
> over nullable columns the results are wrong since the operator incorrectly
> considers {{NULL=NULL}} as {{TRUE}} instead of {{UNKNOWN}}/{{FALSE}}.
> The problem can be reproduced by putting the following test in
> {{EnumerableHashJoinTest}}.
> {code:java}
> @Test void semiJoinWithNulls() {
> tester(false, new JdbcTest.HrSchema())
> .query(
> "SELECT e1.name FROM emps e1 WHERE e1.commission in (SELECT
> e2.commission FROM emps e2)")
> .explainContains("EnumerableCalc(expr#0..1=[{inputs}], name=[$t0])\n"
> + " EnumerableHashJoin(condition=[=($1, $6)],
> joinType=[semi])\n"
> + "EnumerableCalc(expr#0..4=[{inputs}], name=[$t2],
> commission=[$t4])\n"
> + " EnumerableTableScan(table=[[s, emps]])\n"
> + "EnumerableTableScan(table=[[s, emps]])\n\n")
> .returnsUnordered(
> "name=Bill",
> "name=Eric",
> "name=Theodore");
> }
> {code}
> At the moment 'Sebastian' (with commission NULL) is wrongly included in the
> result set.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4560) Wrong plan when decorrelating EXISTS subquery with COALESCE in the predicate
[
https://issues.apache.org/jira/browse/CALCITE-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326121#comment-17326121
]
Haisheng Yuan commented on CALCITE-4560:
[~Chunwei Lei] Can you take a look? you might be familiar with that part.
> Wrong plan when decorrelating EXISTS subquery with COALESCE in the predicate
>
>
> Key: CALCITE-4560
> URL: https://issues.apache.org/jira/browse/CALCITE-4560
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.27.0
>Reporter: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The problem can be seen by adding the following test in
> {{SqlToRelConverterTest}}.
> {code:java}
> @Test void testExistsCorrelatedDecorrelate01() {
> final String sql = "select e1.empno from empnullables e1 where exists (\n"
> + " select 1 from empnullables e2 where
> COALESCE(e1.ename,'M')=COALESCE(e2.ename,'M'))";
> sql(sql).decorrelate(true).ok();
> }
> {code}
> The plan after decorrelation is shown below:
> {noformat}
> LogicalProject(EMPNO=[$0])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8], ENAME0=[$9],
> $f1=[CAST($10):BOOLEAN])
> LogicalJoin(condition=[=($1, $9)], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMPNULLABLES]])
> LogicalAggregate(group=[{0}], agg#0=[MIN($1)])
> LogicalProject(ENAME0=[$9], $f0=[true])
> LogicalJoin(condition=[=(CASE(IS NOT NULL($9), $9,
> 'M':VARCHAR(20)), CASE(IS NOT NULL($1), CAST($1):VARCHAR(20) NOT NULL,
> 'M':VARCHAR(20)))], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, EMPNULLABLES]])
> LogicalAggregate(group=[{0}])
> LogicalProject(ENAME=[$1])
> LogicalTableScan(table=[[CATALOG, SALES, EMPNULLABLES]])
> {noformat}
> The problem lies in the {{LogicalJoin(condition=[=($1, $9)],
> joinType=[inner])}} operator. If there are rows with {{NULL}} values in the
> {{ENAME}} column these are going to be incorrectly removed from the result
> set. The COALESCE operator is present in the SQL query to ensure that rows
> with NULL values are retained in the result.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4557) Sort costing should account the number of sort keys: the more keys to compare, the more the cost
[ https://issues.apache.org/jira/browse/CALCITE-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326073#comment-17326073 ] Haisheng Yuan commented on CALCITE-4557: But if "hiredate" is not in the output columns, or not used by parent operators, dropping "hiredate" may benefit us. > Sort costing should account the number of sort keys: the more keys to > compare, the more the cost > > > Key: CALCITE-4557 > URL: https://issues.apache.org/jira/browse/CALCITE-4557 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.26.0 >Reporter: Vladimir Sitnikov >Priority: Major > > Current costing for sort does not account for the number of keys in the > comparison function which might result in worse plan selection. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4557) Sort costing should account the number of sort keys: the more keys to compare, the more the cost
[ https://issues.apache.org/jira/browse/CALCITE-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326069#comment-17326069 ] Haisheng Yuan commented on CALCITE-4557: I don't think it is necessary to simplify "ORDER BY deptno, empno, hiredate" to "Emp ORDER BY deptno, empno", because the comparison in SORT operator will stop at "empno" anyway, given that empno is unique. > Sort costing should account the number of sort keys: the more keys to > compare, the more the cost > > > Key: CALCITE-4557 > URL: https://issues.apache.org/jira/browse/CALCITE-4557 > Project: Calcite > Issue Type: Improvement > Components: core >Affects Versions: 1.26.0 >Reporter: Vladimir Sitnikov >Priority: Major > > Current costing for sort does not account for the number of keys in the > comparison function which might result in worse plan selection. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4557) Sort costing should account the number of sort keys: the more keys to compare, the more the cost
[
https://issues.apache.org/jira/browse/CALCITE-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325482#comment-17325482
]
Haisheng Yuan commented on CALCITE-4557:
{quote}
I doubt that a 5-key sort is significantly more expensive than a 1- or 2-key
sort. Most likely, they all terminate in 1 or 1.1 comparisons on average.
{quote}
It depends on the number of distinct values for each sort column, but generally
more keys are more expensive. I also remember in Postgres mailing list, there
is a discussion about the optimization to reorder the sorting columns based on
the number of distinct values. But it is hard to give accurate formula based on
the NDV, because there can also be some correlation between columns, just a
very coarse level estimation, sometimes not even close.
> Sort costing should account the number of sort keys: the more keys to
> compare, the more the cost
>
>
> Key: CALCITE-4557
> URL: https://issues.apache.org/jira/browse/CALCITE-4557
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Current costing for sort does not account for the number of keys in the
> comparison function which might result in worse plan selection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4577) RelDigestWriter#done has performance issue when node's inputs is large
[
https://issues.apache.org/jira/browse/CALCITE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325474#comment-17325474
]
Haisheng Yuan commented on CALCITE-4577:
See comments in CALCITE-4578, since 1.24.0, the issue should exist anymore.
> RelDigestWriter#done has performance issue when node's inputs is large
> --
>
> Key: CALCITE-4577
> URL: https://issues.apache.org/jira/browse/CALCITE-4577
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Critical
> Attachments: image-2021-04-11-16-53-35-466.png,
> image-2021-04-11-16-53-58-757.png
>
>
> Example like that: insert xxx VALUES (1, 'one'), (2, 'two'), (3,
> 'three')...(1, 'ten thousand');
> It generates a Union node with 1 inputs, and the performance is poor,
> here's the profile:
>
> !image-2021-04-11-16-53-58-757.png|width=735,height=109!
> A point that affects performance is the array copy of StringBuilder, current
> we just new StringBuilder with default capacity, my proposal is to consider
> the node's input size, like that(we can discuss the strategy):
> org.apache.calcite.rel.AbstractRelNode.RelDigestWriter#done
> {code:java}
> StringBuilder sb = new StringBuilder((values.size() + 1) * 16);{code}
> It's about 20% performance improvement and mem is more smooth.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4577) RelDigestWriter#done has performance issue when node's inputs is large
[
https://issues.apache.org/jira/browse/CALCITE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325474#comment-17325474
]
Haisheng Yuan edited comment on CALCITE-4577 at 4/20/21, 4:54 AM:
--
See comments in CALCITE-4578, since 1.24.0, the issue should not exist anymore.
was (Author: hyuan):
See comments in CALCITE-4578, since 1.24.0, the issue should exist anymore.
> RelDigestWriter#done has performance issue when node's inputs is large
> --
>
> Key: CALCITE-4577
> URL: https://issues.apache.org/jira/browse/CALCITE-4577
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Critical
> Attachments: image-2021-04-11-16-53-35-466.png,
> image-2021-04-11-16-53-58-757.png
>
>
> Example like that: insert xxx VALUES (1, 'one'), (2, 'two'), (3,
> 'three')...(1, 'ten thousand');
> It generates a Union node with 1 inputs, and the performance is poor,
> here's the profile:
>
> !image-2021-04-11-16-53-58-757.png|width=735,height=109!
> A point that affects performance is the array copy of StringBuilder, current
> we just new StringBuilder with default capacity, my proposal is to consider
> the node's input size, like that(we can discuss the strategy):
> org.apache.calcite.rel.AbstractRelNode.RelDigestWriter#done
> {code:java}
> StringBuilder sb = new StringBuilder((values.size() + 1) * 16);{code}
> It's about 20% performance improvement and mem is more smooth.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4578) redundant HepPlanner#buildFinalPlan when there's no rule fired in the program
[
https://issues.apache.org/jira/browse/CALCITE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325468#comment-17325468
]
Haisheng Yuan commented on CALCITE-4578:
{quote}
The best plan for your query should perhaps be a Values operator, not a Union?
Or possibly Unnest, if the expression is not constant.
{quote}
I agree, I don't see any reason to generate Union operator, instead of Values,
for constant literals.
> redundant HepPlanner#buildFinalPlan when there's no rule fired in the program
> -
>
> Key: CALCITE-4578
> URL: https://issues.apache.org/jira/browse/CALCITE-4578
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
> Attachments: image-2021-04-11-17-42-38-595.png
>
>
> Example like that: insert xxx VALUES (1, 'one'), (2, 'two'), (3,
> 'three')...(1, 'ten thousand');
> It generates a Union node with 1 inputs, and the performance is poor,
> here's the profile:
> !image-2021-04-11-17-42-38-595.png|width=1086,height=240!
> We create a program(HepProgram) to do the union merge, in this case, we only
> have one union, so the rule is not fired, but it will still build the
> buildFinalPlan, and AbstractRelNode#explain has performance issue in this
> case(see the flame graph):
> {code:java}
> public RelNode findBestExp() {
> assert root != null;
> executeProgram(mainProgram);
> // Get rid of everything except what's in the final plan.
> collectGarbage();
> return buildFinalPlan(root);
> }
> {code}
> And the union has large inputs, so the performance is poor(in
> SetOp#replaceInput, we have to recomputeDigest ), we can skip buildFinalPlan
> if there's no rule fired.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4578) redundant HepPlanner#buildFinalPlan when there's no rule fired in the program
[
https://issues.apache.org/jira/browse/CALCITE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325465#comment-17325465
]
Haisheng Yuan edited comment on CALCITE-4578 at 4/20/21, 4:31 AM:
--
Hi [~Aron.tao], which version are you using? According to the flame graph, it
is definitely NOT the latest version. Since v1.24.0 (see CALCITE-3786), the
digest recompute is almost no-op, just clear data. Can you try with 1.24.0 or
later?
was (Author: hyuan):
Hi [~Aron.tao], which version are you using? According to the flame graph, it
is definitely the latest version. Since v1.24.0 (see CALCITE-3786), the digest
recompute is almost no-op, just clear data. Can you try with 1.24.0 or later?
> redundant HepPlanner#buildFinalPlan when there's no rule fired in the program
> -
>
> Key: CALCITE-4578
> URL: https://issues.apache.org/jira/browse/CALCITE-4578
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
> Attachments: image-2021-04-11-17-42-38-595.png
>
>
> Example like that: insert xxx VALUES (1, 'one'), (2, 'two'), (3,
> 'three')...(1, 'ten thousand');
> It generates a Union node with 1 inputs, and the performance is poor,
> here's the profile:
> !image-2021-04-11-17-42-38-595.png|width=1086,height=240!
> We create a program(HepProgram) to do the union merge, in this case, we only
> have one union, so the rule is not fired, but it will still build the
> buildFinalPlan, and AbstractRelNode#explain has performance issue in this
> case(see the flame graph):
> {code:java}
> public RelNode findBestExp() {
> assert root != null;
> executeProgram(mainProgram);
> // Get rid of everything except what's in the final plan.
> collectGarbage();
> return buildFinalPlan(root);
> }
> {code}
> And the union has large inputs, so the performance is poor(in
> SetOp#replaceInput, we have to recomputeDigest ), we can skip buildFinalPlan
> if there's no rule fired.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4578) redundant HepPlanner#buildFinalPlan when there's no rule fired in the program
[
https://issues.apache.org/jira/browse/CALCITE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325465#comment-17325465
]
Haisheng Yuan commented on CALCITE-4578:
Hi [~Aron.tao], which version are you using? According to the flame graph, it
is definitely the latest version. Since v1.24.0 (see CALCITE-3786), the digest
recompute is almost no-op, just clear data. Can you try with 1.24.0 or later?
> redundant HepPlanner#buildFinalPlan when there's no rule fired in the program
> -
>
> Key: CALCITE-4578
> URL: https://issues.apache.org/jira/browse/CALCITE-4578
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Major
> Attachments: image-2021-04-11-17-42-38-595.png
>
>
> Example like that: insert xxx VALUES (1, 'one'), (2, 'two'), (3,
> 'three')...(1, 'ten thousand');
> It generates a Union node with 1 inputs, and the performance is poor,
> here's the profile:
> !image-2021-04-11-17-42-38-595.png|width=1086,height=240!
> We create a program(HepProgram) to do the union merge, in this case, we only
> have one union, so the rule is not fired, but it will still build the
> buildFinalPlan, and AbstractRelNode#explain has performance issue in this
> case(see the flame graph):
> {code:java}
> public RelNode findBestExp() {
> assert root != null;
> executeProgram(mainProgram);
> // Get rid of everything except what's in the final plan.
> collectGarbage();
> return buildFinalPlan(root);
> }
> {code}
> And the union has large inputs, so the performance is poor(in
> SetOp#replaceInput, we have to recomputeDigest ), we can skip buildFinalPlan
> if there's no rule fired.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4559) Create 'interface RexRule', a modular rewrite for row-expressions
[
https://issues.apache.org/jira/browse/CALCITE-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324726#comment-17324726
]
Haisheng Yuan commented on CALCITE-4559:
I agree with [~vozerov] that this is a great change, and the two improvements
would be nice to have given that the customized patterns may be very complex.
>>> Currently, the rules are applied bottom-up only. There is a chance that in
>>> the future we may need alternative strategies.
[~vozerov] It is common to apply bottom-up for rex expressions. I may be not
familiar with the other cases, can you shed light on the scenario where we may
want to apply the rex rule in other strategy, like top-down? Thanks.
> Create 'interface RexRule', a modular rewrite for row-expressions
> -
>
> Key: CALCITE-4559
> URL: https://issues.apache.org/jira/browse/CALCITE-4559
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
>
> We propose to add {{class RexRule}}, a rewrite rule for row-expressions
> ({{class RexNode}}).
> {{class RexRule}} is analogous to how {{class RelRule}} (and the older
> {{class RelOptRule}}) operates on relational expressions ({{interface
> RelNode}}). Also, {{class RexRuleProgram}} is analogous to {{HepProgram}} and
> {{VolcanoPlanner}} (it indexes rules so that we do not have to try every rule
> against every part of the expression). And a rule describes which operands it
> matches using {{RexRule.describe(RexRule.OperandBuilder)}}, similar to
> calling {{RelRule.Config.operandSupplier().apply()}}.
> The advantages of {{RexRule}} are similar to {{RelRule}}: rules can be
> defined in a modular way, can be documented and tested individually, and can
> be enabled individually.
> The rules could be applied in various ways. {{RelBuilder.Config}} could
> contain a {{RexRuleProgram}} that would be applied every time an expression
> is simplified by a {{RelBuilder}}. There could also be a sub-class of
> {{interface RelShuttle}} that applies the rules to every {{RexNode}} in a
> tree (e.g. inside {{Filter}}, {{Project}} and {{Join}}).
> I don't yet know whether, or how, rules might support 3-valued boolean logic
> ({{RexUnknownAs}}). For example, a rule that simplifies "x = x" to "TRUE" is
> valid in an "unknownAsFalse" context (e.g. as top-level of {{Filter}}
> condition), but not in an "unknownAsUnknown" context (e.g. in {{Project}}
> expression).
> This case is related to CALCITE-3470 (making relational and row-expression
> rules more similar, as in CockroachDB), but would deliver an API rather than
> a textual DSL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4559) Create 'interface RexRule', a modular rewrite for row-expressions
[
https://issues.apache.org/jira/browse/CALCITE-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324123#comment-17324123
]
Haisheng Yuan commented on CALCITE-4559:
Thanks Julian for the proposal, I will take a look.
> Create 'interface RexRule', a modular rewrite for row-expressions
> -
>
> Key: CALCITE-4559
> URL: https://issues.apache.org/jira/browse/CALCITE-4559
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
>
> We propose to add {{class RexRule}}, a rewrite rule for row-expressions
> ({{class RexNode}}).
> {{class RexRule}} is analogous to how {{class RelRule}} (and the older
> {{class RelOptRule}}) operates on relational expressions ({{interface
> RelNode}}). Also, {{class RexRuleProgram}} is analogous to {{HepProgram}} and
> {{VolcanoPlanner}} (it indexes rules so that we do not have to try every rule
> against every part of the expression). And a rule describes which operands it
> matches using {{RexRule.describe(RexRule.OperandBuilder)}}, similar to
> calling {{RelRule.Config.operandSupplier().apply()}}.
> The advantages of {{RexRule}} are similar to {{RelRule}}: rules can be
> defined in a modular way, can be documented and tested individually, and can
> be enabled individually.
> The rules could be applied in various ways. {{RelBuilder.Config}} could
> contain a {{RexRuleProgram}} that would be applied every time an expression
> is simplified by a {{RelBuilder}}. There could also be a sub-class of
> {{interface RelShuttle}} that applies the rules to every {{RexNode}} in a
> tree (e.g. inside {{Filter}}, {{Project}} and {{Join}}).
> I don't yet know whether, or how, rules might support 3-valued boolean logic
> ({{RexUnknownAs}}). For example, a rule that simplifies "x = x" to "TRUE" is
> valid in an "unknownAsFalse" context (e.g. as top-level of {{Filter}}
> condition), but not in an "unknownAsUnknown" context (e.g. in {{Project}}
> expression).
> This case is related to CALCITE-3470 (making relational and row-expression
> rules more similar, as in CockroachDB), but would deliver an API rather than
> a textual DSL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4432) When the RelNode's input is the same subset as the subset node belonged to, not choose this node as best.
[ https://issues.apache.org/jira/browse/CALCITE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246350#comment-17246350 ] Haisheng Yuan commented on CALCITE-4432: Hmm, since which version? > When the RelNode's input is the same subset as the subset node belonged to, > not choose this node as best. > - > > Key: CALCITE-4432 > URL: https://issues.apache.org/jira/browse/CALCITE-4432 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: Ziwei Liu >Assignee: Ziwei Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4427) Make SUBSTRING operator comply with ISO standard SQL
[
https://issues.apache.org/jira/browse/CALCITE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245499#comment-17245499
]
Haisheng Yuan commented on CALCITE-4427:
+1 to comply with ISO standard SQL.
> Make SUBSTRING operator comply with ISO standard SQL
>
>
> Key: CALCITE-4427
> URL: https://issues.apache.org/jira/browse/CALCITE-4427
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Priority: Major
>
> Make {{SUBSTRING}} operator comply with ISO standard SQL. It currently
> complies with BigQuery standard SQL, which gives different behavior when
> start is negative. See discussion in CALCITE-4408.
> Add an option to get the BigQuery behavior, for those who want it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4386) Support RelShuttle visit specific logical operators
[ https://issues.apache.org/jira/browse/CALCITE-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229718#comment-17229718 ] Haisheng Yuan commented on CALCITE-4386: Sorry, I still don't get it. Can you add a test case that will fail without the patch? > Support RelShuttle visit specific logical operators > --- > > Key: CALCITE-4386 > URL: https://issues.apache.org/jira/browse/CALCITE-4386 > Project: Calcite > Issue Type: Wish >Reporter: xzh_dz >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the current implementation, some operators, such as `LogicalProject` and > `LogicalFilter`, can be directly accessed by `Relshuttle`, while others, such > as `LogicalCalc` and `LogicalWindow`, cannot be accessed directly. > `RelShuttle` is a very important tool class. Usually, we will do equivalent > transformation of relational algebra in the process of relational algebra > optimization. This PR will cover the method of the operator visit > `RelShuttle`, and the specific operator can be accessed directly by visiting > `RelNode`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4386) Support RelShuttle visit specific logical operators
[ https://issues.apache.org/jira/browse/CALCITE-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229641#comment-17229641 ] Haisheng Yuan commented on CALCITE-4386: AbstractRelNode has default implementation. Do we really need the PR? > Support RelShuttle visit specific logical operators > --- > > Key: CALCITE-4386 > URL: https://issues.apache.org/jira/browse/CALCITE-4386 > Project: Calcite > Issue Type: Wish >Reporter: xzh_dz >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the current implementation, some operators, such as `LogicalProject` and > `LogicalFilter`, can be directly accessed by `Relshuttle`, while others, such > as `LogicalCalc` and `LogicalWindow`, cannot be accessed directly. > `RelShuttle` is a very important tool class. Usually, we will do equivalent > transformation of relational algebra in the process of relational algebra > optimization. This PR will cover the method of the operator visit > `RelShuttle`, and the specific operator can be accessed directly by visiting > `RelNode`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3679) Allow lambda expressions in SQL queries
[ https://issues.apache.org/jira/browse/CALCITE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211827#comment-17211827 ] Haisheng Yuan commented on CALCITE-3679: [~ritesh.kapoor] We appreciate your contribution and are looking forward to your update. Do you have any plan to continue working on this issue? > Allow lambda expressions in SQL queries > --- > > Key: CALCITE-3679 > URL: https://issues.apache.org/jira/browse/CALCITE-3679 > Project: Calcite > Issue Type: New Feature >Reporter: Ritesh >Assignee: Ritesh >Priority: Major > Labels: pull-request-available > Attachments: [CALCITE-3679]_Basic_implementation.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > [https://teradata.github.io/presto/docs/0.167-t/functions/lambda.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4027) Add -Doverwrite option to SqlToRelTestBase
[ https://issues.apache.org/jira/browse/CALCITE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-4027: --- Labels: (was: pull-request-available) > Add -Doverwrite option to SqlToRelTestBase > -- > > Key: CALCITE-4027 > URL: https://issues.apache.org/jira/browse/CALCITE-4027 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Attachments: image-2020-06-14-09-28-05-925.png > > Time Spent: 2h 40m > Remaining Estimate: 0h > > So that by setting overwrite=true, the expected xml file will be overwritten > by actual output file automatically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4027) Add -Doverwrite option to SqlToRelTestBase
[ https://issues.apache.org/jira/browse/CALCITE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211770#comment-17211770 ] Haisheng Yuan commented on CALCITE-4027: [~amaliujia] Can you continue on this JIRA? It will be helpful for Calcite developers. > Add -Doverwrite option to SqlToRelTestBase > -- > > Key: CALCITE-4027 > URL: https://issues.apache.org/jira/browse/CALCITE-4027 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Attachments: image-2020-06-14-09-28-05-925.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > So that by setting overwrite=true, the expected xml file will be overwritten > by actual output file automatically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4034) Implement a MySQL InnoDB adapter
[
https://issues.apache.org/jira/browse/CALCITE-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211433#comment-17211433
]
Haisheng Yuan commented on CALCITE-4034:
Hi [~neoremind], 1.26.0 is already released. We will try to get this into
1.27.0.
> Implement a MySQL InnoDB adapter
>
>
> Key: CALCITE-4034
> URL: https://issues.apache.org/jira/browse/CALCITE-4034
> Project: Calcite
> Issue Type: New Feature
> Components: csv-adapter
>Affects Versions: 1.23.0
>Reporter: neoremind
>Assignee: Julian Hyde
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Calcite’s InnoDB adapter allows you to query the data based on InnoDB data
> files directy, data files are also known as .ibd files, this adapter is
> different from JDBC adapter which requires a MySQL server to serve response.
> With .ibd files and the corresponding DDLs, InnoDB adapter is able to work
> like a simple "MySQL server", it accepts SQL query and attempts to compile
> the query based on InnoDB file accessing APIs provided by
> [innodb-java-reader|[https://github.com/alibaba/innodb-java-reader]] (I am
> the author of the library, which is well-tested and production-ready), it
> will exploit projecting, filtering and sorting directly in InnoDB data file
> where possible. What’s more, with DDLs, the adapter is "index aware", it
> leverages rules to choose the right index to scan, for example, using primary
> key or secondary keys to look up data, then it tries to push down some
> conditions into storage engine. Also, the adapter leaves option to provide
> hint as well, so that user can indicate the optimizer to force use one
> specific index.
> The InnoDB adapter can,
> 1) Full scan a table.
> 2) Scan by index such as primary key or secondary key, single column or
> composite keys.
> 3) Push down filtering condition to storage engine, support both point query
> and range query.
> 4) Push down projection.
> 5) Push down ordering if it matches the natural collation of an index.
> 6) Support almost all the commonly used data types.
> 7) Work as a simple MySQL server being able to read .ibd data files directly
> through file system, this can be a tool to query or dump table data by
> offloading from MySQL process under some conditions.
> Before I describe the adapter and its design, I was wondering if it is
> appropriate to deliver the work by contributing the codebase back to Calcite,
> or if it would be better to stay in another project, probably being referred
> from Calcite adapter link. Looking forward to Calcite developer's advice.
>
> Below is the first version of the usage.
> --
> [MySQL]([https://www.mysql.com/]) is the most popular Open Source SQL
> database management system, is developed, distributed, and supported by
> Oracle Corporation. InnoDB is a general-purpose storage engine that balances
> high reliability and high performance in MySQL, since 5.6 InnoDB has become
> the default MySQL storage engine.
> Calcite’s InnoDB adapter allows you to query the data based on InnoDB data
> files directy, data files are also known as .ibd files. It leverages
> [innodb-java-reader]([https://github.com/alibaba/innodb-java-reader]). This
> adapter is different from JDBC adapter which maps a schema in a JDBC data
> source and requires a MySQL server to serve response. With .ibd files and the
> corresponding DDLs, InnoDB adapter is able to work like a simple "MySQL
> server", it accepts SQL query and attempts to compile the query based on
> InnoDB file accessing APIs provided by innodb-java-reader, it will exploit
> projecting, filtering and sorting directly in InnoDB data file where possible.
> What’s more, with DDLs, the adapter is "index aware", it leverages rules to
> choose the right index to scan, for example, using primary key or secondary
> keys to look up data, then it tries to push down some conditions into storage
> engine. Also, the adapter leaves option to provide hint as well, so that user
> can indicate the optimizer to force use one specific index.
> A basic example of a model file is given below, this schema reads from a
> MySQL "scott" database:
> {code:java}
> {
> "version": "1.0",
> "defaultSchema": "scott",
> "schemas": [
> {
> "name": "scott",
> "type": "custom",
> "factory": "org.apache.calcite.adapter.innodb.InnodbSchemaFactory",
> "operand": {
> "sqlFilePath": [ "/path/scott.sql" ],
> "ibdDataFileBasePath": "/usr/local/mysql/data/scott"
> }
> }
> ]
> }
> {code}
> `sqlFilePath` is a list of DDL files, you can generate table definitions by
> executing `_mysqldump -d -u -p -h _`
> in command-line.
> The file content of `/path/scott.sql`
[jira] [Commented] (CALCITE-4290) Not optimal subqueries due to a "*" in them
[ https://issues.apache.org/jira/browse/CALCITE-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211430#comment-17211430 ] Haisheng Yuan commented on CALCITE-4290: The cost model doesn't take column width into account, so it is possible that the less optimal plan has the same cost with the expected column pruning plan, but this is just a speculation. It might be better to attach a test case, so whoever interested in this issue can work on it. > Not optimal subqueries due to a "*" in them > --- > > Key: CALCITE-4290 > URL: https://issues.apache.org/jira/browse/CALCITE-4290 > Project: Calcite > Issue Type: Bug > Components: jdbc-adapter >Affects Versions: 1.25.0 >Reporter: Roman Churganov >Priority: Major > > Run a query which should be distributed into sub-queries onto multiple > schemas through JDBC adapter, for example > select ft.id, ft.c11, tt.c41 from sch1.foo ft join sch2.tab tt on > ft.id = tt.id > Calcite will make two sub-queries like SELECT * FROM "TAB" ORDER BY "ID" and > SELECT * FROM "FOO" ORDER BY "ID" which are not optimal due to an excessive > columns data requested > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-3841) Change downloads page to use downloads.apache.org
[
https://issues.apache.org/jira/browse/CALCITE-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan reassigned CALCITE-3841:
--
Assignee: (was: Haisheng Yuan)
> Change downloads page to use downloads.apache.org
> -
>
> Key: CALCITE-3841
> URL: https://issues.apache.org/jira/browse/CALCITE-3841
> Project: Calcite
> Issue Type: Bug
> Components: site
>Reporter: Julian Hyde
>Priority: Major
> Fix For: 1.26.0
>
>
> Infra has
> [decided|https://lists.apache.org/thread.html/rcd2864e75e417597d342b8eb83080eb2d7a0cafea84fd4521a4d9cfd%40%3Cusers.infra.apache.org%3E]
> (login required for that email link) to deprecate
> [www.apache.org/dist|https://www.apache.org/dist] and move downloads to
> [https://downloads.apache.org|https://downloads.apache.org].
> On [Calcite's downloads page|https://calcite.apache.org/downloads/], we need
> to change the 'digest' link from (for example)
> {{https://www.apache.org/dist/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}}
> to
> {{https://downloads.apache.org/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}},
> and similarly the 'gpg' link.
> I believe that the 'tar' link can remain as
> {{https://www.apache.org/dyn/closer.lua?filename=calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz=download}}
> for the latest release and
> {{https://archive.apache.org/dist/calcite/apache-calcite-1.20.0/apache-calcite-1.20.0-src.tar.gz}}
> for older releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3841) Change downloads page to use downloads.apache.org
[
https://issues.apache.org/jira/browse/CALCITE-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203014#comment-17203014
]
Haisheng Yuan commented on CALCITE-3841:
Hi [~rubenql], thanks for asking. I have been busy working on internal stuff
recently, don't have much bandwidth starting. I will leave it to other
contributors, sorry about that.
> Change downloads page to use downloads.apache.org
> -
>
> Key: CALCITE-3841
> URL: https://issues.apache.org/jira/browse/CALCITE-3841
> Project: Calcite
> Issue Type: Bug
> Components: site
>Reporter: Julian Hyde
>Assignee: Haisheng Yuan
>Priority: Major
> Fix For: 1.26.0
>
>
> Infra has
> [decided|https://lists.apache.org/thread.html/rcd2864e75e417597d342b8eb83080eb2d7a0cafea84fd4521a4d9cfd%40%3Cusers.infra.apache.org%3E]
> (login required for that email link) to deprecate
> [www.apache.org/dist|https://www.apache.org/dist] and move downloads to
> [https://downloads.apache.org|https://downloads.apache.org].
> On [Calcite's downloads page|https://calcite.apache.org/downloads/], we need
> to change the 'digest' link from (for example)
> {{https://www.apache.org/dist/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}}
> to
> {{https://downloads.apache.org/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}},
> and similarly the 'gpg' link.
> I believe that the 'tar' link can remain as
> {{https://www.apache.org/dyn/closer.lua?filename=calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz=download}}
> for the latest release and
> {{https://archive.apache.org/dist/calcite/apache-calcite-1.20.0/apache-calcite-1.20.0-src.tar.gz}}
> for older releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4027) Add -Doverwrite option to SqlToRelTestBase
[ https://issues.apache.org/jira/browse/CALCITE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182820#comment-17182820 ] Haisheng Yuan commented on CALCITE-4027: [~xzh_dz] Are you still working on this issue and trying to address the comments? > Add -Doverwrite option to SqlToRelTestBase > -- > > Key: CALCITE-4027 > URL: https://issues.apache.org/jira/browse/CALCITE-4027 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Attachments: image-2020-06-14-09-28-05-925.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > So that by setting overwrite=true, the expected xml file will be overwritten > by actual output file automatically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3782) Bitwise operator Bit_And, Bit_OR and Bit_XOR support binary and varbinary type
[ https://issues.apache.org/jira/browse/CALCITE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3782. Fix Version/s: 1.26.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/9cf829bc28113ba01f3ee0c826b502143478680e, thanks for the PR, [~hailong wang]! > Bitwise operator Bit_And, Bit_OR and Bit_XOR support binary and varbinary type > -- > > Key: CALCITE-3782 > URL: https://issues.apache.org/jira/browse/CALCITE-3782 > Project: Calcite > Issue Type: Sub-task > Components: core >Affects Versions: 1.22.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Major > Labels: pull-request-available > Fix For: 1.26.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > According to the discussion link CALCITE-3732 , We should make bitwise > operators work on all integer types, BINARY and VARBINARY. So Bit_And, Bit_OR > and Bit_XOR agg operator should also support BINARY and VARBINARY. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3916) Support cascades style top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176726#comment-17176726 ] Haisheng Yuan commented on CALCITE-3916: Thanks for your detailed explanation., [~rubenql]! We need to investigate the root cause of the regression for your pattern, I can imagine it might hard to create a repro test case, but if you can, that would be great. Usually the operator(IndexScan45) generated from "passThrough" should better have the same convention with original operator (IndexScan38), ignore the required "ENUMERABLE" convention, have you tried that? > Support cascades style top-down driven rule apply > - > > Key: CALCITE-3916 > URL: https://issues.apache.org/jira/browse/CALCITE-3916 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Jinpeng Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 15.5h > Remaining Estimate: 0h > > Apply rules by leaf RelSet -> root RelSet order. For every RelNode in a > RelSet, rule is matched and applied sequentially. No RuleQueue and > DeferringRuleCall is needed anymore. This will make space pruning and rule > mutual exclusivity check possible. > Rule that use AbstractConverter as operand is an exception, to keep backward > compatibility, this kind of rule still needs top-down apply. > This should be done after CALCITE-3896. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-3916) Support cascades style top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175077#comment-17175077 ] Haisheng Yuan commented on CALCITE-3916: thanks for reporting, [~rubenql]. how is the scan operator with the specific rel id generated? > Support cascades style top-down driven rule apply > - > > Key: CALCITE-3916 > URL: https://issues.apache.org/jira/browse/CALCITE-3916 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Jinpeng Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 15.5h > Remaining Estimate: 0h > > Apply rules by leaf RelSet -> root RelSet order. For every RelNode in a > RelSet, rule is matched and applied sequentially. No RuleQueue and > DeferringRuleCall is needed anymore. This will make space pruning and rule > mutual exclusivity check possible. > Rule that use AbstractConverter as operand is an exception, to keep backward > compatibility, this kind of rule still needs top-down apply. > This should be done after CALCITE-3896. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4102) Some improvements to aggregate related operations
[ https://issues.apache.org/jira/browse/CALCITE-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-4102. Fix Version/s: 1.25.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/ce2ae649c9ec072ceb8462b2783f8c6bba989241, thanks for the PR, [~fan_li_ya]! > Some improvements to aggregate related operations > - > > Key: CALCITE-4102 > URL: https://issues.apache.org/jira/browse/CALCITE-4102 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Liya Fan >Assignee: Liya Fan >Priority: Minor > Labels: pull-request-available > Fix For: 1.25.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > 1. Improve the use of data structures > 2. Extract common methods > 3. Avoid expensive operations -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4111) Remove VolcanoPlannerPhase in VolcanoPlanner
[ https://issues.apache.org/jira/browse/CALCITE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-4111. Fix Version/s: 1.25.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/462528077d64d47cef781c2d54ab4484dad76dde, thanks for your PR, [~Aron.tao]! > Remove VolcanoPlannerPhase in VolcanoPlanner > > > Key: CALCITE-4111 > URL: https://issues.apache.org/jira/browse/CALCITE-4111 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Minor > Labels: pull-request-available > Fix For: 1.25.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4129) Support equality check for whole rel plan tree
[
https://issues.apache.org/jira/browse/CALCITE-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan resolved CALCITE-4129.
Fix Version/s: 1.25.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/b7e245495f5824d5f15143b1b519142876542bf3.
> Support equality check for whole rel plan tree
> --
>
> Key: CALCITE-4129
> URL: https://issues.apache.org/jira/browse/CALCITE-4129
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.25.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently the only way to check rel node tree deep equality is transforming
> into String by {{RelOptUtil.toString(rel)}} with
> {{SqlExplainLevel.EXPPLAN_ATTRIBUTES}}, which is inefficient. One example is
> RexSubQuery. It has to do it this way, because the rel being reference by
> RexSubQuery is possibly not yet registered to VolcanoPlanner, and the digest
> {{equals}} checks the input RelNode by identity (not content). That is OK for
> RelSubset and HepRelVertex, if the RelNode is already registered in planner,
> but not for plain RelNode that is outside of planner. Due to this, we have to
> implement another set of deep equals logic in our system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4032) Mark CalcMergeRule as TransformationRule
[ https://issues.apache.org/jira/browse/CALCITE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-4032: --- Description: Previously it was removed in CALCITE-3997. By marking CalcMergeRule as TransformationRule, the CalcMergeRule won't match PhysicalNode (including EnumerableCalc) in VolcanoPlanner. was: Previously it was removed in CALCITE-3997. By marking > Mark CalcMergeRule as TransformationRule > > > Key: CALCITE-4032 > URL: https://issues.apache.org/jira/browse/CALCITE-4032 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.24.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Previously it was removed in CALCITE-3997. > By marking CalcMergeRule as TransformationRule, the CalcMergeRule won't match > PhysicalNode (including EnumerableCalc) in VolcanoPlanner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4032) Mark CalcMergeRule as TransformationRule
[ https://issues.apache.org/jira/browse/CALCITE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan updated CALCITE-4032: --- Description: Previously it was removed in CALCITE-3997. By marking was:Previously it was removed in CALCITE-3997. > Mark CalcMergeRule as TransformationRule > > > Key: CALCITE-4032 > URL: https://issues.apache.org/jira/browse/CALCITE-4032 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.24.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Previously it was removed in CALCITE-3997. > By marking -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4032) Mark CalcMergeRule as TransformationRule
[ https://issues.apache.org/jira/browse/CALCITE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164144#comment-17164144 ] Haisheng Yuan commented on CALCITE-4032: With this change, the CalcMergeRule won't match PhysicalNode (including EnumerableCalc) in VolcanoPlanner. Discussion: https://lists.apache.org/thread.html/r615200d1af5ea7aa5e44aafa7220823d09053fd6a1fa411c60b51386%40%3Cdev.calcite.apache.org%3E > Mark CalcMergeRule as TransformationRule > > > Key: CALCITE-4032 > URL: https://issues.apache.org/jira/browse/CALCITE-4032 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Fix For: 1.24.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Previously it was removed in CALCITE-3997. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4115) Improve the prompt of using SQL keywords for sql parser
[
https://issues.apache.org/jira/browse/CALCITE-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162963#comment-17162963
]
Haisheng Yuan commented on CALCITE-4115:
[~danny0405] You might want to refrain from committing to master branch for the
moment, because the release isn't finished yet.
> Improve the prompt of using SQL keywords for sql parser
> ---
>
> Key: CALCITE-4115
> URL: https://issues.apache.org/jira/browse/CALCITE-4115
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.23.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Current when a user uses a reserved-keyword in the SQL, for example:
> {code:sql}
> select DEFAULT from emp;
> {code}
> the parser would report with error message:
> {noformat}
> Encountered "DEFAULT" at line 1, column 8.
> Was expecting one of:
> "ABS" ...
> "ALL" ...
> "ARRAY" ...
> "AVG" ...
> "CARDINALITY" ...
> "CASE" ...
> "CAST" ...
> "CEIL" ...
> "CEILING" ...
> "CHAR_LENGTH" ...
> "CHARACTER_LENGTH" ...
> {noformat}
> It is hard to infer that the DEFAULT is actually a reserved-keyword, we can
> promote the message to make it more clear.
> For example, in SQL-SERVER, if you use such a keyword, the server reports
> "Incorrect syntax near the keyword 'DEFAULT'", which is very clear that the
> DEFAULT is a keyword.
> Based on the SQL-SERVER message, i would suggest to also report the position
> with a suggested fix solution, the template is like this:
> {code:java}
> Incorrect syntax near the keyword {keyword} at line {line_number},
> column {column_number}.\n
> Either uses a non-reserved keyword or quotes the keyword with
> character: {quoting_character}.
> {code}
> We would tell user that the mistake is a keyword and its position, also how
> to fix it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4126) Stackoverflow error when applying JoinCommuteRule
[
https://issues.apache.org/jira/browse/CALCITE-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162940#comment-17162940
]
Haisheng Yuan commented on CALCITE-4126:
[~fan_li_ya] Feel feel to pick it up. Thanks!
> Stackoverflow error when applying JoinCommuteRule
> -
>
> Key: CALCITE-4126
> URL: https://issues.apache.org/jira/browse/CALCITE-4126
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Add the test case to RelOptRulesTest.java
> {code:java}
> @Test void testJoinCommute() {
> final HepProgram program = HepProgram.builder()
> .addRuleInstance(JoinCommuteRule.INSTANCE)
> .build();
> final String sql = "select * \n"
> + "from sales.emp e join sales.dept d on e.empno > d.deptno";
> sql(sql).with(program).check();
> }
> {code}
> Stacktrace:
> {code:java}
> java.lang.StackOverflowError
> at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:210)
> at org.apache.calcite.rel.core.Project.(Project.java:90)
> at
> org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:67)
> at
> org.apache.calcite.rel.logical.LogicalProject.copy(LogicalProject.java:128)
> at
> org.apache.calcite.rel.logical.LogicalProject.copy(LogicalProject.java:45)
> at org.apache.calcite.rel.core.Project.copy(Project.java:122)
> at
> org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:803)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyTransformationResults(HepPlanner.java:732)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:543)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:360)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> at
> org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
> {code}
> I check master branch and 1.23.0, both throw StackOverflowError.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4131) ERROR message in SqlToRelConverterExtendedTest
Haisheng Yuan created CALCITE-4131:
--
Summary: ERROR message in SqlToRelConverterExtendedTest
Key: CALCITE-4131
URL: https://issues.apache.org/jira/browse/CALCITE-4131
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Haisheng Yuan
The error doesn't affect the proceeding of the test suite, and doesn't fail the
test. But the error message is not so pleasing:
{code:java}
SqlToRelConverterExtendedTest > testCrossJoinLateral1() STANDARD_ERROR
[Fatal Error] :1:2: XML document structures must start and end within the
same entity.
ERROR: 'Could not compile stylesheet'
FATAL ERROR: 'XML document structures must start and end within the same
entity.'
:XML document structures must start and end within the same
entity.
SqlToRelConverterExtendedTest > testMatchRecognizeIn() STANDARD_ERROR
ERROR: 'XML document structures must start and end within the same entity.'
ERROR: 'com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: XML
document structures must start and end within the same entity.'
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4129) Support equality check for whole rel plan tree
[
https://issues.apache.org/jira/browse/CALCITE-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17160443#comment-17160443
]
Haisheng Yuan commented on CALCITE-4129:
Thanks for your quick response, [~julianhyde].
I don't get it. Can you explain a little bit of {{the approach should be able
to compare RelNode trees without actually building them}}? I thought when we
create the RelNode, we must pass the input relnodes to the RelNode.
If it is in planner, it is OK, we just compare the subset by its identity, no
need to go deeper.
In some cases, we compare the node tree after the best plan is generated.
> Support equality check for whole rel plan tree
> --
>
> Key: CALCITE-4129
> URL: https://issues.apache.org/jira/browse/CALCITE-4129
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Priority: Major
>
> Currently the only way to check rel node tree deep equality is transforming
> into String by {{RelOptUtil.toString(rel)}} with
> {{SqlExplainLevel.EXPPLAN_ATTRIBUTES}}, which is inefficient. One example is
> RexSubQuery. It has to do it this way, because the rel being reference by
> RexSubQuery is possibly not yet registered to VolcanoPlanner, and the digest
> {{equals}} checks the input RelNode by identity (not content). That is OK for
> RelSubset and HepRelVertex, if the RelNode is already registered in planner,
> but not for plain RelNode that is outside of planner. Due to this, we have to
> implement another set of deep equals logic in our system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4129) Support equality check for whole rel plan tree
Haisheng Yuan created CALCITE-4129:
--
Summary: Support equality check for whole rel plan tree
Key: CALCITE-4129
URL: https://issues.apache.org/jira/browse/CALCITE-4129
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Haisheng Yuan
Currently the only way to check rel node tree deep equality is transforming
into String by {{RelOptUtil.toString(rel)}} with
{{SqlExplainLevel.EXPPLAN_ATTRIBUTES}}, which is inefficient. One example is
RexSubQuery. It has to do it this way, because the rel being reference by
RexSubQuery is possibly not yet registered to VolcanoPlanner, and the digest
{{equals}} checks the input RelNode by identity (not content). That is OK for
RelSubset and HepRelVertex, if the RelNode is already registered in planner,
but not for plain RelNode that is outside of planner. Due to this, we have to
implement another set of deep equals logic in our system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4127) Remove final from AbstractRelNode#getRelTypeName
[ https://issues.apache.org/jira/browse/CALCITE-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-4127. Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/cdf8678f33f8ea49e5291fce4a4651a24edaf19c. > Remove final from AbstractRelNode#getRelTypeName > > > Key: CALCITE-4127 > URL: https://issues.apache.org/jira/browse/CALCITE-4127 > Project: Calcite > Issue Type: Task >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > See email discussion > [here|https://mail-archives.apache.org/mod_mbox/calcite-dev/202007.mbox/%3ccaez+v3txpk8hevbuk6myzvee61xwrogkostzyzubxk7yan_...@mail.gmail.com%3e] > Remove "final" from AbstractRelNode#getRelTypeName, so that downstream > projects have the possibility to provide their own implementation, at their > own risk. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4126) Stackoverflow error when applying JoinCommuteRule
Haisheng Yuan created CALCITE-4126:
--
Summary: Stackoverflow error when applying JoinCommuteRule
Key: CALCITE-4126
URL: https://issues.apache.org/jira/browse/CALCITE-4126
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Haisheng Yuan
Add the test case to RelOptRulesTest.java
{code:java}
@Test void testJoinCommute() {
final HepProgram program = HepProgram.builder()
.addRuleInstance(JoinCommuteRule.INSTANCE)
.build();
final String sql = "select * \n"
+ "from sales.emp e join sales.dept d on e.empno > d.deptno";
sql(sql).with(program).check();
}
{code}
Stacktrace:
{code:java}
java.lang.StackOverflowError
at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
at org.apache.calcite.rel.core.Project.isValid(Project.java:210)
at org.apache.calcite.rel.core.Project.(Project.java:90)
at
org.apache.calcite.rel.logical.LogicalProject.(LogicalProject.java:67)
at
org.apache.calcite.rel.logical.LogicalProject.copy(LogicalProject.java:128)
at
org.apache.calcite.rel.logical.LogicalProject.copy(LogicalProject.java:45)
at org.apache.calcite.rel.core.Project.copy(Project.java:122)
at
org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:803)
at
org.apache.calcite.plan.hep.HepPlanner.applyTransformationResults(HepPlanner.java:732)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:543)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:360)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
at
org.apache.calcite.plan.hep.HepPlanner.depthFirstApply(HepPlanner.java:372)
{code}
I check master branch and 1.23.0, both throw StackOverflowError.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4124) Stop invalidating metadata cache in VolcanoRuleCall
[ https://issues.apache.org/jira/browse/CALCITE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-4124. Fix Version/s: 1.24.0 Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/88d4de0f800b84ae10a7a5be8a6c96eeb30eedf7. > Stop invalidating metadata cache in VolcanoRuleCall > --- > > Key: CALCITE-4124 > URL: https://issues.apache.org/jira/browse/CALCITE-4124 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 40m > Remaining Estimate: 0h > > After CALCITE-2018, we don't need to invalidate metadata query when a new > RelNode is generated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4124) Stop invalidating metadata cache in VolcanoRuleCall
Haisheng Yuan created CALCITE-4124: -- Summary: Stop invalidating metadata cache in VolcanoRuleCall Key: CALCITE-4124 URL: https://issues.apache.org/jira/browse/CALCITE-4124 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan After CALCITE-2018, we don't need to invalidate metadata query when a new RelNode is generated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4015) Pass through parent collation request on subset or superset of join keys for EnumerableMergeJoin
[
https://issues.apache.org/jira/browse/CALCITE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157720#comment-17157720
]
Haisheng Yuan commented on CALCITE-4015:
Sorry for my late review. Let's do this in 1.25.
> Pass through parent collation request on subset or superset of join keys for
> EnumerableMergeJoin
>
>
> Key: CALCITE-4015
> URL: https://issues.apache.org/jira/browse/CALCITE-4015
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Rui Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Pass through parent collation request on subset or superset of join keys for
> EnumerableMergeJoin.
> e.g.
> {code:java}
> select * from foo join bar on foo.a=bar.a and foo.b=bar.b order by foo.b;
> {code}
> collation on b is passed to mergejoin, mergejoin should generate an
> alternative with collation on b, a.
> {code:java}
> select * from foo join bar on foo.a=bar.a and foo.b=bar.b order by foo.a,
> foo.b, foo.c;
> {code}
> collation on a,b,c is passed to mergejoin, mergejoin should generate an
> alternative with collation on a,b,c, even the join keys are a,b, but since
> foo.c is also the output column, mergejoin can still request its child foo to
> deliver collation on a,b,c. The join keys must be the collation request's
> prefixes (keys order is not required to be reserved).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4015) Pass through parent collation request on subset or superset of join keys for EnumerableMergeJoin
[
https://issues.apache.org/jira/browse/CALCITE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-4015:
---
Fix Version/s: (was: 1.24.0)
> Pass through parent collation request on subset or superset of join keys for
> EnumerableMergeJoin
>
>
> Key: CALCITE-4015
> URL: https://issues.apache.org/jira/browse/CALCITE-4015
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Haisheng Yuan
>Assignee: Rui Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Pass through parent collation request on subset or superset of join keys for
> EnumerableMergeJoin.
> e.g.
> {code:java}
> select * from foo join bar on foo.a=bar.a and foo.b=bar.b order by foo.b;
> {code}
> collation on b is passed to mergejoin, mergejoin should generate an
> alternative with collation on b, a.
> {code:java}
> select * from foo join bar on foo.a=bar.a and foo.b=bar.b order by foo.a,
> foo.b, foo.c;
> {code}
> collation on a,b,c is passed to mergejoin, mergejoin should generate an
> alternative with collation on a,b,c, even the join keys are a,b, but since
> foo.c is also the output column, mergejoin can still request its child foo to
> deliver collation on a,b,c. The join keys must be the collation request's
> prefixes (keys order is not required to be reserved).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4122) Support on-demand rule matching
Haisheng Yuan created CALCITE-4122: -- Summary: Support on-demand rule matching Key: CALCITE-4122 URL: https://issues.apache.org/jira/browse/CALCITE-4122 Project: Calcite Issue Type: Improvement Components: core Reporter: Haisheng Yuan Currently in top-down planner, rule is matched whenever a new relnode is added to memo, no matter the group is pruned or not. We should do the rule matching for each operator only when needed, hence rule queue is not required anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3916) Support cascades style top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haisheng Yuan resolved CALCITE-3916. Resolution: Fixed Fixed in https://github.com/apache/calcite/commit/33aa61ca404018cc9fe8ad2ec2c02ba269c67ebe, thanks for the PR, [~FatLittle]! > Support cascades style top-down driven rule apply > - > > Key: CALCITE-3916 > URL: https://issues.apache.org/jira/browse/CALCITE-3916 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Jinpeng Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 14h > Remaining Estimate: 0h > > Apply rules by leaf RelSet -> root RelSet order. For every RelNode in a > RelSet, rule is matched and applied sequentially. No RuleQueue and > DeferringRuleCall is needed anymore. This will make space pruning and rule > mutual exclusivity check possible. > Rule that use AbstractConverter as operand is an exception, to keep backward > compatibility, this kind of rule still needs top-down apply. > This should be done after CALCITE-3896. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-3841) Change downloads page to use downloads.apache.org
[
https://issues.apache.org/jira/browse/CALCITE-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haisheng Yuan updated CALCITE-3841:
---
Fix Version/s: (was: 1.24.0)
> Change downloads page to use downloads.apache.org
> -
>
> Key: CALCITE-3841
> URL: https://issues.apache.org/jira/browse/CALCITE-3841
> Project: Calcite
> Issue Type: Bug
> Components: site
>Reporter: Julian Hyde
>Assignee: Haisheng Yuan
>Priority: Major
>
> Infra has
> [decided|https://lists.apache.org/thread.html/rcd2864e75e417597d342b8eb83080eb2d7a0cafea84fd4521a4d9cfd%40%3Cusers.infra.apache.org%3E]
> (login required for that email link) to deprecate
> [www.apache.org/dist|https://www.apache.org/dist] and move downloads to
> [https://downloads.apache.org|https://downloads.apache.org].
> On [Calcite's downloads page|https://calcite.apache.org/downloads/], we need
> to change the 'digest' link from (for example)
> {{https://www.apache.org/dist/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}}
> to
> {{https://downloads.apache.org/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}},
> and similarly the 'gpg' link.
> I believe that the 'tar' link can remain as
> {{https://www.apache.org/dyn/closer.lua?filename=calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz=download}}
> for the latest release and
> {{https://archive.apache.org/dist/calcite/apache-calcite-1.20.0/apache-calcite-1.20.0-src.tar.gz}}
> for older releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3841) Change downloads page to use downloads.apache.org
[
https://issues.apache.org/jira/browse/CALCITE-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157114#comment-17157114
]
Haisheng Yuan commented on CALCITE-3841:
Not yet. I will do it after 1.24 is released.
> Change downloads page to use downloads.apache.org
> -
>
> Key: CALCITE-3841
> URL: https://issues.apache.org/jira/browse/CALCITE-3841
> Project: Calcite
> Issue Type: Bug
> Components: site
>Reporter: Julian Hyde
>Assignee: Haisheng Yuan
>Priority: Major
> Fix For: 1.24.0
>
>
> Infra has
> [decided|https://lists.apache.org/thread.html/rcd2864e75e417597d342b8eb83080eb2d7a0cafea84fd4521a4d9cfd%40%3Cusers.infra.apache.org%3E]
> (login required for that email link) to deprecate
> [www.apache.org/dist|https://www.apache.org/dist] and move downloads to
> [https://downloads.apache.org|https://downloads.apache.org].
> On [Calcite's downloads page|https://calcite.apache.org/downloads/], we need
> to change the 'digest' link from (for example)
> {{https://www.apache.org/dist/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}}
> to
> {{https://downloads.apache.org/calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz.sha256}},
> and similarly the 'gpg' link.
> I believe that the 'tar' link can remain as
> {{https://www.apache.org/dyn/closer.lua?filename=calcite/apache-calcite-1.21.0/apache-calcite-1.21.0-src.tar.gz=download}}
> for the latest release and
> {{https://archive.apache.org/dist/calcite/apache-calcite-1.20.0/apache-calcite-1.20.0-src.tar.gz}}
> for older releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3916) Support cascades style top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157108#comment-17157108 ] Haisheng Yuan commented on CALCITE-3916: Yes, doing the final round review > Support cascades style top-down driven rule apply > - > > Key: CALCITE-3916 > URL: https://issues.apache.org/jira/browse/CALCITE-3916 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Haisheng Yuan >Assignee: Jinpeng Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.24.0 > > Time Spent: 12h 40m > Remaining Estimate: 0h > > Apply rules by leaf RelSet -> root RelSet order. For every RelNode in a > RelSet, rule is matched and applied sequentially. No RuleQueue and > DeferringRuleCall is needed anymore. This will make space pruning and rule > mutual exclusivity check possible. > Rule that use AbstractConverter as operand is an exception, to keep backward > compatibility, this kind of rule still needs top-down apply. > This should be done after CALCITE-3896. -- This message was sent by Atlassian Jira (v8.3.4#803005)
