[jira] [Updated] (CALCITE-4368) TopDownOptTest fails if applying non-substitution rule first
[
https://issues.apache.org/jira/browse/CALCITE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-4368:
-
Description:
If we apply non-substitution rules first, some tests in TopDownOptTest would
fail and it's not supposed to be. It can be reproduced by the following change:
{code:java}
// TopDownRuleQueue.java
private void addMatch(VolcanoRuleMatch match, Deque queue) {
if (!names.add(match.toString())) {
return;
}
if (planner.isSubstituteRule(match)) { // change this line
queue.addFirst(match);
} else {
queue.addLast(match);
}
}
{code}
The error message is:
{code:java}
java.lang.AssertionError
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:75)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:380)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:65)
at org.apache.calcite.rel.core.Join.copy(Join.java:310) at
org.apache.calcite.rel.core.Join.copy(Join.java:59)
at org.apache.calcite.rel.PhysicalNode.passThrough(PhysicalNode.java:88) at
org.apache.calcite.plan.volcano.RelSubset.passThrough(RelSubset.java:474)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.convert(TopDownRuleDriver.java:582)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.getOptimizeInputTask(TopDownRuleDriver.java:547)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.access$400(TopDownRuleDriver.java:49)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver$OptimizeGroup.perform(TopDownRuleDriver.java:349)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:103)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:520)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:122)
at org.apache.calcite.test.RelOptTestBase.access$000(RelOptTestBase.java:64)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:303)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:282)
at org.apache.calcite.test.Query.check(TopDownOptTest.java:824) at
org.apache.calcite.test.TopDownOptTest.testSortMergeJoin(TopDownOptTest.java:100)
{code}
was:
If we apply non-substitution rules first, some tests in TopDownOptTest would
fail and it's not supposed to be. It can be reproduced by the following change:
{code:java}
// TopDownRuleQueue.java
private void addMatch(VolcanoRuleMatch match, Deque queue) {
if (!names.add(match.toString())) {
return;
}
if (planner.isSubstituteRule(match)) {
queue.addFirst(match);
} else {
queue.addLast(match);
}
}
{code}
The error message is:
{code:java}
java.lang.AssertionError
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:75)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:380)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:65)
at org.apache.calcite.rel.core.Join.copy(Join.java:310) at
org.apache.calcite.rel.core.Join.copy(Join.java:59)
at org.apache.calcite.rel.PhysicalNode.passThrough(PhysicalNode.java:88) at
org.apache.calcite.plan.volcano.RelSubset.passThrough(RelSubset.java:474)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.convert(TopDownRuleDriver.java:582)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.getOptimizeInputTask(TopDownRuleDriver.java:547)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.access$400(TopDownRuleDriver.java:49)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver$OptimizeGroup.perform(TopDownRuleDriver.java:349)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:103)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:520)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:122)
at org.apache.calcite.test.RelOptTestBase.access$000(RelOptTestBase.java:64)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:303)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:282)
at org.apache.calcite.test.Query.check(TopDownOptTest.java:824) at
org.apache.calcite.test.TopDownOptTest.testSortMergeJoin(TopDownOptTest.java:100)
{code}
> TopDownOptTest fails if applying non-substitution rule first
>
>
> Key: CALCITE-4368
> URL: https://issues.apache.org/jira/browse/CALCITE-4368
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> If we apply non-substitution rules first, some tests in TopDownOptTest would
> fail and it's not supposed to be. It can be reproduced by the
[jira] [Updated] (CALCITE-4368) TopDownOptTest fails if applying non-substitution rule first
[
https://issues.apache.org/jira/browse/CALCITE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-4368:
-
Description:
If we apply non-substitution rules first, some tests in TopDownOptTest would
fail and it's not supposed to be. It can be reproduced by the following change:
{code:java}
// TopDownRuleQueue.java
private void addMatch(VolcanoRuleMatch match, Deque queue) {
if (!names.add(match.toString())) {
return;
}
if (planner.isSubstituteRule(match)) {
queue.addFirst(match);
} else {
queue.addLast(match);
}
}
{code}
The error message is:
{code:java}
java.lang.AssertionError
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:75)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:380)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:65)
at org.apache.calcite.rel.core.Join.copy(Join.java:310) at
org.apache.calcite.rel.core.Join.copy(Join.java:59)
at org.apache.calcite.rel.PhysicalNode.passThrough(PhysicalNode.java:88) at
org.apache.calcite.plan.volcano.RelSubset.passThrough(RelSubset.java:474)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.convert(TopDownRuleDriver.java:582)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.getOptimizeInputTask(TopDownRuleDriver.java:547)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.access$400(TopDownRuleDriver.java:49)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver$OptimizeGroup.perform(TopDownRuleDriver.java:349)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:103)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:520)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:122)
at org.apache.calcite.test.RelOptTestBase.access$000(RelOptTestBase.java:64)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:303)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:282)
at org.apache.calcite.test.Query.check(TopDownOptTest.java:824) at
org.apache.calcite.test.TopDownOptTest.testSortMergeJoin(TopDownOptTest.java:100)
{code}
{code:java}
{code}
was:
If we apply non-substitution rules first, some tests in TopDownOptTest would
fail. For example: TopDownOptTest#testSortMergeJoin
{code:java}
// code placeholder
java.lang.AssertionErrorjava.lang.AssertionError at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:75)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:380)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:65)
at org.apache.calcite.rel.core.Join.copy(Join.java:310) at
org.apache.calcite.rel.core.Join.copy(Join.java:59) at
org.apache.calcite.rel.PhysicalNode.passThrough(PhysicalNode.java:88) at
org.apache.calcite.plan.volcano.RelSubset.passThrough(RelSubset.java:474) at
org.apache.calcite.plan.volcano.TopDownRuleDriver.convert(TopDownRuleDriver.java:582)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.getOptimizeInputTask(TopDownRuleDriver.java:547)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.access$400(TopDownRuleDriver.java:49)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver$OptimizeGroup.perform(TopDownRuleDriver.java:349)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:103)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:520)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:122)
at org.apache.calcite.test.RelOptTestBase.access$000(RelOptTestBase.java:64) at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:303) at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:282) at
org.apache.calcite.test.Query.check(TopDownOptTest.java:824) at
org.apache.calcite.test.TopDownOptTest.testSortMergeJoin(TopDownOptTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at
org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
at
[jira] [Updated] (CALCITE-4368) TopDownOptTest fails if applying non-substitution rule first
[
https://issues.apache.org/jira/browse/CALCITE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-4368:
-
Description:
If we apply non-substitution rules first, some tests in TopDownOptTest would
fail and it's not supposed to be. It can be reproduced by the following change:
{code:java}
// TopDownRuleQueue.java
private void addMatch(VolcanoRuleMatch match, Deque queue) {
if (!names.add(match.toString())) {
return;
}
if (planner.isSubstituteRule(match)) {
queue.addFirst(match);
} else {
queue.addLast(match);
}
}
{code}
The error message is:
{code:java}
java.lang.AssertionError
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:75)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:380)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:65)
at org.apache.calcite.rel.core.Join.copy(Join.java:310) at
org.apache.calcite.rel.core.Join.copy(Join.java:59)
at org.apache.calcite.rel.PhysicalNode.passThrough(PhysicalNode.java:88) at
org.apache.calcite.plan.volcano.RelSubset.passThrough(RelSubset.java:474)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.convert(TopDownRuleDriver.java:582)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.getOptimizeInputTask(TopDownRuleDriver.java:547)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.access$400(TopDownRuleDriver.java:49)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver$OptimizeGroup.perform(TopDownRuleDriver.java:349)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:103)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:520)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:122)
at org.apache.calcite.test.RelOptTestBase.access$000(RelOptTestBase.java:64)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:303)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:282)
at org.apache.calcite.test.Query.check(TopDownOptTest.java:824) at
org.apache.calcite.test.TopDownOptTest.testSortMergeJoin(TopDownOptTest.java:100)
{code}
was:
If we apply non-substitution rules first, some tests in TopDownOptTest would
fail and it's not supposed to be. It can be reproduced by the following change:
{code:java}
// TopDownRuleQueue.java
private void addMatch(VolcanoRuleMatch match, Deque queue) {
if (!names.add(match.toString())) {
return;
}
if (planner.isSubstituteRule(match)) {
queue.addFirst(match);
} else {
queue.addLast(match);
}
}
{code}
The error message is:
{code:java}
java.lang.AssertionError
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:75)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:380)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:65)
at org.apache.calcite.rel.core.Join.copy(Join.java:310) at
org.apache.calcite.rel.core.Join.copy(Join.java:59)
at org.apache.calcite.rel.PhysicalNode.passThrough(PhysicalNode.java:88) at
org.apache.calcite.plan.volcano.RelSubset.passThrough(RelSubset.java:474)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.convert(TopDownRuleDriver.java:582)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.getOptimizeInputTask(TopDownRuleDriver.java:547)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.access$400(TopDownRuleDriver.java:49)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver$OptimizeGroup.perform(TopDownRuleDriver.java:349)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:103)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:520)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:122)
at org.apache.calcite.test.RelOptTestBase.access$000(RelOptTestBase.java:64)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:303)
at org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:282)
at org.apache.calcite.test.Query.check(TopDownOptTest.java:824) at
org.apache.calcite.test.TopDownOptTest.testSortMergeJoin(TopDownOptTest.java:100)
{code}
{code:java}
{code}
> TopDownOptTest fails if applying non-substitution rule first
>
>
> Key: CALCITE-4368
> URL: https://issues.apache.org/jira/browse/CALCITE-4368
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> If we apply non-substitution rules first, some tests in TopDownOptTest would
> fail and it's not supposed to be. It can be reproduced by
[jira] [Created] (CALCITE-4368) TopDownOptTest fails if applying non-substitution rule first
Chunwei Lei created CALCITE-4368:
Summary: TopDownOptTest fails if applying non-substitution rule
first
Key: CALCITE-4368
URL: https://issues.apache.org/jira/browse/CALCITE-4368
Project: Calcite
Issue Type: Improvement
Reporter: Chunwei Lei
Assignee: Chunwei Lei
If we apply non-substitution rules first, some tests in TopDownOptTest would
fail. For example: TopDownOptTest#testSortMergeJoin
{code:java}
// code placeholder
java.lang.AssertionErrorjava.lang.AssertionError at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.(EnumerableMergeJoin.java:75)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:380)
at
org.apache.calcite.adapter.enumerable.EnumerableMergeJoin.copy(EnumerableMergeJoin.java:65)
at org.apache.calcite.rel.core.Join.copy(Join.java:310) at
org.apache.calcite.rel.core.Join.copy(Join.java:59) at
org.apache.calcite.rel.PhysicalNode.passThrough(PhysicalNode.java:88) at
org.apache.calcite.plan.volcano.RelSubset.passThrough(RelSubset.java:474) at
org.apache.calcite.plan.volcano.TopDownRuleDriver.convert(TopDownRuleDriver.java:582)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.getOptimizeInputTask(TopDownRuleDriver.java:547)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.access$400(TopDownRuleDriver.java:49)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver$OptimizeGroup.perform(TopDownRuleDriver.java:349)
at
org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:103)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:520)
at
org.apache.calcite.test.RelOptTestBase.checkPlanning(RelOptTestBase.java:122)
at org.apache.calcite.test.RelOptTestBase.access$000(RelOptTestBase.java:64) at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:303) at
org.apache.calcite.test.RelOptTestBase$Sql.check(RelOptTestBase.java:282) at
org.apache.calcite.test.Query.check(TopDownOptTest.java:824) at
org.apache.calcite.test.TopDownOptTest.testSortMergeJoin(TopDownOptTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at
org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:139)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:81)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
[jira] [Comment Edited] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222756#comment-17222756 ] Chunwei Lei edited comment on CALCITE-4360 at 10/29/20, 7:51 AM: - BTW, I found all the test cases in TopDownOptTest failed if applying the substitution rule last in the investigation and It's supposed not to be. I would continue to investigate it and file another issue if there is something wrong. was (Author: chunwei lei): BTW, I found all the test cases in TopDownOptTest failed if applying the substitution rule last in the investigation and It's supposed not to be. I would continue to investigate it. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222756#comment-17222756 ] Chunwei Lei commented on CALCITE-4360: -- BTW, I found all the test cases in TopDownOptTest failed if applying the substitution rule last in the investigation and It's supposed not to be. I would continue to investigate it. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222749#comment-17222749 ] Chunwei Lei commented on CALCITE-4360: -- [~fan_li_ya] Makes sense. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei resolved CALCITE-4360. -- Resolution: Invalid > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222730#comment-17222730 ] Chunwei Lei commented on CALCITE-4360: -- I am sorry that I misunderstood the codes. Currently, the substitution rule is already applied first though it is not so straightforward. The process looks like: 1.put the non-substitution rule at the front and substitution rule at the end of the queue 2.get each rule from the queue in order from first to last and generate an ApplyRule task 3.push the ApplyRule task into the stack As you can see, the ApplyRule(substitution) task is poped earlier than the ApplyRule(non-substitution) task from the stack. Thus substitution rule is executed first. > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Attachments: image-2020-10-27-21-55-55-155.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Summary: Apply SubstitutionRule first in top-down driven rule apply (was: Should apply SubstitutionRule first in top-down driven rule apply) > Apply SubstitutionRule first in top-down driven rule apply > -- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Should apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Description: In the current implementation, TopDownRuleQueue adds substitute rules in the end wrongly. The SubstitutionRule should be executed first. !image-2020-10-27-21-55-55-155.png! was: In the current implementation, TopDownRuleQueue adds substitute rules in the end wrongly. The !image-2020-10-27-21-55-55-155.png! > Should apply SubstitutionRule first in top-down driven rule apply > - > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The SubstitutionRule should be executed first. > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Should apply SubstitutionRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Summary: Should apply SubstitutionRule first in top-down driven rule apply (was: Should apply SubstituteRule first in top-down driven rule apply) > Should apply SubstitutionRule first in top-down driven rule apply > - > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4360) Should apply SubstituteRule first in top-down driven rule apply
[ https://issues.apache.org/jira/browse/CALCITE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4360: - Description: In the current implementation, TopDownRuleQueue adds substitute rules in the end wrongly. The !image-2020-10-27-21-55-55-155.png! was:!image-2020-10-27-21-55-55-155.png! > Should apply SubstituteRule first in top-down driven rule apply > --- > > Key: CALCITE-4360 > URL: https://issues.apache.org/jira/browse/CALCITE-4360 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Attachments: image-2020-10-27-21-55-55-155.png > > > In the current implementation, TopDownRuleQueue adds substitute rules in the > end wrongly. The > !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4360) Should apply SubstituteRule first in top-down driven rule apply
Chunwei Lei created CALCITE-4360: Summary: Should apply SubstituteRule first in top-down driven rule apply Key: CALCITE-4360 URL: https://issues.apache.org/jira/browse/CALCITE-4360 Project: Calcite Issue Type: Improvement Reporter: Chunwei Lei Assignee: Chunwei Lei Attachments: image-2020-10-27-21-55-55-155.png !image-2020-10-27-21-55-55-155.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-947) Allow optional parameters to have default values other than NULL
[ https://issues.apache.org/jira/browse/CALCITE-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221113#comment-17221113 ] Chunwei Lei commented on CALCITE-947: - I am sorry that I cannot find time to finish it. Removed the assignee. > Allow optional parameters to have default values other than NULL > > > Key: CALCITE-947 > URL: https://issues.apache.org/jira/browse/CALCITE-947 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > > As of CALCITE-941 user-defined functions can have optional parameters and you > can omit them or use the DEFAULT keyword. But the default value must be NULL, > and therefore the parameter must have an optional type. > This case would remove that restriction and allow default values which are > literals, or calls to system functions (e.g. USER) or references to other > parameters, or expressions composed thereof. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-947) Allow optional parameters to have default values other than NULL
[ https://issues.apache.org/jira/browse/CALCITE-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei reassigned CALCITE-947: --- Assignee: (was: Chunwei Lei) > Allow optional parameters to have default values other than NULL > > > Key: CALCITE-947 > URL: https://issues.apache.org/jira/browse/CALCITE-947 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > > As of CALCITE-941 user-defined functions can have optional parameters and you > can omit them or use the DEFAULT keyword. But the default value must be NULL, > and therefore the parameter must have an optional type. > This case would remove that restriction and allow default values which are > literals, or calls to system functions (e.g. USER) or references to other > parameters, or expressions composed thereof. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-3746) RexSimplify changes the order of IS NOT NULL in And RexNode
[
https://issues.apache.org/jira/browse/CALCITE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei resolved CALCITE-3746.
--
Resolution: Invalid
> RexSimplify changes the order of IS NOT NULL in And RexNode
> ---
>
> Key: CALCITE-3746
> URL: https://issues.apache.org/jira/browse/CALCITE-3746
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.20.0
>Reporter: pengzhiwei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The RexSimplify changes the order of IS NOT NULL in And RexNode.The following
> expression
> {code:java}
> a is not null and length(a) > 0{code}
> is optimazted to
> {code:java}
> length(a) > 0 and a is not null{code}
> which will affect the logic short circuit for null-test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3746) RexSimplify changes the order of IS NOT NULL in And RexNode
[
https://issues.apache.org/jira/browse/CALCITE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221110#comment-17221110
]
Chunwei Lei commented on CALCITE-3746:
--
Since we cannot reach a consensus, close it now.
> RexSimplify changes the order of IS NOT NULL in And RexNode
> ---
>
> Key: CALCITE-3746
> URL: https://issues.apache.org/jira/browse/CALCITE-3746
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.20.0
>Reporter: pengzhiwei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The RexSimplify changes the order of IS NOT NULL in And RexNode.The following
> expression
> {code:java}
> a is not null and length(a) > 0{code}
> is optimazted to
> {code:java}
> length(a) > 0 and a is not null{code}
> which will affect the logic short circuit for null-test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-907) Add AggregateExchangeTransposeRule
[ https://issues.apache.org/jira/browse/CALCITE-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei reassigned CALCITE-907: --- Assignee: (was: Chunwei Lei) > Add AggregateExchangeTransposeRule > -- > > Key: CALCITE-907 > URL: https://issues.apache.org/jira/browse/CALCITE-907 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Priority: Major > > A rule that would recognize an Aggregate on top of an Exchange and push down > the aggregation. The result is a local Aggregate followed by an Exchange > followed by a summarizing Aggregate. > It would exploit SqlSplittableAggFunction, similar to CALCITE-751. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4332) Improve error when planning rule produces a relational expression with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei resolved CALCITE-4332.
--
Fix Version/s: 1.27.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/f9d23a62145e8c490492214836ce9cd7d1dcc35e.
> Improve error when planning rule produces a relational expression with wrong
> row type
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Current code in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4277) skip rule match when rel's relset is obsolete
[ https://issues.apache.org/jira/browse/CALCITE-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216421#comment-17216421 ] Chunwei Lei commented on CALCITE-4277: -- Sounds fine. > skip rule match when rel's relset is obsolete > - > > Key: CALCITE-4277 > URL: https://issues.apache.org/jira/browse/CALCITE-4277 > Project: Calcite > Issue Type: Improvement >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4338) Support MERGE INTO Clause
[
https://issues.apache.org/jira/browse/CALCITE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215214#comment-17215214
]
Chunwei Lei commented on CALCITE-4338:
--
Just out of curiosity， do other systems support this syntax except for Hive,
Oracle?
> Support MERGE INTO Clause
> -
>
> Key: CALCITE-4338
> URL: https://issues.apache.org/jira/browse/CALCITE-4338
> Project: Calcite
> Issue Type: Improvement
>Reporter: Forward Xu
>Priority: Major
>
> {code:java}
> MERGE INTO target_table
> USING source_table
> ON search_condition
> WHEN MATCHED THEN
> UPDATE SET col1 = value1, col2 = value2,...
> WHERE
> [DELETE WHERE ]
> WHEN NOT MATCHED THEN
> INSERT (col1,col2,...)
> values(value1,value2,...)
> WHERE ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4338) Support MERGE INTO Clause
[
https://issues.apache.org/jira/browse/CALCITE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215214#comment-17215214
]
Chunwei Lei edited comment on CALCITE-4338 at 10/16/20, 6:29 AM:
-
Just out of curiosity, do other systems support this syntax except for Hive,
Oracle?
was (Author: chunwei lei):
Just out of curiosity， do other systems support this syntax except for Hive,
Oracle?
> Support MERGE INTO Clause
> -
>
> Key: CALCITE-4338
> URL: https://issues.apache.org/jira/browse/CALCITE-4338
> Project: Calcite
> Issue Type: Improvement
>Reporter: Forward Xu
>Priority: Major
>
> {code:java}
> MERGE INTO target_table
> USING source_table
> ON search_condition
> WHEN MATCHED THEN
> UPDATE SET col1 = value1, col2 = value2,...
> WHERE
> [DELETE WHERE ]
> WHEN NOT MATCHED THEN
> INSERT (col1,col2,...)
> values(value1,value2,...)
> WHERE ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4341) Support double/float mod
[ https://issues.apache.org/jira/browse/CALCITE-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4341: - Summary: Support double/float mod (was: Calcite support double/float mod) > Support double/float mod > > > Key: CALCITE-4341 > URL: https://issues.apache.org/jira/browse/CALCITE-4341 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4332) Improve error when planning rule produces a relational expression with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215201#comment-17215201
]
Chunwei Lei commented on CALCITE-4332:
--
I am working on it.
> Improve error when planning rule produces a relational expression with wrong
> row type
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Assignee: Chunwei Lei
>Priority: Major
>
> Current code in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-4332) Improve error when planning rule produces a relational expression with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei reassigned CALCITE-4332:
Assignee: Chunwei Lei
> Improve error when planning rule produces a relational expression with wrong
> row type
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Assignee: Chunwei Lei
>Priority: Major
>
> Current code in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4260) Support plan explain with dot format
[ https://issues.apache.org/jira/browse/CALCITE-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei resolved CALCITE-4260. -- Fix Version/s: 1.27.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/e2f9296688ff0f99bf2966a481bd87b8fe11b075.] Thank you for your PR, [~fan_li_ya]! > Support plan explain with dot format > > > Key: CALCITE-4260 > URL: https://issues.apache.org/jira/browse/CALCITE-4260 > Project: Calcite > Issue Type: Task > Components: core >Reporter: Liya Fan >Assignee: Liya Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.27.0 > > Time Spent: 40m > Remaining Estimate: 0h > > After supporting the dot explain format and the dot relwriter, we are ready > to support plan explain with dot format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4332) Improve error when planning rule produces a relational expression with wrong row type
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213554#comment-17213554
]
Chunwei Lei commented on CALCITE-4332:
--
[~vladimirsitnikov] if you don'mind, I would like to take over this issue.
> Improve error when planning rule produces a relational expression with wrong
> row type
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Current code in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4332) Improve error when planning rule produces a relation with a wrong rowtype
[
https://issues.apache.org/jira/browse/CALCITE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212228#comment-17212228
]
Chunwei Lei commented on CALCITE-4332:
--
That would be great. I met the same confusing error message couple of times. It
is hard to figure out which field went wrong in the current message.
> Improve error when planning rule produces a relation with a wrong rowtype
> -
>
> Key: CALCITE-4332
> URL: https://issues.apache.org/jira/browse/CALCITE-4332
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.26.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> Current coed in {{VolcanoPlanner}}:
> {code:java}
> assert RelOptUtil.equal(
> "rel rowtype",
> rel.getRowType(),
> "equivRel rowtype",
> equivRel.getRowType(),
> Litmus.THROW);
> equivRel = ensureRegistered(equivRel, null);
> {code}
> The suggested changes:
> 1) Use regular error {{IllegalArgumentException}} rather than an assertion
> 2) Make sure the exception message contains fine-grained information on what
> went wrong. For instance: {{name: non nullable -> nullable; description:
> character -> int; ...}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212173#comment-17212173 ] Chunwei Lei edited comment on CALCITE-4223 at 10/12/20, 7:13 AM: - I would check it. Thank you for your reply( :D Just back from vacation). was (Author: chunwei lei): I would check it. Thank you for your reply(:DJust back from vacation). > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212173#comment-17212173 ] Chunwei Lei commented on CALCITE-4223: -- I would check it. Thank you for your reply(:DJust back from vacation). > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4310) Add API backward compatibility verification
[
https://issues.apache.org/jira/browse/CALCITE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212165#comment-17212165
]
Chunwei Lei commented on CALCITE-4310:
--
Interesting tools(y)
> Add API backward compatibility verification
> ---
>
> Key: CALCITE-4310
> URL: https://issues.apache.org/jira/browse/CALCITE-4310
> Project: Calcite
> Issue Type: Improvement
>Affects Versions: 1.25.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> One of the key questions when preparing a release is "what are the breaking
> changes?", and it would be nice if we could document the changes and minimize
> them.
> Here are the tools that might help:
> * japicmp: https://github.com/melix/japicmp-gradle-plugin (integrated with
> Gradle, enables to configure extra filters so we could exclude
> {{@API(internal)}} from public apis)
> * https://abi-laboratory.pro/java/tracker/ : (no idea how it compares with
> japicmp)
> * Kotlin-specific: https://github.com/Kotlin/binary-compatibility-validator
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203237#comment-17203237
]
Chunwei Lei commented on CALCITE-4223:
--
{quote}I don't know why Flink and Drill have not integrated their statistics
into Calcite. Maybe they didn't know how. They could have asked. Or we could
have written better documentation.
{quote}
I think it is not a good practice if users have to ask the community how to do
it and how to do it correctly. I have to admit that my proposal is not as
extendable as expected. What I want to do is that introducing column statistics
and taking advantage of them explicitly. BTW, it would be great if you can give
a PR to show how to introduce column statistics in the way you think it should
be. Thank you for your time~~
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202725#comment-17202725 ] Chunwei Lei commented on CALCITE-4223: -- Let's think about it in another way. A good interface is usually extendable and easily-and-widely-used by others, right? But what I find is that nobody uses the way you propose to introduce column statistics, including Flink[1], Drill[2]. I think it deserves to think twice. I am also glad to hear what others think about it. [1]https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/plan/stats/ColumnStats.java [2]https://github.com/apache/drill/blob/master/metastore/metastore-api/src/main/java/org/apache/drill/metastore/statistics/ColumnStatistics.java > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4241) Some improvements to metadata query
[
https://issues.apache.org/jira/browse/CALCITE-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200474#comment-17200474
]
Chunwei Lei commented on CALCITE-4241:
--
Fixed in
[https://github.com/apache/calcite/commit/e7ac26d6ec083848531b456d2acf3f7f58a499b5.]
Thank you for your PR, [~fan_li_ya]!
> Some improvements to metadata query
> ---
>
> Key: CALCITE-4241
> URL: https://issues.apache.org/jira/browse/CALCITE-4241
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> # For {{RelMdColumnUniqueness#areColumnsUnique(RelSubset, RelMetadataQuery,
> ImmutableBitSet, boolean)}}, we can return early as we encounter the first
> null.
> # For {{RelMdDistinctRowCount#getDistinctRowCount(Values, RelMetadataQuery,
> ImmutableBitSet, RexNode)}}, we can get the distinct row count accurately,
> instead of guessing that half of the elements are distinct.
> # {{RelMdUtil}}#{{areColumnsDefinitelyUniqueWhenNullsFiltered}} could be
> implemented in a more efficient way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4192) RelMdColumnOrigins gets the wrong index of group by columns after RelNode was optimized by AggregateProjectMergeRule rule
[ https://issues.apache.org/jira/browse/CALCITE-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei resolved CALCITE-4192. -- Fix Version/s: (was: next) 1.26.0 Resolution: Fixed Fixed in [https://github.com/apache/calcite/commit/fb1835c3741018f805b61aae7888f0afe65761ee.] Thank you for your PR, [~cuglifangzheng]! > RelMdColumnOrigins gets the wrong index of group by columns after RelNode was > optimized by AggregateProjectMergeRule rule > - > > Key: CALCITE-4192 > URL: https://issues.apache.org/jira/browse/CALCITE-4192 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.25.0 >Reporter: lifangzheng >Priority: Major > Labels: pull-request-available > Fix For: 1.26.0 > > Attachments: image-2020-08-25-15-53-19-575.png, > image-2020-08-25-15-55-31-702.png, image-2020-08-25-16-16-34-185.png > > Time Spent: 2h 20m > Remaining Estimate: 0h > > !image-2020-08-25-15-53-19-575.png|width=415,height=280! > > !image-2020-08-25-15-55-31-702.png|width=423,height=270! > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4192) RelMdColumnOrigins gets the wrong index of group by columns after RelNode was optimized by AggregateProjectMergeRule rule
[ https://issues.apache.org/jira/browse/CALCITE-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4192: - Summary: RelMdColumnOrigins gets the wrong index of group by columns after RelNode was optimized by AggregateProjectMergeRule rule (was: RelMdColumnOrigins get the wrong index of group by columns after RelNode was optimized by AggregateProjectMergeRule rule) > RelMdColumnOrigins gets the wrong index of group by columns after RelNode was > optimized by AggregateProjectMergeRule rule > - > > Key: CALCITE-4192 > URL: https://issues.apache.org/jira/browse/CALCITE-4192 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.25.0 >Reporter: lifangzheng >Priority: Major > Labels: pull-request-available > Fix For: next > > Attachments: image-2020-08-25-15-53-19-575.png, > image-2020-08-25-15-55-31-702.png, image-2020-08-25-16-16-34-185.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > !image-2020-08-25-15-53-19-575.png|width=415,height=280! > > !image-2020-08-25-15-55-31-702.png|width=423,height=270! > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CALCITE-4192) RelMdColumnOrigins get the wrong index of group by columns after RelNode was optimized by AggregateProjectMergeRule rule
[ https://issues.apache.org/jira/browse/CALCITE-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei updated CALCITE-4192: - Summary: RelMdColumnOrigins get the wrong index of group by columns after RelNode was optimized by AggregateProjectMergeRule rule (was: RelMdColumnOrigins get the wrong index of group by columns after relnode was optimized by AggregateProjectMergeRule rule.) > RelMdColumnOrigins get the wrong index of group by columns after RelNode was > optimized by AggregateProjectMergeRule rule > > > Key: CALCITE-4192 > URL: https://issues.apache.org/jira/browse/CALCITE-4192 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.25.0 >Reporter: lifangzheng >Priority: Major > Labels: pull-request-available > Fix For: next > > Attachments: image-2020-08-25-15-53-19-575.png, > image-2020-08-25-15-55-31-702.png, image-2020-08-25-16-16-34-185.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > !image-2020-08-25-15-53-19-575.png|width=415,height=280! > > !image-2020-08-25-15-55-31-702.png|width=423,height=270! > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199143#comment-17199143 ] Chunwei Lei commented on CALCITE-4223: -- Do you mean that we don't need to do any changes since we already have such a framework for users who want to introduce column statistics? > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198161#comment-17198161
]
Chunwei Lei commented on CALCITE-4223:
--
{quote}That doesn't work, because your {{ColStatistics}} doesn't have an
{{unwrap}} method.
{quote}
{{ColStatistics}} has an unwrap method since it extends {{Wrapper}}.
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198105#comment-17198105
]
Chunwei Lei commented on CALCITE-4223:
--
Thank you for your review, Julian.
{quote}it does not easily allow people to add new kinds of metadata, and it
does not accommodate differences in data structures that may have more
information (e.g. a system that has a histogram that returns not just number of
distinct values, but the number of distinct values between 100 and 1000)
{quote}
Users can implement ColStatistics and add new methods. Then use unwarp() to get
the customized ColStatistics. Comparing
{{table.unwrap(BuiltinMetadata.Size.class)}}, I think
sum({{table.getColumnStatistics(col).getAvgColLen())}} is much more
straightforward and readable.
Besides, does it mean that {{RelOptTable}} has to implement interfaces like
{{BuiltinMetadata.size/BuiltinMetadata.DistinctRowCount}} in your proposal?
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4197) Provide utility to visualize RelNode plans
[
https://issues.apache.org/jira/browse/CALCITE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197336#comment-17197336
]
Chunwei Lei edited comment on CALCITE-4197 at 9/17/20, 3:06 AM:
[~julianhyde], sorry about that. I did not notice that there're some
unaddressed comments, especially you said looks good and the PR got +1 from
other reviewers.
was (Author: chunwei lei):
[~julianhyde], I did not notice that there're some unaddressed comments,
especially you said looks good and the PR got +1 from other reviewers.
> Provide utility to visualize RelNode plans
> --
>
> Key: CALCITE-4197
> URL: https://issues.apache.org/jira/browse/CALCITE-4197
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.26.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png, screenshot-6.png, screenshot-7.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> From time to time, we want to visualize the plan generated the optimizer as a
> DAG (Directed Acyclic Graph). This gives us an overview of the plan, and
> helps us to find problems with the plan quickly.
> A common way to visualize a DAG is to dump it in the {{dot}} format, and then
> display the DAG through GraphViz. Currently, we already have a utility to
> dump the Volcano planner in dot format. In this issue, we want to dump a
> RelNode plan (a RelNode DAG) in the dot format, which will help users to see
> their plans more clearly.
> The utility should have some options that allow users to specify the
> visuzalization details, for example:
> 1. the max length of a node description.
> 2. whether RelSubset/HepRelVertex should be displayed.
> 3. Nodes that should be highlighted.
> Could you please give some feedback?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4197) Provide utility to visualize RelNode plans
[
https://issues.apache.org/jira/browse/CALCITE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197336#comment-17197336
]
Chunwei Lei commented on CALCITE-4197:
--
[~julianhyde], I did not notice that there're some unaddressed comments,
especially you said looks good and the PR got +1 from other reviewers.
> Provide utility to visualize RelNode plans
> --
>
> Key: CALCITE-4197
> URL: https://issues.apache.org/jira/browse/CALCITE-4197
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.26.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png, screenshot-6.png, screenshot-7.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> From time to time, we want to visualize the plan generated the optimizer as a
> DAG (Directed Acyclic Graph). This gives us an overview of the plan, and
> helps us to find problems with the plan quickly.
> A common way to visualize a DAG is to dump it in the {{dot}} format, and then
> display the DAG through GraphViz. Currently, we already have a utility to
> dump the Volcano planner in dot format. In this issue, we want to dump a
> RelNode plan (a RelNode DAG) in the dot format, which will help users to see
> their plans more clearly.
> The utility should have some options that allow users to specify the
> visuzalization details, for example:
> 1. the max length of a node description.
> 2. whether RelSubset/HepRelVertex should be displayed.
> 3. Nodes that should be highlighted.
> Could you please give some feedback?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4256) RexSimplify should not simplify P AND P to P, if it contains a call to RAND or RAND_INTEGER
[
https://issues.apache.org/jira/browse/CALCITE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195833#comment-17195833
]
Chunwei Lei commented on CALCITE-4256:
--
Agree with [~danny0405]. In our system, we treat SqlRandFunction as a
non-deterministic function.
> RexSimplify should not simplify P AND P to P, if it contains a call to RAND
> or RAND_INTEGER
> ---
>
> Key: CALCITE-4256
> URL: https://issues.apache.org/jira/browse/CALCITE-4256
> Project: Calcite
> Issue Type: Bug
>Reporter: Thomas Rebele
>Priority: Major
>
> Example: RAND_INTEGER() = 1 AND RAND_INTEGER() = 1 is false with a higher
> probability than RAND_INTEGER() = 1
> Here a test case for RexProgramTest:
> {code}
> @Test void testSimplifyRandomAnd() {
> checkSimplifyUnchanged(
> and(
> eq(rexBuilder.makeCall(SqlStdOperatorTable.RAND_INTEGER),
> literal(1)),
> eq(rexBuilder.makeCall(SqlStdOperatorTable.RAND_INTEGER),
> literal(1))
> ));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4250) Failed to get original column of aggregate operator
[
https://issues.apache.org/jira/browse/CALCITE-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195153#comment-17195153
]
Chunwei Lei commented on CALCITE-4250:
--
It seems duplicate with CALCITE-4192.
> Failed to get original column of aggregate operator
> ---
>
> Key: CALCITE-4250
> URL: https://issues.apache.org/jira/browse/CALCITE-4250
> Project: Calcite
> Issue Type: Wish
>Reporter: xzh_dz
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-09-12-20-58-06-444.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> We often use `RelMetadataQuery` to track column lineage, but failed to get
> original column of aggregate operator.
> This case can be reproduced as below, the correct column index is 7, but got
> 5.
> {code:java}
> // @Test void testAggregateRelMdColumnOrigins() {
> final RelBuilder relBuilder = RelBuilder.create(config().build());
> final LogicalProject project = (LogicalProject) relBuilder.scan("EMP")
> .project(relBuilder.field("EMPNO"),
> relBuilder.field("ENAME"),
> relBuilder.field("JOB"),
> relBuilder.field("SAL"),
> relBuilder.field("DEPTNO")).build();
> final LogicalAggregate aggregate = (LogicalAggregate)
> relBuilder.push(project)
> .aggregate(
> relBuilder.groupKey(relBuilder.field(1, 0, "DEPTNO")),
> relBuilder.count(relBuilder.field(1, 0, "SAL")))
> .build();
> RelMetadataQuery mq = aggregate.getCluster().getMetadataQuery();
> final RelColumnOrigin nameColumn = mq.getColumnOrigin(aggregate , 0);
>
> assertThat(nameColumn.getOriginTable().getRowType().getFieldNames().get(7).equals("DEPTNO"),
> is(true));
> System.out.println(nameColumn.getOriginColumnOrdinal()); // 5
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4239) RelMdUniqueKeys returns wrong unique keys for Aggregate with grouping sets
[
https://issues.apache.org/jira/browse/CALCITE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei resolved CALCITE-4239.
--
Fix Version/s: 1.26.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/1ae20f3e47d9ffab52e934c077d1b7629f798a2f.
> RelMdUniqueKeys returns wrong unique keys for Aggregate with grouping sets
> --
>
> Key: CALCITE-4239
> URL: https://issues.apache.org/jira/browse/CALCITE-4239
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.26.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> For Aggregate with grouping sets, group by keys might not form a unique key.
> For example:
> {code:java}
> //PostgreSQL
> create table test2(key bigint, value bigint);
> insert into test2 values
> (1, null),
> (null, 1);
> select key, value, count(*) from test2
> group by GROUPING SETS (key,value)
> ;
> postgres=# select key, value, count(*) from test2
> postgres-# group by GROUPING SETS (key,value)
> postgres-# ;
> key | value | count
> -+---+---
> | | 1
>1 | | 1
> | | 1
> | 1 | 1
> (4 rows)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4197) Provide utility to visualize RelNode plans
[
https://issues.apache.org/jira/browse/CALCITE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei resolved CALCITE-4197.
--
Fix Version/s: 1.26.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/99c0fefeca7caf638de875133f8c836472e8b28c.]
Thank you for your PR, [~fan_li_ya]!
> Provide utility to visualize RelNode plans
> --
>
> Key: CALCITE-4197
> URL: https://issues.apache.org/jira/browse/CALCITE-4197
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.26.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png, screenshot-6.png, screenshot-7.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> From time to time, we want to visualize the plan generated the optimizer as a
> DAG (Directed Acyclic Graph). This gives us an overview of the plan, and
> helps us to find problems with the plan quickly.
> A common way to visualize a DAG is to dump it in the {{dot}} format, and then
> display the DAG through GraphViz. Currently, we already have a utility to
> dump the Volcano planner in dot format. In this issue, we want to dump a
> RelNode plan (a RelNode DAG) in the dot format, which will help users to see
> their plans more clearly.
> The utility should have some options that allow users to specify the
> visuzalization details, for example:
> 1. the max length of a node description.
> 2. whether RelSubset/HepRelVertex should be displayed.
> 3. Nodes that should be highlighted.
> Could you please give some feedback?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193391#comment-17193391 ] Chunwei Lei commented on CALCITE-4223: -- Thank you for your helpful insight, [~julianhyde]. I will start to prepare the PR. > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4239) RelMdUniqueKeys returns wrong unique keys for Aggregate with grouping sets
[
https://issues.apache.org/jira/browse/CALCITE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei reassigned CALCITE-4239:
Assignee: Chunwei Lei
> RelMdUniqueKeys returns wrong unique keys for Aggregate with grouping sets
> --
>
> Key: CALCITE-4239
> URL: https://issues.apache.org/jira/browse/CALCITE-4239
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> For Aggregate with grouping sets, group by keys might not form a unique key.
> For example:
> {code:java}
> //PostgreSQL
> create table test2(key bigint, value bigint);
> insert into test2 values
> (1, null),
> (null, 1);
> select key, value, count(*) from test2
> group by GROUPING SETS (key,value)
> ;
> postgres=# select key, value, count(*) from test2
> postgres-# group by GROUPING SETS (key,value)
> postgres-# ;
> key | value | count
> -+---+---
> | | 1
>1 | | 1
> | | 1
> | 1 | 1
> (4 rows)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4239) RelMdUniqueKeys returns wrong unique keys for Aggregate with grouping sets
Chunwei Lei created CALCITE-4239:
Summary: RelMdUniqueKeys returns wrong unique keys for Aggregate
with grouping sets
Key: CALCITE-4239
URL: https://issues.apache.org/jira/browse/CALCITE-4239
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Chunwei Lei
For Aggregate with grouping sets, group by keys might not form a unique key.
For example:
{code:java}
//PostgreSQL
create table test2(key bigint, value bigint);
insert into test2 values
(1, null),
(null, 1);
select key, value, count(*) from test2
group by GROUPING SETS (key,value)
;
postgres=# select key, value, count(*) from test2
postgres-# group by GROUPING SETS (key,value)
postgres-# ;
key | value | count
-+---+---
| | 1
1 | | 1
| | 1
| 1 | 1
(4 rows)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192601#comment-17192601
]
Chunwei Lei commented on CALCITE-4223:
--
There is always an alternative due to the flexible interface. But I think it
would be great if Calcite has column statistics since it helps generate a
better plan.
Even if the column statistics are provided, we still get the statistics like
NDV using {{RelMetadataQuery}}. What we need to do is add/modify some methods
in RelMdxxx. Taking NDV for example:
{code:java}
// RelMdDistinctRowCount.java
public Double getDistinctRowCount(TableScan rel,
RelMetadataQuery mq, ImmutableBitSet groupKey,
RexNode predicate) {
..
List allDistinctValue =
getAllDistinctValue(rel.getTable().getColumnStatistics(), groupKey);
return getJointDistinctValue(allDistinctValue);
}{code}
Regarding no fixed definition of statistics, we can provide some basic and
frequently-used column statistics including NDV/AverageColumnSize/nullCount.
Let me conclude to help others understand more. There are two questions:
1) Do you agree to introduce column statistics?
2) If so, where should we put them? (RelOptTable? Statistics? Or other places)
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192601#comment-17192601
]
Chunwei Lei edited comment on CALCITE-4223 at 9/9/20, 3:50 AM:
---
There is always an alternative due to the flexible interface. But I think it
would be great if Calcite has column statistics since it helps generate a
better plan.
Even if the column statistics are provided, we still get the statistics like
NDV using {{RelMetadataQuery}}. What we need to do is add/modify some methods
in RelMdxxx. Taking NDV for example:
{code:java}
// RelMdDistinctRowCount.java
public Double getDistinctRowCount(TableScan rel,
RelMetadataQuery mq, ImmutableBitSet groupKey,
RexNode predicate) {
..
List allDistinctValue =
getAllDistinctValue(rel.getTable().getColumnStatistics(), groupKey);
return getJointDistinctValue(allDistinctValue);
}{code}
Regarding no fixed definition of statistics, we can provide some basic and
frequently-used column statistics including NDV/AverageColumnSize/nullCount.
Let me conclude to help others understand more. There are two questions:
1) Do you agree to introduce column statistics?
2) If so, where should we put them? (RelOptTable? Statistics? Or other places)
was (Author: chunwei lei):
There is always an alternative due to the flexible interface. But I think it
would be great if Calcite has column statistics since it helps generate a
better plan.
Even if the column statistics are provided, we still get the statistics like
NDV using {{RelMetadataQuery}}. What we need to do is add/modify some methods
in RelMdxxx. Taking NDV for example:
{code:java}
// RelMdDistinctRowCount.java
public Double getDistinctRowCount(TableScan rel,
RelMetadataQuery mq, ImmutableBitSet groupKey,
RexNode predicate) {
..
List allDistinctValue =
getAllDistinctValue(rel.getTable().getColumnStatistics(), groupKey);
return getJointDistinctValue(allDistinctValue);
}{code}
Regarding no fixed definition of statistics, we can provide some basic and
frequently-used column statistics including NDV/AverageColumnSize/nullCount.
Let me conclude to help others understand more. There are two questions:
1) Do you agree to introduce column statistics?
2) If so, where should we put them? (RelOptTable? Statistics? Or other places)
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191916#comment-17191916
]
Chunwei Lei commented on CALCITE-4223:
--
Thanks for sharing, [~danny0405]. IMO, one of the inconvenient places is that
Flink has to implement its own {{RelOptTable}} to get {{Statistic}} due to
{{RelOptTable}} does not provide a method to get {{Statistic}}.
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4224) Add an method for RelNode to output its tree digest like RelOptUtil.toString
[ https://issues.apache.org/jira/browse/CALCITE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191913#comment-17191913 ] Chunwei Lei commented on CALCITE-4224: -- You can add a method in a class that is already loaded before. Then you can always use this method. I don't have a strong opinion about it. Let us see what others think about it:D. > Add an method for RelNode to output its tree digest like RelOptUtil.toString > > > Key: CALCITE-4224 > URL: https://issues.apache.org/jira/browse/CALCITE-4224 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4224) Add an method for RelNode to output its tree digest like RelOptUtil.toString
[ https://issues.apache.org/jira/browse/CALCITE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191419#comment-17191419 ] Chunwei Lei commented on CALCITE-4224: -- Considering we can get it via RelOptUtil.toString()(get the result in Inteilj watches), is it really necessary? > Add an method for RelNode to output its tree digest like RelOptUtil.toString > > > Key: CALCITE-4224 > URL: https://issues.apache.org/jira/browse/CALCITE-4224 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191409#comment-17191409
]
Chunwei Lei edited comment on CALCITE-4223 at 9/7/20, 2:35 AM:
---
Let me make myself clear. What in my mind is that there is a method called
getColumnStatistics() in RelOptTable. Thus, we can get NDV(or other statistics)
of TableScan easily via
{{rel.getTable().getColumnStatistics().get(columnName).getNdv()}}. The
implementation of it can be delgated to {{Statistic#getColumnStatistics()}}. Do
you have any other ideas?
was (Author: chunwei lei):
Let me make myself clear. What in my mind is that there is a method called
getColumnStatistics() in RelOptTable. Thus, we can get NDV(or other statistics)
of TableScan easily via
{{rel.getTable().getColumnStatistics().get(columnName).getNdv()}}. The
implementation of it can be delgated to {{Statistic#getColumnStatistics()}}. Do
you have any other ideas?
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[
https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191409#comment-17191409
]
Chunwei Lei commented on CALCITE-4223:
--
Let me make myself clear. What in my mind is that there is a method called
getColumnStatistics() in RelOptTable. Thus, we can get NDV(or other statistics)
of TableScan easily via
{{rel.getTable().getColumnStatistics().get(columnName).getNdv()}}. The
implementation of it can be delgated to {{Statistic#getColumnStatistics()}}. Do
you have any other ideas?
> Introducing column statistics to RelOptTable
>
>
> Key: CALCITE-4223
> URL: https://issues.apache.org/jira/browse/CALCITE-4223
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
>
> Many systems depend on column statistics to compute more accurate stats, such
> as NDV, average column size, and so on. It would be nice if Calcite can
> provide such an interface.
> Column statistics might include NDV, average/max column length, number of
> nulls, number of trues, number of falses and so on.
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4224) Add an method for RelNode to output its tree digest like RelOptUtil.toString
[ https://issues.apache.org/jira/browse/CALCITE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190508#comment-17190508 ] Chunwei Lei commented on CALCITE-4224: -- I don't understand much about your proposal. Could you please provide an example? > Add an method for RelNode to output its tree digest like RelOptUtil.toString > > > Key: CALCITE-4224 > URL: https://issues.apache.org/jira/browse/CALCITE-4224 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Jiatao Tao >Assignee: Jiatao Tao >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190504#comment-17190504 ] Chunwei Lei edited comment on CALCITE-4223 at 9/4/20, 2:58 AM: --- [~julianhyde], column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses, TopK. Some systems like Hive[1] provide a command to collect these stats. Providing we have such column stats, we can: 1) get more accurate NDV of table scan than estimation. 2) estimate more accurate size of inputs of Join if the columns' types include varchar, which helps decide whether to use HashJoin or MergeJoin(Because we have average column length). [1] https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics[1] was (Author: chunwei lei): [~julianhyde], column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses, TopK. Some systems like Hive[1] provide a command to collect these stats. Providing we have such column stats, we can: 1) get more accurate NDV of table scan than estimation. 2) estimate more accurate size of inputs of Join if the columns' types include varchar, which helps decide whether to use HashJoin or MergeJoin(Because we have average column length). [1]https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190504#comment-17190504 ] Chunwei Lei edited comment on CALCITE-4223 at 9/4/20, 2:58 AM: --- [~julianhyde], column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses, TopK. Some systems like Hive[1] provide a command to collect these stats. Providing we have such column stats, we can: 1) get more accurate NDV of table scan than estimation. 2) estimate more accurate size of inputs of Join if the columns' types include varchar, which helps decide whether to use HashJoin or MergeJoin(Because we have average column length). [1] https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics was (Author: chunwei lei): [~julianhyde], column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses, TopK. Some systems like Hive[1] provide a command to collect these stats. Providing we have such column stats, we can: 1) get more accurate NDV of table scan than estimation. 2) estimate more accurate size of inputs of Join if the columns' types include varchar, which helps decide whether to use HashJoin or MergeJoin(Because we have average column length). [1] [https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics|https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics[1] > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190504#comment-17190504 ] Chunwei Lei edited comment on CALCITE-4223 at 9/4/20, 2:58 AM: --- [~julianhyde], column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses, TopK. Some systems like Hive[1] provide a command to collect these stats. Providing we have such column stats, we can: 1) get more accurate NDV of table scan than estimation. 2) estimate more accurate size of inputs of Join if the columns' types include varchar, which helps decide whether to use HashJoin or MergeJoin(Because we have average column length). [1] [https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics|https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics[1] was (Author: chunwei lei): [~julianhyde], column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses, TopK. Some systems like Hive[1] provide a command to collect these stats. Providing we have such column stats, we can: 1) get more accurate NDV of table scan than estimation. 2) estimate more accurate size of inputs of Join if the columns' types include varchar, which helps decide whether to use HashJoin or MergeJoin(Because we have average column length). [1] https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics[1] > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190504#comment-17190504 ] Chunwei Lei commented on CALCITE-4223: -- [~julianhyde], column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses, TopK. Some systems like Hive[1] provide a command to collect these stats. Providing we have such column stats, we can: 1) get more accurate NDV of table scan than estimation. 2) estimate more accurate size of inputs of Join if the columns' types include varchar, which helps decide whether to use HashJoin or MergeJoin(Because we have average column length). [1]https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ColumnStatistics > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190493#comment-17190493 ] Chunwei Lei commented on CALCITE-4223: -- [~amaliujia] , column stats are used in Hive[1], Flink[2], Spark[3]. [1] [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java] [2][https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/plan/stats/ColumnStats.java] [[3] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala] > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CALCITE-4223) Introducing column statistics to RelOptTable
[ https://issues.apache.org/jira/browse/CALCITE-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunwei Lei reassigned CALCITE-4223: Assignee: Chunwei Lei > Introducing column statistics to RelOptTable > > > Key: CALCITE-4223 > URL: https://issues.apache.org/jira/browse/CALCITE-4223 > Project: Calcite > Issue Type: Improvement >Reporter: Chunwei Lei >Assignee: Chunwei Lei >Priority: Major > > Many systems depend on column statistics to compute more accurate stats, such > as NDV, average column size, and so on. It would be nice if Calcite can > provide such an interface. > Column statistics might include NDV, average/max column length, number of > nulls, number of trues, number of falses and so on. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CALCITE-4223) Introducing column statistics to RelOptTable
Chunwei Lei created CALCITE-4223: Summary: Introducing column statistics to RelOptTable Key: CALCITE-4223 URL: https://issues.apache.org/jira/browse/CALCITE-4223 Project: Calcite Issue Type: Improvement Reporter: Chunwei Lei Many systems depend on column statistics to compute more accurate stats, such as NDV, average column size, and so on. It would be nice if Calcite can provide such an interface. Column statistics might include NDV, average/max column length, number of nulls, number of trues, number of falses and so on. What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CALCITE-4203) RelMdUniqueKeys should not return empty when meeting Intersect and Minus if its input has unique keys
[
https://issues.apache.org/jira/browse/CALCITE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei resolved CALCITE-4203.
--
Fix Version/s: 1.26.0
Resolution: Fixed
Fixed in
https://github.com/apache/calcite/commit/103430bcc76ed1a7cbab6861ecdd39814cf4ef83.
> RelMdUniqueKeys should not return empty when meeting Intersect and Minus if
> its input has unique keys
> -
>
> Key: CALCITE-4203
> URL: https://issues.apache.org/jira/browse/CALCITE-4203
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.26.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, we get an empty unique key for Intersect and Minus if its all is
> true.
>
> {code:java}
> // code placeholder
> public Set getUniqueKeys(SetOp rel, RelMetadataQuery mq,
> boolean ignoreNulls) {
> if (!rel.all) {
> return ImmutableSet.of(
> ImmutableBitSet.range(rel.getRowType().getFieldCount()));
> }
> return ImmutableSet.of();
> }
> {code}
> However, from the semantic of Intersect and Minus, we can get their unique
> keys from its input even if its all is true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4180) Support for Elasticsearch basic authentication
[
https://issues.apache.org/jira/browse/CALCITE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188940#comment-17188940
]
Chunwei Lei commented on CALCITE-4180:
--
Updated the title to make it clearer.
> Support for Elasticsearch basic authentication
> --
>
> Key: CALCITE-4180
> URL: https://issues.apache.org/jira/browse/CALCITE-4180
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: fa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> {code:java}
> Properties config = new Properties();
> config.put("lex", "JAVA");
> String sql = "select * from index";
> try (Connection con =
> DriverManager.getConnection("jdbc:calcite:model=src/main/resources/es.json",
> config)) {
> try (Statement stmt = con.createStatement()) {
> try (ResultSet rs = stmt.executeQuery(sql)) {
> printRs(rs);
> }
> }
> }
> {code}
>
>
> es.json
> {code:java}
> {
> "version": "1.0",
> "defaultSchema": "elasticsearch",
> "schemas": [
> {
> "type": "custom",
> "name": "elasticsearch",
> "factory":
> "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
> "operand": {
> "coordinates": "{'192.168.133.104': 9200}",
> "jdbcUser": "elastic",
> "jdbcPassword": "elastic"
> }
> }
> ]
> }
> {code}
> and throw Exception
> {code:java}
> {
> "error":{
> "root_cause":[
> {
> "type":"security_exception",
> "reason":"missing authentication token for REST request
> [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security"
> charset="UTF-8""
> }
> }
> ],
> "type":"security_exception",
> "reason":"missing authentication token for REST request [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security" charset="UTF-8""
> }
> },
> "status":401
> }
> {code}
> Where to set Elasticsearch username/password?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4180) Support for Elasticsearch basic authentication
[
https://issues.apache.org/jira/browse/CALCITE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-4180:
-
Summary: Support for Elasticsearch basic authentication (was: Where to set
elasticsearch username/password?)
> Support for Elasticsearch basic authentication
> --
>
> Key: CALCITE-4180
> URL: https://issues.apache.org/jira/browse/CALCITE-4180
> Project: Calcite
> Issue Type: Bug
> Components: elasticsearch-adapter
>Reporter: fa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> {code:java}
> Properties config = new Properties();
> config.put("lex", "JAVA");
> String sql = "select * from index";
> try (Connection con =
> DriverManager.getConnection("jdbc:calcite:model=src/main/resources/es.json",
> config)) {
> try (Statement stmt = con.createStatement()) {
> try (ResultSet rs = stmt.executeQuery(sql)) {
> printRs(rs);
> }
> }
> }
> {code}
>
>
> es.json
> {code:java}
> {
> "version": "1.0",
> "defaultSchema": "elasticsearch",
> "schemas": [
> {
> "type": "custom",
> "name": "elasticsearch",
> "factory":
> "org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
> "operand": {
> "coordinates": "{'192.168.133.104': 9200}",
> "jdbcUser": "elastic",
> "jdbcPassword": "elastic"
> }
> }
> ]
> }
> {code}
> and throw Exception
> {code:java}
> {
> "error":{
> "root_cause":[
> {
> "type":"security_exception",
> "reason":"missing authentication token for REST request
> [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security"
> charset="UTF-8""
> }
> }
> ],
> "type":"security_exception",
> "reason":"missing authentication token for REST request [/_alias]",
> "header":{
> "WWW-Authenticate":"Basic realm="security" charset="UTF-8""
> }
> },
> "status":401
> }
> {code}
> Where to set Elasticsearch username/password?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4209) RelBuilder: ability to avoid generating empty() on limit 0
[
https://issues.apache.org/jira/browse/CALCITE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188939#comment-17188939
]
Chunwei Lei commented on CALCITE-4209:
--
{{relBuilder.filter(false)}} can generate empty() too. Should we also provide
this ability?
> RelBuilder: ability to avoid generating empty() on limit 0
> --
>
> Key: CALCITE-4209
> URL: https://issues.apache.org/jira/browse/CALCITE-4209
> Project: Calcite
> Issue Type: Improvement
>Reporter: Steven Talbot
>Priority: Major
>
> When you call
> {code:java}
> relBuilder.limit(0, 0){code}
> It triggers a call to RelBuilder.empty at
> [https://github.com/apache/calcite/blob/88d18185e6177c9df587bdd23dd4049f59adc2e4/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2531].
> This is fine as the default behavior, but for cases where we intend to
> convert the relational algebra to SQL and display it somewhere, it would be
> preferable to turn this behavior off, either as an extra flag to this method
> or a configuration parameter. The call to empty() often results in a VALUES
> or a select list of literal NULLs, which looks ugly and can confuse a user.
> Moreover, there are certain databases (like BigQuery) that can cheaply handle
> a LIMIT 0 query as a form of validation, and if we munge the query with
> 'empty()' we lose the ability to validate its correctness by going to the DB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4203) RelMdUniqueKeys should not return empty when meeting Intersect and Minus if its input has unique keys
[
https://issues.apache.org/jira/browse/CALCITE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188937#comment-17188937
]
Chunwei Lei commented on CALCITE-4203:
--
I have no problem with that. I would update the PR.
> RelMdUniqueKeys should not return empty when meeting Intersect and Minus if
> its input has unique keys
> -
>
> Key: CALCITE-4203
> URL: https://issues.apache.org/jira/browse/CALCITE-4203
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, we get an empty unique key for Intersect and Minus if its all is
> true.
>
> {code:java}
> // code placeholder
> public Set getUniqueKeys(SetOp rel, RelMetadataQuery mq,
> boolean ignoreNulls) {
> if (!rel.all) {
> return ImmutableSet.of(
> ImmutableBitSet.range(rel.getRowType().getFieldCount()));
> }
> return ImmutableSet.of();
> }
> {code}
> However, from the semantic of Intersect and Minus, we can get their unique
> keys from its input even if its all is true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4197) Provide utility to visualize RelNode plans
[
https://issues.apache.org/jira/browse/CALCITE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188932#comment-17188932
]
Chunwei Lei commented on CALCITE-4197:
--
I like this feature. It is really helpful.
> Provide utility to visualize RelNode plans
> --
>
> Key: CALCITE-4197
> URL: https://issues.apache.org/jira/browse/CALCITE-4197
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> From time to time, we want to visualize the plan generated the optimizer as a
> DAG (Directed Acyclic Graph). This gives us an overview of the plan, and
> helps us to find problems with the plan quickly.
> A common way to visualize a DAG is to dump it in the {{dot}} format, and then
> display the DAG through GraphViz. Currently, we already have a utility to
> dump the Volcano planner in dot format. In this issue, we want to dump a
> RelNode plan (a RelNode DAG) in the dot format, which will help users to see
> their plans more clearly.
> The utility should have some options that allow users to specify the
> visuzalization details, for example:
> 1. the max length of a node description.
> 2. whether RelSubset/HepRelVertex should be displayed.
> 3. Nodes that should be highlighted.
> Could you please give some feedback?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4208) Improve metadata row count for Join
[
https://issues.apache.org/jira/browse/CALCITE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188931#comment-17188931
]
Chunwei Lei commented on CALCITE-4208:
--
AFAIK, the row count of Join is related to the NDV. Should we move toward this
direction?
> Improve metadata row count for Join
> ---
>
> Key: CALCITE-4208
> URL: https://issues.apache.org/jira/browse/CALCITE-4208
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Ruben Q L
>Priority: Major
>
> Currently, the default metadata row count for join
> {{RelMdRowCount#getRowCount(Join rel, RelMetadataQuery mq)}} relies on
> {{RelMdUtil.getJoinRowCount}}. This method has several issues:
> - In case of ANTI join, it returns the same estimation as a SEMI join
> - In other cases (INNER, LEFT, RIGHT, FULL), it returns always the same
> formula:
> {{leftRowCount * rightRowCount * mq.getSelectivity(join, condition)}}
> which seems valid for an INNER join, but not for LEFT / RIGHT / FULL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4203) RelMdUniqueKeys should not return empty when meeting Intersect and Minus if its input has unique keys
[
https://issues.apache.org/jira/browse/CALCITE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188173#comment-17188173
]
Chunwei Lei commented on CALCITE-4203:
--
Thanks for your review, Julian.
> Is there any benefit in delegating to {{getUniqueKeys((SetOp) r, mq,
>ignoreNulls)}}?
For reusing the code.
> In fact I think {{getUniqueKeys(SetOp, RelMetadataQuery, boolean)}} should be
> changed to {{getUniqueKeys(Union, RelMetadataQuery, boolean)}}
I don't do this in case it may break the downstream project if it has a
subclass of SetOp except Minus and Intersect.
> Use {{LinkedHashSet}} or {{TreeSet}} rather than {{HashSet}}, for determinism.
> Add tests
> Some comments would be nice. The unique keys of Minus are precisely the
>unique keys of its first input. Any unique key of any input of Intersect is a
>unique key of the Intersect.
Makes sense.
> Are your methods consistent with {{RelMdColumnUniqueness}} for {{Intersect}}
>and {{Minus?}}
I think so.
> RelMdUniqueKeys should not return empty when meeting Intersect and Minus if
> its input has unique keys
> -
>
> Key: CALCITE-4203
> URL: https://issues.apache.org/jira/browse/CALCITE-4203
> Project: Calcite
> Issue Type: Improvement
>Reporter: Chunwei Lei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, we get an empty unique key for Intersect and Minus if its all is
> true.
>
> {code:java}
> // code placeholder
> public Set getUniqueKeys(SetOp rel, RelMetadataQuery mq,
> boolean ignoreNulls) {
> if (!rel.all) {
> return ImmutableSet.of(
> ImmutableBitSet.range(rel.getRowType().getFieldCount()));
> }
> return ImmutableSet.of();
> }
> {code}
> However, from the semantic of Intersect and Minus, we can get their unique
> keys from its input even if its all is true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CALCITE-4203) RelMdUniqueKeys should not return empty when meeting Intersect and Minus if its input has unique keys
Chunwei Lei created CALCITE-4203:
Summary: RelMdUniqueKeys should not return empty when meeting
Intersect and Minus if its input has unique keys
Key: CALCITE-4203
URL: https://issues.apache.org/jira/browse/CALCITE-4203
Project: Calcite
Issue Type: Improvement
Reporter: Chunwei Lei
Assignee: Chunwei Lei
Currently, we get an empty unique key for Intersect and Minus if its all is
true.
{code:java}
// code placeholder
public Set getUniqueKeys(SetOp rel, RelMetadataQuery mq,
boolean ignoreNulls) {
if (!rel.all) {
return ImmutableSet.of(
ImmutableBitSet.range(rel.getRowType().getFieldCount()));
}
return ImmutableSet.of();
}
{code}
However, from the semantic of Intersect and Minus, we can get their unique keys
from its input even if its all is true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4199) Add nullability annotations to the methods and fields, ensure consistency with checkerframework
[
https://issues.apache.org/jira/browse/CALCITE-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187590#comment-17187590
]
Chunwei Lei commented on CALCITE-4199:
--
If it is enforced, does it take a lot of effort to change the current code
base? BTW, does your findings 1) mean the checkerframework would increase much
compile time?
> Add nullability annotations to the methods and fields, ensure consistency
> with checkerframework
> ---
>
> Key: CALCITE-4199
> URL: https://issues.apache.org/jira/browse/CALCITE-4199
> Project: Calcite
> Issue Type: New Feature
>Affects Versions: 1.25.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> The current codebase uses jsr305 javax.annotation.Nullable / NonNull which
> helps to catch bugs while developing Calcite and libraries.
> Unfortunately, the current annotation is not enforced, and it lacks support
> for generics.
> jsr305.jar is probably abandoned (see
> https://github.com/google/guava/issues/2960), so we should probably migrate
> to something else.
> https://checkerframework.org/ is a solid framework for machine verification
> which is tailored to Java.
> The releases are quite frequent:
> https://github.com/typetools/checker-framework/releases
> Their annotations are recognized by major IDEs.
> So I see the following options:
> a) Leave the code as is
> b) Annotate (gradually?) the code with checkerframework annotations
> c) Migrate (gradually?) the code to Kotlin
> I've created a PR to verify which changes would be needed, verify if CI is
> able to type check in a reasonable time, and so on:
> https://github.com/apache/calcite/pull/1929
> My findings regarding checkerframework so far are:
> 0) It does detect NPEs (which were hidden in the current code), and it does
> detect initialized files in the constructors
> 1) Checkerframework takes ~2 minutes for linq4j, and 13+min (unknown yet, 13m
> produces 100 errors and stops) for core
> 2) "non-nullable by default" is quite close to the current Calcite
> conventions.
> 3) There are cases when javadoc comments says "or null", however, the code
> reads much easier if {{@Nullable}} nullable appears in the signature
> 4) If a third-party library supplies invalid type annotations, there's a way
> to fix that. For instance, Guava's Function is annotated as "always
> nullable", and we can overrule that (so the nullability is taken from generic
> signature rather than "always nullable"). The override files are placed to
> src/main/config/checkerframework
> 5) Generic-heavy code might be challenging (they are always like that),
> however, in the most obscure cases there's always a way to suppress the
> warning
> 6) I've filed a Gradle improvement so it schedules recently modified files
> first for the compilation: https://github.com/gradle/gradle/issues/14332
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4199) Add nullability annotations to the methods and fields, ensure consistency with checkerframework
[
https://issues.apache.org/jira/browse/CALCITE-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187373#comment-17187373
]
Chunwei Lei commented on CALCITE-4199:
--
Does it mean the annotation is enforced?
> Add nullability annotations to the methods and fields, ensure consistency
> with checkerframework
> ---
>
> Key: CALCITE-4199
> URL: https://issues.apache.org/jira/browse/CALCITE-4199
> Project: Calcite
> Issue Type: New Feature
>Affects Versions: 1.25.0
>Reporter: Vladimir Sitnikov
>Priority: Major
>
> The current codebase uses jsr305 javax.annotation.Nullable / NonNull which
> helps to catch bugs while developing Calcite and libraries.
> Unfortunately, the current annotation is not enforced, and it lacks support
> for generics.
> jsr305.jar is probably abandoned (see
> https://github.com/google/guava/issues/2960), so we should probably migrate
> to something else.
> https://checkerframework.org/ is a solid framework for machine verification
> which is tailored to Java.
> The releases are quite frequent:
> https://github.com/typetools/checker-framework/releases
> Their annotations are recognized by major IDEs.
> So I see the following options:
> a) Leave the code as is
> b) Annotate (gradually?) the code with checkerframework annotations
> c) Migrate (gradually?) the code to Kotlin
> I've created a PR to verify which changes would be needed, verify if CI is
> able to type check in a reasonable time, and so on:
> https://github.com/apache/calcite/pull/1929
> My findings regarding checkerframework so far are:
> 0) It does detect NPEs (which were hidden in the current code), and it does
> detect initialized files in the constructors
> 1) Checkerframework takes ~2 minutes for linq4j, and 13+min (unknown yet, 13m
> produces 100 errors and stops) for core
> 2) "non-nullable by default" is quite close to the current Calcite
> conventions.
> 3) There are cases when javadoc comments says "or null", however, the code
> reads much easier if {{@Nullable}} nullable appears in the signature
> 4) If a third-party library supplies invalid type annotations, there's a way
> to fix that. For instance, Guava's Function is annotated as "always
> nullable", and we can overrule that (so the nullability is taken from generic
> signature rather than "always nullable"). The override files are placed to
> src/main/config/checkerframework
> 5) Generic-heavy code might be challenging (they are always like that),
> however, in the most obscure cases there's always a way to suppress the
> warning
> 6) I've filed a Gradle improvement so it schedules recently modified files
> first for the compilation: https://github.com/gradle/gradle/issues/14332
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4191) Improve the logic of creating aggregate calls
[
https://issues.apache.org/jira/browse/CALCITE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184884#comment-17184884
]
Chunwei Lei commented on CALCITE-4191:
--
Personally I would like to have such a way to create {{AggregateCall}}
considering so many parameters.
> Improve the logic of creating aggregate calls
> -
>
> Key: CALCITE-4191
> URL: https://issues.apache.org/jira/browse/CALCITE-4191
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> According to the current code base, the only way to create \{{AggregateCall}}
> objects is by calling one of the two \{{AggregateCall#create}} methods (other
> create methods are deprecated).
> The two {{create}} methods have 9 and 11 parameters, respectively, 3 of which
> are booleans and 2 are ints. We find this makes the code less readable and
> error-prone, as some bugs are caused by specifying the wrong parameters.
> In this issue, we improve the related logic by the builder pattern, which
> results in the following benefits:
> 1. By creating the objects by the builder pattern, there is no need to
> maintain multiple overrides of the {{create}} methods.
> 2. There is no need to maintain multiple overrides of the {{copy}} methods,
> either.
> 3. The code becomes more readable and less error-prone, as it is less like to
> specify the wrong parameter.
> 4. Creating {{AggregateCall}} objects becomes easier, as the user does not
> have specify the default parameters repeatedly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-1581) UDTF like in hive
[
https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176737#comment-17176737
]
Chunwei Lei commented on CALCITE-1581:
--
I am not sure whether it should use {{LogicalCorrelate when meeting
LogicalTableFunctionScan}}. But Some existed examples use
{{LogicalCorrelate}}[1]. Any suggestions will be appreciated.
[1]
[https://github.com/apache/calcite/blob/master/core/src/test/resources/org/apache/calcite/test/SqlToRelConverterTest.xml#L320]
> UDTF like in hive
> -
>
> Key: CALCITE-1581
> URL: https://issues.apache.org/jira/browse/CALCITE-1581
> Project: Calcite
> Issue Type: New Feature
>Reporter: Xiaoyong Deng
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available, udtf
> Fix For: 1.25.0
>
> Time Spent: 14h 50m
> Remaining Estimate: 0h
>
> Support one row in and multi-column/multi-row out(one-to-many mapping), just
> like udtf in hive.
> The query would like this:
> {code}
> select
> func(c0, c1) as (f0, f1, f2)
> from table_name;
> {code}
> c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-3351) SQLException when executing SQL containing Chinese
[
https://issues.apache.org/jira/browse/CALCITE-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-3351:
-
Summary: SQLException when executing SQL containing Chinese (was: calcite
mysql utf8 throws exception in Chinese)
> SQLException when executing SQL containing Chinese
> --
>
> Key: CALCITE-3351
> URL: https://issues.apache.org/jira/browse/CALCITE-3351
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.19.0
> Environment: Happened on both Windows and MacOS.
>Reporter: cui
>Priority: Major
> Labels: pull-request-available
> Attachments: e.condition.png, toSql Result.png
>
> Original Estimate: 24h
> Time Spent: 10m
> Remaining Estimate: 23h 50m
>
> Exception was thrown when connecting with mySQL to query Chinese. The
> following SQL was executed:
> select * from \"test\".\"score_new\" where \"name\"= \'催\' limit 1
> ，if change "getDefaultCharset" in "RelDataTypeFactoryImpl" to
> return Charset.forName("UTF8");
> there will be also exception thrown as the following:
> java.sql.SQLException: Error while executing SQL "select * from
> "test"."score_new" where "name"= '催' limit 1": While executing SQL [SELECT
> *java.sql.SQLException: Error while executing SQL "select * from
> "test"."score_new" where "name"= '催' limit 1": While executing SQL [SELECT
> *FROM `score_new`WHERE `name` = u&'\50ac'LIMIT 1] on JDBC sub-schema at
> org.apache.calcite.avatica.Helper.createException(Helper.java:56) at
> org.apache.calcite.avatica.Helper.createException(Helper.java:41) at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at QueryMysql.main(QueryMysql.java:42)Caused by: java.lang.RuntimeException:
> While executing SQL [SELECT *FROM `score_new`WHERE `name` = u&'\50ac'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3409) Add an interface in MaterializedViewSubstitutionVisitor to allow registering UnifyRule
[
https://issues.apache.org/jira/browse/CALCITE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175946#comment-17175946
]
Chunwei Lei commented on CALCITE-3409:
--
I left some comments on PR#2094.
> Add an interface in MaterializedViewSubstitutionVisitor to allow registering
> UnifyRule
> --
>
> Key: CALCITE-3409
> URL: https://issues.apache.org/jira/browse/CALCITE-3409
> Project: Calcite
> Issue Type: Improvement
>Reporter: Jin Xing
>Assignee: Jin Xing
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> In current code of MaterializedViewSubstitutionVisitor, all matching rules
> are internal defined. The existing rules support the most popular scenarios.
> But my customers sometimes ask for the ability to self define some matching
> rules, thus to support some special scenarios.
> I take below example as an illustration:
> {code:java}
> Query:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd hh") >= "20190909 00"
> and from_unixtime(_EVENT_TIME_, "mmdd hh") <= "20190909 23" ;
> Materialized View:
> select * from table
> where from_unixtime(_EVENT_TIME_, "mmdd") = "20190909";{code}
> It's hard to enumerate the matching pattern for different functions in
> internal matching rules. We can expose a method to register new UnifyRules
> and allow user to extend the ability of MV matching
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4081) Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter -> Planner throws validation error
[
https://issues.apache.org/jira/browse/CALCITE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei resolved CALCITE-4081.
--
Fix Version/s: 1.25.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/63c1667459d594cf4761e794b5f882a49a9ae197].
> Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter ->
> Planner throws validation error
>
>
> Key: CALCITE-4081
> URL: https://issues.apache.org/jira/browse/CALCITE-4081
> Project: Calcite
> Issue Type: Bug
>Reporter: Steven Talbot
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.25.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Take the following SQL:
> SELECT CAST('.11' AS DECIMAL(38,6)) AS
> "num"
> And, in the default Calcite SQL dialect with all defaults, send it through
> Planner parse -> valiate -> rel to get a RelNode. The RelNode will have a
> Project with the "111.1" literal in there as a RexLiteral
> with Decimal type.
>
> The use RelToSqlConverter to convert that RelNode back to SQL. The SQL now
> looks like
> SELECT .11 AS "num"
>
> If you attempt to parse and validate that SQL, you get "Numeric literal
> '.11' out of range" throw by
> SqlValidatorImpl.validateLiteral()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4156) ReflectiveRelMetadataProvider constructor should throw an exception (instead of assertion) when called with an empty map
[ https://issues.apache.org/jira/browse/CALCITE-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171433#comment-17171433 ] Chunwei Lei commented on CALCITE-4156: -- Makes sense. > ReflectiveRelMetadataProvider constructor should throw an exception (instead > of assertion) when called with an empty map > > > Key: CALCITE-4156 > URL: https://issues.apache.org/jira/browse/CALCITE-4156 > Project: Calcite > Issue Type: Task > Components: core >Reporter: Ruben Q L >Assignee: Ruben Q L >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > ReflectiveRelMetadataProvider's constructor verifies that it is not created > with an empty map, using an assertion. However, this is not the most reliable > way of verifying this situation, since assertions can be deactivated. In such > scenario, we could silently end up having an invalid > ReflectiveRelMetadataProvider, with no actual methods attached. > Also, since the map is private and has no getter, there is no way for a > caller module to verify this situation on its side. > For this reason, it is proposed a minor change: replace the assertion with an > IllegalArgumentException, which will work in 100% of the cases and will > always prevent constructing an invalid ReflectiveRelMetadataProvider. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4155) Simplify in expression of discreet values
[
https://issues.apache.org/jira/browse/CALCITE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171429#comment-17171429
]
Chunwei Lei commented on CALCITE-4155:
--
If any of those numbers is lack, it can not be reduced. I am not sure whether
it is easy to do that.
> Simplify in expression of discreet values
> -
>
> Key: CALCITE-4155
> URL: https://issues.apache.org/jira/browse/CALCITE-4155
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: Liya Fan
>Assignee: Liya Fan
>Priority: Major
>
> For example, given expressions like
> {{a in (1, 2, 3, 4, 5)}}
> or
> {{a = 1 or a = 2 or a = 3 or a = 4 or a = 5}},
> we can simplify it to
> {{a >= 1 and a <= 5}}
> Such simplification reduces the number of value comparisons from 5 to 2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-1581) UDTF like in hive
[
https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei reassigned CALCITE-1581:
Assignee: Chunwei Lei (was: pengzhiwei)
> UDTF like in hive
> -
>
> Key: CALCITE-1581
> URL: https://issues.apache.org/jira/browse/CALCITE-1581
> Project: Calcite
> Issue Type: New Feature
>Reporter: Xiaoyong Deng
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available, udtf
> Fix For: 1.25.0
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> Support one row in and multi-column/multi-row out(one-to-many mapping), just
> like udtf in hive.
> The query would like this:
> {code}
> select
> func(c0, c1) as (f0, f1, f2)
> from table_name;
> {code}
> c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-1581) UDTF like in hive
[
https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168550#comment-17168550
]
Chunwei Lei commented on CALCITE-1581:
--
Opened a new PR[1] with some fix-ups.
[1][https://github.com/apache/calcite/pull/2092/]
> UDTF like in hive
> -
>
> Key: CALCITE-1581
> URL: https://issues.apache.org/jira/browse/CALCITE-1581
> Project: Calcite
> Issue Type: New Feature
>Reporter: Xiaoyong Deng
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available, udtf
> Fix For: 1.25.0
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> Support one row in and multi-column/multi-row out(one-to-many mapping), just
> like udtf in hive.
> The query would like this:
> {code}
> select
> func(c0, c1) as (f0, f1, f2)
> from table_name;
> {code}
> c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-1581) UDTF like in hive
[
https://issues.apache.org/jira/browse/CALCITE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-1581:
-
Fix Version/s: 1.25.0
> UDTF like in hive
> -
>
> Key: CALCITE-1581
> URL: https://issues.apache.org/jira/browse/CALCITE-1581
> Project: Calcite
> Issue Type: New Feature
>Reporter: Xiaoyong Deng
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available, udtf
> Fix For: 1.25.0
>
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> Support one row in and multi-column/multi-row out(one-to-many mapping), just
> like udtf in hive.
> The query would like this:
> {code}
> select
> func(c0, c1) as (f0, f1, f2)
> from table_name;
> {code}
> c0 and c1 are 'table_name' columns. f0, f1 and f2 are new generated columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4081) Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter -> Planner throws validation error
[
https://issues.apache.org/jira/browse/CALCITE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168497#comment-17168497
]
Chunwei Lei edited comment on CALCITE-4081 at 7/31/20, 7:29 AM:
> Should we remove the valid check for decimal literal ? The SQL standard does
>not have limitation on the precision and scale of the Decimal type, the
>precision should be based on per-system.
I think so. It's also what the comment says[1]. Is there any other concern,
[~julianhyde]?
[1][https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L3043]
was (Author: chunwei lei):
> Should we remove the valid check for decimal literal ? The SQL standard does
>not have limitation on the precision and scale of the Decimal type, the
>precision should be based on per-system.
I think so. It's also what the comment says[1]. Is there any other concern,
[~julianhyde]?
[1][https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L3043]
> Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter ->
> Planner throws validation error
>
>
> Key: CALCITE-4081
> URL: https://issues.apache.org/jira/browse/CALCITE-4081
> Project: Calcite
> Issue Type: Bug
>Reporter: Steven Talbot
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Take the following SQL:
> SELECT CAST('.11' AS DECIMAL(38,6)) AS
> "num"
> And, in the default Calcite SQL dialect with all defaults, send it through
> Planner parse -> valiate -> rel to get a RelNode. The RelNode will have a
> Project with the "111.1" literal in there as a RexLiteral
> with Decimal type.
>
> The use RelToSqlConverter to convert that RelNode back to SQL. The SQL now
> looks like
> SELECT .11 AS "num"
>
> If you attempt to parse and validate that SQL, you get "Numeric literal
> '.11' out of range" throw by
> SqlValidatorImpl.validateLiteral()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4081) Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter -> Planner throws validation error
[
https://issues.apache.org/jira/browse/CALCITE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168494#comment-17168494
]
Chunwei Lei edited comment on CALCITE-4081 at 7/31/20, 7:29 AM:
> Shouldn't RexExecutor have thrown? .11 is
>not a valid value (or DECIMAL(38, 6) is not a valid type, or something).
It should. With my fix[1], it would throw an exception while it is caught by
RexExecutable[2].
[1] [https://github.com/apache/calcite/pull/2089]
[2][https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rex/RexExecutable.java#L88]
was (Author: chunwei lei):
> Shouldn't RexExecutor have thrown? .11 is
>not a valid value (or DECIMAL(38, 6) is not a valid type, or something).
It should. With my fix[1], it would throw an exception while it is caught by
RexExecutable[2].
[1] [https://github.com/apache/calcite/pull/2089]
[2][https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rex/RexExecutable.java#L88]
> Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter ->
> Planner throws validation error
>
>
> Key: CALCITE-4081
> URL: https://issues.apache.org/jira/browse/CALCITE-4081
> Project: Calcite
> Issue Type: Bug
>Reporter: Steven Talbot
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Take the following SQL:
> SELECT CAST('.11' AS DECIMAL(38,6)) AS
> "num"
> And, in the default Calcite SQL dialect with all defaults, send it through
> Planner parse -> valiate -> rel to get a RelNode. The RelNode will have a
> Project with the "111.1" literal in there as a RexLiteral
> with Decimal type.
>
> The use RelToSqlConverter to convert that RelNode back to SQL. The SQL now
> looks like
> SELECT .11 AS "num"
>
> If you attempt to parse and validate that SQL, you get "Numeric literal
> '.11' out of range" throw by
> SqlValidatorImpl.validateLiteral()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4081) Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter -> Planner throws validation error
[
https://issues.apache.org/jira/browse/CALCITE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168497#comment-17168497
]
Chunwei Lei commented on CALCITE-4081:
--
> Should we remove the valid check for decimal literal ? The SQL standard does
>not have limitation on the precision and scale of the Decimal type, the
>precision should be based on per-system.
I think so. It's also what the comment says[1]. Is there any other concern,
[~julianhyde]?
[1][https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L3043]
> Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter ->
> Planner throws validation error
>
>
> Key: CALCITE-4081
> URL: https://issues.apache.org/jira/browse/CALCITE-4081
> Project: Calcite
> Issue Type: Bug
>Reporter: Steven Talbot
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Take the following SQL:
> SELECT CAST('.11' AS DECIMAL(38,6)) AS
> "num"
> And, in the default Calcite SQL dialect with all defaults, send it through
> Planner parse -> valiate -> rel to get a RelNode. The RelNode will have a
> Project with the "111.1" literal in there as a RexLiteral
> with Decimal type.
>
> The use RelToSqlConverter to convert that RelNode back to SQL. The SQL now
> looks like
> SELECT .11 AS "num"
>
> If you attempt to parse and validate that SQL, you get "Numeric literal
> '.11' out of range" throw by
> SqlValidatorImpl.validateLiteral()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-4081) Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter -> Planner throws validation error
[
https://issues.apache.org/jira/browse/CALCITE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168494#comment-17168494
]
Chunwei Lei commented on CALCITE-4081:
--
> Shouldn't RexExecutor have thrown? .11 is
>not a valid value (or DECIMAL(38, 6) is not a valid type, or something).
It should. With my fix[1], it would throw an exception while it is caught by
RexExecutable[2].
[1] [https://github.com/apache/calcite/pull/2089]
[2][https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rex/RexExecutable.java#L88]
> Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter ->
> Planner throws validation error
>
>
> Key: CALCITE-4081
> URL: https://issues.apache.org/jira/browse/CALCITE-4081
> Project: Calcite
> Issue Type: Bug
>Reporter: Steven Talbot
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Take the following SQL:
> SELECT CAST('.11' AS DECIMAL(38,6)) AS
> "num"
> And, in the default Calcite SQL dialect with all defaults, send it through
> Planner parse -> valiate -> rel to get a RelNode. The RelNode will have a
> Project with the "111.1" literal in there as a RexLiteral
> with Decimal type.
>
> The use RelToSqlConverter to convert that RelNode back to SQL. The SQL now
> looks like
> SELECT .11 AS "num"
>
> If you attempt to parse and validate that SQL, you get "Numeric literal
> '.11' out of range" throw by
> SqlValidatorImpl.validateLiteral()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (CALCITE-4118) RexSimplify might remove CAST from RexNode incorrectly
[
https://issues.apache.org/jira/browse/CALCITE-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei resolved CALCITE-4118.
--
Fix Version/s: 1.25.0
Resolution: Fixed
Fixed in
[https://github.com/apache/calcite/commit/2088488ac8327b19512a76a122cae2961fc551c3].
> RexSimplify might remove CAST from RexNode incorrectly
> --
>
> Key: CALCITE-4118
> URL: https://issues.apache.org/jira/browse/CALCITE-4118
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Shuo Cheng
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.25.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code:java}
> @Test void testCastError() {
> final String sql = "select cast(cast(count(distinct empno) as
> varchar(65536)) as varbinary)\n"
> + "from emp group by deptno";
> sql(sql).ok();
> }
> {code}
> Consider the above test case, we get the following plan after SqlToRel.
> {code:java}
> LogicalProject(EXPR$0=[CAST($1):VARBINARY NOT NULL])
> LogicalAggregate(group=[{0}], agg#0=[COUNT(DISTINCT $1)])
> LogicalProject(DEPTNO=[$7], EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> As it is shown, `cast(* as varchar)` is removed (by
> RexSimplify#SimplifyCast), which is obviously wrong, because BIGINT can not
> cast to VARBINARY.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CALCITE-4118) RexSimplify might remove CAST from RexNode incorrectly
[
https://issues.apache.org/jira/browse/CALCITE-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei updated CALCITE-4118:
-
Affects Version/s: 1.24.0
> RexSimplify might remove CAST from RexNode incorrectly
> --
>
> Key: CALCITE-4118
> URL: https://issues.apache.org/jira/browse/CALCITE-4118
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0, 1.24.0
>Reporter: Shuo Cheng
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.25.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code:java}
> @Test void testCastError() {
> final String sql = "select cast(cast(count(distinct empno) as
> varchar(65536)) as varbinary)\n"
> + "from emp group by deptno";
> sql(sql).ok();
> }
> {code}
> Consider the above test case, we get the following plan after SqlToRel.
> {code:java}
> LogicalProject(EXPR$0=[CAST($1):VARBINARY NOT NULL])
> LogicalAggregate(group=[{0}], agg#0=[COUNT(DISTINCT $1)])
> LogicalProject(DEPTNO=[$7], EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> As it is shown, `cast(* as varchar)` is removed (by
> RexSimplify#SimplifyCast), which is obviously wrong, because BIGINT can not
> cast to VARBINARY.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-2772) Support varargs for user-defined functions (UDFs)
[
https://issues.apache.org/jira/browse/CALCITE-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168346#comment-17168346
]
Chunwei Lei commented on CALCITE-2772:
--
Appreciate it, [~pzw2018].
> Support varargs for user-defined functions (UDFs)
> -
>
> Key: CALCITE-2772
> URL: https://issues.apache.org/jira/browse/CALCITE-2772
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: pengzhiwei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Attachments: support_varargs_udf.patch
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Support varargs for user-defined functions as the case followed:
> {code:java}
> public class ConcatWs {
> public String eval(String sep, String... strs) {...}
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-2772) Support varargs for user-defined functions (UDFs)
[
https://issues.apache.org/jira/browse/CALCITE-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei reassigned CALCITE-2772:
Assignee: Chunwei Lei (was: pengzhiwei)
> Support varargs for user-defined functions (UDFs)
> -
>
> Key: CALCITE-2772
> URL: https://issues.apache.org/jira/browse/CALCITE-2772
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: pengzhiwei
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Attachments: support_varargs_udf.patch
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Support varargs for user-defined functions as the case followed:
> {code:java}
> public class ConcatWs {
> public String eval(String sep, String... strs) {...}
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CALCITE-4147) Rename "master" branch to "main"
[ https://issues.apache.org/jira/browse/CALCITE-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167712#comment-17167712 ] Chunwei Lei edited comment on CALCITE-4147 at 7/30/20, 8:37 AM: >From what I can see, currently the master branch is open for commits. Right? was (Author: chunwei lei): So I think the master branch is open for commits. Right? > Rename "master" branch to "main" > > > Key: CALCITE-4147 > URL: https://issues.apache.org/jira/browse/CALCITE-4147 > Project: Calcite > Issue Type: Improvement >Reporter: Michael Mior >Assignee: Michael Mior >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Settings to change: > * GIT_COMMIT in [Calcite-Snapshots Jenkins > pipeline|https://builds.apache.org/view/A-D/view/Calcite/job/Calcite-Snapshots/configure] > * Rename [Calcite-Master Jenkins > pipeline|https://builds.apache.org/view/A-D/view/Calcite/job/Calcite-Master/configure] > * Default branch on GitHub (open INFRA ticket) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-4147) Rename "master" branch to "main"
[ https://issues.apache.org/jira/browse/CALCITE-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167712#comment-17167712 ] Chunwei Lei commented on CALCITE-4147: -- So I think the master branch is open for commits. Right? > Rename "master" branch to "main" > > > Key: CALCITE-4147 > URL: https://issues.apache.org/jira/browse/CALCITE-4147 > Project: Calcite > Issue Type: Improvement >Reporter: Michael Mior >Assignee: Michael Mior >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Settings to change: > * GIT_COMMIT in [Calcite-Snapshots Jenkins > pipeline|https://builds.apache.org/view/A-D/view/Calcite/job/Calcite-Snapshots/configure] > * Rename [Calcite-Master Jenkins > pipeline|https://builds.apache.org/view/A-D/view/Calcite/job/Calcite-Master/configure] > * Default branch on GitHub (open INFRA ticket) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CALCITE-2772) Support varargs for user-defined functions (UDFs)
[
https://issues.apache.org/jira/browse/CALCITE-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167622#comment-17167622
]
Chunwei Lei commented on CALCITE-2772:
--
[~pzw2018], are you making any progress? I would like to take over if you don't
mind.
> Support varargs for user-defined functions (UDFs)
> -
>
> Key: CALCITE-2772
> URL: https://issues.apache.org/jira/browse/CALCITE-2772
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
> Attachments: support_varargs_udf.patch
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Support varargs for user-defined functions as the case followed:
> {code:java}
> public class ConcatWs {
> public String eval(String sep, String... strs) {...}
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CALCITE-3923) Refactor how planner rules are parameterized
[
https://issues.apache.org/jira/browse/CALCITE-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167045#comment-17167045
]
Chunwei Lei commented on CALCITE-3923:
--
A minor fix of Site:
[https://github.com/apache/calcite/commit/b047cf5335442f9b5175cba0bb8cf33878539c52].
> Refactor how planner rules are parameterized
>
>
> Key: CALCITE-3923
> URL: https://issues.apache.org/jira/browse/CALCITE-3923
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: Julian Hyde
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.25.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> People often want different variants of planner rules. An example is
> {{FilterJoinRule}}, which has a 'boolean smart’ parameter, a predicate (which
> returns whether to pull up filter conditions), operands (which determine the
> precise sub-classes of {{RelNode}} that the rule should match) and a
> {{RelBuilderFactory}} (which controls the type of {{RelNode}} created by this
> rule).
> Suppose you have an instance of {{FilterJoinRule}} and you want to change
> {{smart}} from true to false. The {{smart}} parameter is immutable (good!)
> but you can’t easily create a clone of the rule because you don’t know the
> values of the other parameters. Your instance might even be (unbeknownst to
> you) a sub-class with extra parameters and a private constructor.
> So, my proposal is to put all of the config information of a {{RelOptRule}}
> into a single {{config}} parameter that contains all relevant properties.
> Each sub-class of {{RelOptRule}} would have one constructor with just a
> ‘config’ parameter. Each config knows which sub-class of {{RelOptRule}} to
> create. Therefore it is easy to copy a config, change one or more properties,
> and create a new rule instance.
> Adding a property to a rule’s config does not require us to add or deprecate
> any constructors.
> The operands are part of the config, so if you have a rule that matches a
> {{EnumerableFilter}} on an {{EnumerableJoin}} and you want to make it match
> an {{EnumerableFilter}} on an {{EnumerableNestedLoopJoin}}, you can easily
> create one with one changed operand.
> The config is immutable and self-describing, so we can use it to
> automatically generate a unique description for each rule instance.
> (See the email thread [[DISCUSS] Refactor how planner rules are
> parameterized|https://lists.apache.org/thread.html/rfdf6f9b7821988bdd92b0377e3d293443a6376f4773c4c658c891cf9%40%3Cdev.calcite.apache.org%3E].)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-4081) Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter -> Planner throws validation error
[
https://issues.apache.org/jira/browse/CALCITE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei reassigned CALCITE-4081:
Assignee: Chunwei Lei
> Round-tripping a DECIMAL literal through Planner -> RelToSqlConverter ->
> Planner throws validation error
>
>
> Key: CALCITE-4081
> URL: https://issues.apache.org/jira/browse/CALCITE-4081
> Project: Calcite
> Issue Type: Bug
>Reporter: Steven Talbot
>Assignee: Chunwei Lei
>Priority: Major
>
> Take the following SQL:
> SELECT CAST('.11' AS DECIMAL(38,6)) AS
> "num"
> And, in the default Calcite SQL dialect with all defaults, send it through
> Planner parse -> valiate -> rel to get a RelNode. The RelNode will have a
> Project with the "111.1" literal in there as a RexLiteral
> with Decimal type.
>
> The use RelToSqlConverter to convert that RelNode back to SQL. The SQL now
> looks like
> SELECT .11 AS "num"
>
> If you attempt to parse and validate that SQL, you get "Numeric literal
> '.11' out of range" throw by
> SqlValidatorImpl.validateLiteral()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (CALCITE-4118) RexSimplify might remove CAST from RexNode incorrectly
[
https://issues.apache.org/jira/browse/CALCITE-4118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chunwei Lei reassigned CALCITE-4118:
Assignee: Chunwei Lei
> RexSimplify might remove CAST from RexNode incorrectly
> --
>
> Key: CALCITE-4118
> URL: https://issues.apache.org/jira/browse/CALCITE-4118
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Shuo Cheng
>Assignee: Chunwei Lei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> @Test void testCastError() {
> final String sql = "select cast(cast(count(distinct empno) as
> varchar(65536)) as varbinary)\n"
> + "from emp group by deptno";
> sql(sql).ok();
> }
> {code}
> Consider the above test case, we get the following plan after SqlToRel.
> {code:java}
> LogicalProject(EXPR$0=[CAST($1):VARBINARY NOT NULL])
> LogicalAggregate(group=[{0}], agg#0=[COUNT(DISTINCT $1)])
> LogicalProject(DEPTNO=[$7], EMPNO=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> As it is shown, `cast(* as varchar)` is removed (by
> RexSimplify#SimplifyCast), which is obviously wrong, because BIGINT can not
> cast to VARBINARY.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
