[jira] [Created] (CALCITE-6388) PsTableFunction throws NumberFormatException when the 'user' column has spaces
Alessandro Solimando created CALCITE-6388:
-
Summary: PsTableFunction throws NumberFormatException when the
'user' column has spaces
Key: CALCITE-6388
URL: https://issues.apache.org/jira/browse/CALCITE-6388
Project: Calcite
Issue Type: Bug
Components: os-adapter
Affects Versions: 1.36.0
Reporter: Alessandro Solimando
Assignee: Alessandro Solimando
Fix For: 1.37.0
Line parsing splits on spaces
([PsTableFunction.java#L77|https://github.com/apache/calcite/blob/aa8d81bf1ff39e3632aeb856fc4cc247ce9727e5/plus/src/main/java/org/apache/calcite/adapter/os/PsTableFunction.java#L77]),
which breaks whenever the "user" contains at least a space.
An example output on my laptop is as follows:
{noformat}
$ ps ax -o
ppid=,pid=,pgid=,tpgid=,stat=,user=,pcpu=,pmem=,vsz=,rss=,tty=,start=,time=,uid=,ruid=,sess=,comm=
| grep startup
1 6728 6728 0 S startup user 0.0 0.0 410266096 2528 ??
11Apr24 0:16.97 501 501 0 /usr/sbin/distnoted
1 6729 6729 0 SN startup user 0.0 0.1 410332256 17616 ??
11Apr24 0:42.41 501 501 0
/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdbulkimport
1 6750 6750 0 S startup user 0.0 0.0 410376144 13344 ??
11Apr24 0:40.39 501 501 0 /usr/libexec/lsd
1 10148 10148 0 S startup user 0.0 0.0 410354816 5488 ??
8:26PM 0:00.31 501 501 0 /usr/libexec/containermanagerd
1 95313 95313 0 S startup user 0.0 0.0 410357344 6576 ??
Fri05PM 0:00.32 501 501 0 /usr/libexec/trustd{noformat}
When running the test it fails with the following stack trace:
{code:java}
Error while executing SQL "select distinct `user` from ps": while parsing value
[user] of field [pcpu] in line [ 1 6728 6728 0 S startup user
0.0 0.0 410266096 2528 ?? 11Apr24 0:16.95 501 501 0
/usr/sbin/distnoted]
java.sql.SQLException: Error while executing SQL "select distinct `user` from
ps": while parsing value [user] of field [pcpu] in line [ 1 6728 6728 0
S startup user 0.0 0.0 410266096 2528 ?? 11Apr24 0:16.95
501 501 0 /usr/sbin/distnoted]
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:164)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:228)
at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:566)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1495)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1434)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1493)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1483)
at
org.apache.calcite.adapter.os.OsAdapterTest.testPsDistinct(OsAdapterTest.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
[jira] [Commented] (CALCITE-6387) Calcite build while compiliation with jdk17+
[
https://issues.apache.org/jira/browse/CALCITE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841522#comment-17841522
]
Alessandro Solimando commented on CALCITE-6387:
---
Same for me, it runs fine with 11.0.21-zulu but breaks with 17.0.9-zulu, but
the submitted PR fixes the issue, I approved it
> Calcite build while compiliation with jdk17+
>
>
> Key: CALCITE-6387
> URL: https://issues.apache.org/jira/browse/CALCITE-6387
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.36.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: pull-request-available
>
> The issue appears with newly added Arrow adapter which requires
> {noformat}
> --add-opens=java.base/java.nio=ALL-UNNAMED
> {noformat}
> could be fixed with adding
> {noformat}
> plugins.withType {
> tasks {
> configureEach {
> jvmArgs("-XX:+IgnoreUnrecognizedVMOptions")
> jvmArgs("--add-opens=java.base/java.nio=ALL-UNNAMED")
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6340) RelBuilder always creates Project with Convention.NONE during aggregate_
[
https://issues.apache.org/jira/browse/CALCITE-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-6340:
--
Fix Version/s: 1.37.0
> RelBuilder always creates Project with Convention.NONE during aggregate_
>
>
> Key: CALCITE-6340
> URL: https://issues.apache.org/jira/browse/CALCITE-6340
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Adam Kennedy
>Assignee: James Duong
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> In the RelBuilder method aggregate_, when (config.pruneInputOfAggregate() &&
> r instanceof Project) line 2443 the Project will be rewritten to remove
> unused columns.
> When this happens, the new Project will be created with the following line
> {{{}2487: r =
> {}}}{{{}2488: project.copy(cluster.traitSet(), project.getInput(),
> newProjects,{}}}
> {{2489: builder.build());}}
>
> The use of cluster.traitSet() returns emptyTraitSet which is always going to
> use Convention.NONE regardless of the Rebuilder's ProjectFactory.
> In the case of a query plan using a non-Logical convention FOO, with
> FooProject nodes that require the FOO convention, RelBuilder will normally
> happily produce FooProject nodes with FOO convention, allowing many CoreRules
> to be easily reused for custom Conventions.
> However, while RelBuilder will produce FooProject with FOO convention in the
> majority of cases, for the one specific case of column pruning a Project
> input to an aggregate, it will instead product a FooProject with NONE
> convention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5289) Assertion failure in MultiJoinOptimizeBushyRule
[
https://issues.apache.org/jira/browse/CALCITE-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5289:
--
Fix Version/s: 1.37.0
> Assertion failure in MultiJoinOptimizeBushyRule
> ---
>
> Key: CALCITE-5289
> URL: https://issues.apache.org/jira/browse/CALCITE-5289
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.32.0, 1.35.0
>Reporter: Mihai Budiu
>Assignee: Mou Wu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> The reproduction is easy: just modify the following test case from
> PlannerTest.java:
>
> — a/core/src/test/java/org/apache/calcite/tools/PlannerTest.java
> +++ b/core/src/test/java/org/apache/calcite/tools/PlannerTest.java
> {}@@ -1005,7 +1005,7 @@ private void checkJoinNWay(int n) throws
> Exception {{}
> {{private void checkHeuristic(String sql, String expected) throws Exception
> {}}
> {{ Planner planner = getPlanner(null,}}
> {{- Programs.heuristicJoinOrder(Programs.RULE_SET, false, 0));}}
> {{+ Programs.heuristicJoinOrder(Programs.RULE_SET, true, 0));}}
> {{ SqlNode parse = planner.parse(sql);}}
> {{ SqlNode validate = planner.validate(parse);}}
> {{ RelNode convert = planner.rel(validate).rel;}}
>
> Then the test fails with the exception shown below. This happens with the
> latest version of calcite, the main branch.
> It looks like the rule does not account for the fact that outer joins can
> produce results with a different nullability than the input relations.
> The exception can be triggered even for very simple outer join queries, e.g.:
> SELECT T1.COL3 FROM T AS T1 LEFT JOIN T AS T2 ON T1.COL1 = T2.COL5
>
> The only workaround I found is to make sure this rule is never applied when a
> query contains an outer join.
>
> Here is the Java stack trace:
> {{java.lang.RuntimeException: Error while applying rule
> MultiJoinOptimizeBushyRule, args
> [rel#44:MultiJoin.NONE.[](input#0=RelSubset#42,input#1=RelSubset#43,joinFilter=true,isFullOuterJoin=false,joinTypes=[RIGHT,
> INNER],outerJoinConditions=[=($0, $10), NULL],projFields=[ALL, ALL])]}}
>
> {{ at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)}}
> {{ at
> org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)}}
> {{ at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:523)}}
> {{ at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:318)}}
> {{ at
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:337)}}
> {{ at
> org.apache.calcite.tools.Programs.lambda$heuristicJoinOrder$1(Programs.java:223)}}
> {{ at
> org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:373)}}
> {{ at
> org.apache.calcite.tools.PlannerTest.checkHeuristic(PlannerTest.java:1014)}}
> {{ at
> org.apache.calcite.tools.PlannerTest.testHeuristicRightJoin(PlannerTest.java:1003)}}
> {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{ at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)}}
> {{ at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)}}
> {{ at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)}}
> {{ at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)}}
> {{ at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)}}
> {{ at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)}}
> {{ at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)}}
> {{ at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)}}
> {{ at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)}}
> {{ at
>
[jira] [Assigned] (CALCITE-5289) Assertion failure in MultiJoinOptimizeBushyRule
[
https://issues.apache.org/jira/browse/CALCITE-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-5289:
-
Assignee: Mihai Budiu (was: Mou Wu)
> Assertion failure in MultiJoinOptimizeBushyRule
> ---
>
> Key: CALCITE-5289
> URL: https://issues.apache.org/jira/browse/CALCITE-5289
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.32.0, 1.35.0
>Reporter: Mihai Budiu
>Assignee: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> The reproduction is easy: just modify the following test case from
> PlannerTest.java:
>
> — a/core/src/test/java/org/apache/calcite/tools/PlannerTest.java
> +++ b/core/src/test/java/org/apache/calcite/tools/PlannerTest.java
> {}@@ -1005,7 +1005,7 @@ private void checkJoinNWay(int n) throws
> Exception {{}
> {{private void checkHeuristic(String sql, String expected) throws Exception
> {}}
> {{ Planner planner = getPlanner(null,}}
> {{- Programs.heuristicJoinOrder(Programs.RULE_SET, false, 0));}}
> {{+ Programs.heuristicJoinOrder(Programs.RULE_SET, true, 0));}}
> {{ SqlNode parse = planner.parse(sql);}}
> {{ SqlNode validate = planner.validate(parse);}}
> {{ RelNode convert = planner.rel(validate).rel;}}
>
> Then the test fails with the exception shown below. This happens with the
> latest version of calcite, the main branch.
> It looks like the rule does not account for the fact that outer joins can
> produce results with a different nullability than the input relations.
> The exception can be triggered even for very simple outer join queries, e.g.:
> SELECT T1.COL3 FROM T AS T1 LEFT JOIN T AS T2 ON T1.COL1 = T2.COL5
>
> The only workaround I found is to make sure this rule is never applied when a
> query contains an outer join.
>
> Here is the Java stack trace:
> {{java.lang.RuntimeException: Error while applying rule
> MultiJoinOptimizeBushyRule, args
> [rel#44:MultiJoin.NONE.[](input#0=RelSubset#42,input#1=RelSubset#43,joinFilter=true,isFullOuterJoin=false,joinTypes=[RIGHT,
> INNER],outerJoinConditions=[=($0, $10), NULL],projFields=[ALL, ALL])]}}
>
> {{ at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)}}
> {{ at
> org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)}}
> {{ at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:523)}}
> {{ at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:318)}}
> {{ at
> org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:337)}}
> {{ at
> org.apache.calcite.tools.Programs.lambda$heuristicJoinOrder$1(Programs.java:223)}}
> {{ at
> org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:373)}}
> {{ at
> org.apache.calcite.tools.PlannerTest.checkHeuristic(PlannerTest.java:1014)}}
> {{ at
> org.apache.calcite.tools.PlannerTest.testHeuristicRightJoin(PlannerTest.java:1003)}}
> {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{ at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)}}
> {{ at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)}}
> {{ at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)}}
> {{ at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)}}
> {{ at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)}}
> {{ at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)}}
> {{ at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)}}
> {{ at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)}}
> {{ at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)}}
> {{ at
>
[jira] [Commented] (CALCITE-6365) Support for RETURNING clause of JSON_QUERY
[
https://issues.apache.org/jira/browse/CALCITE-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837223#comment-17837223
]
Alessandro Solimando commented on CALCITE-6365:
---
It would be nice to add at least a simple example to the description
> Support for RETURNING clause of JSON_QUERY
> --
>
> Key: CALCITE-6365
> URL: https://issues.apache.org/jira/browse/CALCITE-6365
> Project: Calcite
> Issue Type: New Feature
>Reporter: Dawid Wysakowicz
>Priority: Major
>
> SQL standard says {{JSON_QUERY}} should support {{RETURNING}} clause similar
> to {{JSON_VALUE}}. Calcite supports the clause for JSON_VALUE already, but
> not for the JSON_QUERY.
> {code}
> ::=
> JSON_QUERY
>
> [ ]
> [ WRAPPER ]
> [ QUOTES [ ON SCALAR STRING ] ]
> [ ON EMPTY ]
> [ ON ERROR ]
>
> ::=
> RETURNING
> [ FORMAT ]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6362) JSON_OBJECT. Internal representation of DATE values leaks into resulting JSON object
[
https://issues.apache.org/jira/browse/CALCITE-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836420#comment-17836420
]
Alessandro Solimando commented on CALCITE-6362:
---
[~mzhuravkov] if you consider the behavior acceptable (days without cast and a
date with the cast), please consider resolving the ticket accordingly.
> JSON_OBJECT. Internal representation of DATE values leaks into resulting JSON
> object
>
>
> Key: CALCITE-6362
> URL: https://issues.apache.org/jira/browse/CALCITE-6362
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Maksim Zhuravkov
>Priority: Minor
>
> When a CAST function is used inside a JSON_OBJECT, then resulting JSON
> contains a number of days instead
> {code}
> SELECT JSON_OBJECT('a': CAST('2010-01-01' AS DATE)) as c1
> {"a":14610}
> {code}
> I expected to get a date in a string form as many databases do in such case:
> Bigquery:
> {code}
> SELECT JSON_OBJECT('a', CAST('2010-01-10' AS DATE))
> {"a":"2010-01-10"}
> {code}
> PostgreSQL
> {code}
> SELECT json_build_object('a', '2010-01-01'::DATE)
> {"a":"2010-01-01"}
> {code}
> MySql
> {code}
> SELECT json_object('a', DATE '2010-01-01')
> {"a": "2010-01-01"}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6362) JSON_OBJECT. Internal representation DATE values leaks into resulting JSON object
[
https://issues.apache.org/jira/browse/CALCITE-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836181#comment-17836181
]
Alessandro Solimando commented on CALCITE-6362:
---
I agree that getting a string with the date would be the expected output,
instead of what we are returning now.
> JSON_OBJECT. Internal representation DATE values leaks into resulting JSON
> object
> -
>
> Key: CALCITE-6362
> URL: https://issues.apache.org/jira/browse/CALCITE-6362
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Maksim Zhuravkov
>Priority: Minor
>
> When a CAST function is used inside a JSON_OBJECT, then resulting JSON
> contains a number of days instead
> {code}
> SELECT JSON_OBJECT('a': CAST('2010-01-01' AS DATE)) as c1
> {"a":14610}
> {code}
> I expected to get a date in a string form as many databases do in such case:
> Bigquery:
> {code}
> SELECT JSON_OBJECT('a', CAST('2010-01-10' AS DATE))
> {"a":"2010-01-10"}
> {code}
> PostgreSQL
> {code}
> SELECT json_build_object('a', '2010-01-01'::DATE)
> {"a":"2010-01-01"}
> {code}
> MySql
> {code}
> SELECT json_object('a', DATE '2010-01-01')
> {"a": "2010-01-01"}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6307) Sonar analysis in CI fails with 'java.lang.OutOfMemoryError: Java heap space'
Alessandro Solimando created CALCITE-6307:
-
Summary: Sonar analysis in CI fails with
'java.lang.OutOfMemoryError: Java heap space'
Key: CALCITE-6307
URL: https://issues.apache.org/jira/browse/CALCITE-6307
Project: Calcite
Issue Type: Bug
Components: tests
Affects Versions: 1.36.0
Reporter: Alessandro Solimando
Recently there has been several Sonar failures due to OOM, one example is the
following:
[https://ci-builds.apache.org/blue/organizations/jenkins/Calcite%2FCalcite-sonar/detail/PR-3687/9/pipeline]
In what follows a relevant portion of the execution logs:
{code:java}
Expiring Daemon because JVM heap space is exhausted
> Task :sonar FAILED
Build calcite FAILURE reason:
Execution failed for task ':sonar':
java.lang.OutOfMemoryError: Java heap space
at B.A.A.A.A.D.B(na:26)
at B.A.A.A.A.D.A(na:163)
at B.A.A.A.A.D.B(na:1209)
at B.A.A.A.A.D.B(na:2138)
at B.A.A.A.A.D.B(na:2138)
at B.A.A.A.A.D.B(na:2138)
at B.A.A.A.A.D.B(na:2138)
at B.A.A.A.A.D.B(na:2138)
at B.A.A.A.A.D.B(na:2221)
at B.A.A.A.A.D$_C.A(na:2102)
at com.sonarsource.A.A.U.A(na:3338)
at com.sonarsource.A.A.B.F.A(na:501)
at com.sonarsource.A.A.U.E(na:1630)
at com.sonarsource.A.A.Z.A(na:2867)
at com.sonarsource.A.A.Z.A(na:2409)
at com.sonarsource.A.A.Z.A(na:2223)
at com.sonarsource.A.A.Z.A(na:920)
at com.sonarsource.A.A.Z.E(na:1730)
at com.sonarsource.A.A.Z.A(na:242)
at com.sonarsource.A.A.Z.A(na:671)
at com.sonarsource.A.A.Z.A(na:2244)
at com.sonarsource.A.F.executeChecksOnFunction(na:896)
at com.sonarsource.A.F.executeChecks(na:1668)
at com.sonarsource.A.F.executeSensor(na:1339)
at com.sonarsource.A.F.execute(na:2143)
at
org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:62)
at
org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:75)
at
org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:51)
at
org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:64)
at
org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:123)
at
org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:109)
at
org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:192)
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sonar'.
> Java heap space {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6305) Increase coverage of unit tests for the Arrow adapter
[ https://issues.apache.org/jira/browse/CALCITE-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando updated CALCITE-6305: -- Summary: Increase coverage of unit tests for the Arrow adapter (was: Increase coverage of unit tests for arrow adapter) > Increase coverage of unit tests for the Arrow adapter > - > > Key: CALCITE-6305 > URL: https://issues.apache.org/jira/browse/CALCITE-6305 > Project: Calcite > Issue Type: Sub-task >Reporter: hongyu guo >Priority: Major > > Add more unit tests for arrow adapter. > for example > * casts, including lossy casts > * complex filter condition > * complex SQL > * and more... > Note: Calcite's support for the arrow feather format is not complete, so > before adding the testing process, we may need to implement new features > first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CALCITE-6265) Type coercion is failing for numeric values in prepared statements
[
https://issues.apache.org/jira/browse/CALCITE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-6265:
-
Assignee: Tim Nieradzik (was: Alessandro Solimando)
> Type coercion is failing for numeric values in prepared statements
> --
>
> Key: CALCITE-6265
> URL: https://issues.apache.org/jira/browse/CALCITE-6265
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Tim Nieradzik
>Assignee: Tim Nieradzik
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> Given a column of type {{{}INT{}}}. When providing a {{short}} value as a
> placeholder in a prepared statement, a {{ClassCastException}} is thrown.
> h2. Test case
> {{final String sql =}}
> {{ "select \"empid\" from \"hr\".\"emps\" where \"empid\" in (?, ?)";}}{{
> CalciteAssert.hr()}}
> {{ .query(sql)}}
> {{ .consumesPreparedStatement(p -> {}}
> {{ p.setShort(1, (short) 100);}}
> {{ p.setShort(2, (short) 110);}}
> {{ })}}
> {{ .returnsUnordered("empid=100", "empid=110");}}
> h2. Stack trace
> {{java.lang.ClassCastException: class java.lang.Short cannot be cast to class
> java.lang.Integer (java.lang.Short and java.lang.Integer are in module
> java.base of loader 'bootstrap')}}
> {{ at Baz$1$1.moveNext(Unknown Source)}}
> {{ at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:679)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CALCITE-6265) Cannot provide different numeric type as placeholder
[
https://issues.apache.org/jira/browse/CALCITE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-6265:
-
Assignee: Alessandro Solimando (was: Tim Nieradzik)
> Cannot provide different numeric type as placeholder
>
>
> Key: CALCITE-6265
> URL: https://issues.apache.org/jira/browse/CALCITE-6265
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Tim Nieradzik
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> The _empid_ column is of type {_}INT{_}. When providing a _short_ value as a
> placeholder, a _ClassCastException_ is thrown.
> *Test case:*
> {{ final String sql =}}
> {{ "select \"empid\" from \"hr\".\"emps\" where \"empid\" in (?,
> ?)";}}{{ CalciteAssert.hr()}}
> {{ .query(sql)}}
> {{ .consumesPreparedStatement(p -> {}}
> {{ p.setShort(1, (short) 100);}}
> {{ p.setShort(2, (short) 110);}}
> {{ })}}
> {{ .returnsUnordered("empid=100", "empid=110");}}
> *Stack trace:*
> {{java.lang.ClassCastException: class java.lang.Short cannot be cast to class
> java.lang.Integer (java.lang.Short and java.lang.Integer are in module
> java.base of loader 'bootstrap')}}
> {{ at Baz$1$1.moveNext(Unknown Source)}}
> {{ at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:679)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CALCITE-6265) Cannot provide different numeric type as placeholder
[
https://issues.apache.org/jira/browse/CALCITE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-6265:
-
Assignee: Tim Nieradzik
> Cannot provide different numeric type as placeholder
>
>
> Key: CALCITE-6265
> URL: https://issues.apache.org/jira/browse/CALCITE-6265
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Tim Nieradzik
>Assignee: Tim Nieradzik
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> The _empid_ column is of type {_}INT{_}. When providing a _short_ value as a
> placeholder, a _ClassCastException_ is thrown.
> *Test case:*
> {{ final String sql =}}
> {{ "select \"empid\" from \"hr\".\"emps\" where \"empid\" in (?,
> ?)";}}{{ CalciteAssert.hr()}}
> {{ .query(sql)}}
> {{ .consumesPreparedStatement(p -> {}}
> {{ p.setShort(1, (short) 100);}}
> {{ p.setShort(2, (short) 110);}}
> {{ })}}
> {{ .returnsUnordered("empid=100", "empid=110");}}
> *Stack trace:*
> {{java.lang.ClassCastException: class java.lang.Short cannot be cast to class
> java.lang.Integer (java.lang.Short and java.lang.Integer are in module
> java.base of loader 'bootstrap')}}
> {{ at Baz$1$1.moveNext(Unknown Source)}}
> {{ at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:679)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6265) Cannot provide different numeric type as placeholder
[
https://issues.apache.org/jira/browse/CALCITE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-6265:
--
Fix Version/s: 1.37.0
> Cannot provide different numeric type as placeholder
>
>
> Key: CALCITE-6265
> URL: https://issues.apache.org/jira/browse/CALCITE-6265
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Tim Nieradzik
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> The _empid_ column is of type {_}INT{_}. When providing a _short_ value as a
> placeholder, a _ClassCastException_ is thrown.
> *Test case:*
> {{ final String sql =}}
> {{ "select \"empid\" from \"hr\".\"emps\" where \"empid\" in (?,
> ?)";}}{{ CalciteAssert.hr()}}
> {{ .query(sql)}}
> {{ .consumesPreparedStatement(p -> {}}
> {{ p.setShort(1, (short) 100);}}
> {{ p.setShort(2, (short) 110);}}
> {{ })}}
> {{ .returnsUnordered("empid=100", "empid=110");}}
> *Stack trace:*
> {{java.lang.ClassCastException: class java.lang.Short cannot be cast to class
> java.lang.Integer (java.lang.Short and java.lang.Integer are in module
> java.base of loader 'bootstrap')}}
> {{ at Baz$1$1.moveNext(Unknown Source)}}
> {{ at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:679)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6301) Extend ‘Must-filter’ columns to support a conditional bypass list
[ https://issues.apache.org/jira/browse/CALCITE-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824124#comment-17824124 ] Alessandro Solimando commented on CALCITE-6301: --- Thanks [~oliverlee], it's clear now with the updated example. One question I still have is under what circumstances . I have seen similar asks in the past for "must-filter" to force users to write queries that could leverage indexes or partitioning, but I am having a hard-time seeing how the "bypass-list" would come to rescue. Could you sketch a use-case where the "bypass-list" would be helpful? Just to be clear, I have nothing against this improvement, just curious to understand the rationale behind it. > Extend ‘Must-filter’ columns to support a conditional bypass list > - > > Key: CALCITE-6301 > URL: https://issues.apache.org/jira/browse/CALCITE-6301 > Project: Calcite > Issue Type: Improvement >Reporter: Oliver Lee >Assignee: Oliver Lee >Priority: Major > > In CALCITE-6219 we introduced SemanticTable, where tables that implement this > interface can define fields to be ‘must-filter’, and a query without those > filters in any of its WHERE or HAVING clauses, it will throw a validation > error. > > I would like to extend this functionality to support a by-pass list of fields > such that if any field from this secondary list is present in a WHERE / > HAVING clause, then the must-filter fields can be ignored and will not raise > an exception if not filtered on. > > Ex. > > EMP table specifies the following: > Must-filter-fields: [EMPNO, DEPTNO] > Bypass-fields: [ENAME, SALARY] > > > SELECT * FROM EMP WHERE EMPNO = 1 and DEPTNO = 2 -> No error > SELECT * FROM EMP WHERE EMPNO = 1 -> Error > SELECT * FROM EMP WHERE EMPNO = 1 and ENAME = ’name’ -> No error > SELECT * FROM EMP WHERE ENAME = ’name’ -> No error > SELECT * FROM EMP WHERE SALARY > 10 -> No error > > > > Again, special considerations are for handling > > * Joins > * CTEs > * Subqueries > > > And a similar exhaustive suite of tests like the one for CALCITE-6219 should > be employed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824121#comment-17824121 ] Alessandro Solimando commented on CALCITE-2040: --- [~hongyuguo] thanks for creating the tickets and adding the missing details. One last ask is to create instead a new umbrella ticket and move the tickets under it, as these tickets are meant to be addressed after completing CALCITE-2040. The new umbrella ticket should be marked as "depending/blocked" on CALCITE-2040. I have marked the ticket's fix version as 1.37.0, as the release is incumbent but I think we can get this in with some more effort. > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando updated CALCITE-2040: -- Fix Version/s: 1.37.0 > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6293) Support OR condition in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823902#comment-17823902
]
Alessandro Solimando commented on CALCITE-6293:
---
[~hongyuguo] is revamping the [PR|https://github.com/apache/calcite/pull/3666]
for the arrow adapter (CALCITE-2040) and I am helping reviewing it (we don't
work for the same organization AFAIK).
In the ticket there seems to be consensus to get the adapter merged even if
some functionalities are not there yet (decision with which I agree,
considering the size of the PR and that it's gone stale two times already).
During the review process I suggested to file a ticket for each broken test we
write and for any important missing functionality (e.g., supporting more data
types).
At the moment, the code throws for disjunctive filters:
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:69)
at org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:47)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:99)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:89)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
{code}
What you suggest would work if the logical filters weren't translated to
ArrowFilter by the ArrowFilterRule if a disjunction is detected in the filter's
condition, which can be easily implemented in a second step.
I agree that at least the stacktrace of the current error, or discrepancy
w.r.t. the expected result, should be provided in the ticket description.
[~hongyuguo], could you please improve accordingly the tickets you have filed
already?
> Support OR condition in arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> for example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6301) Extend ‘Must-filter’ columns to support a conditional bypass list
[ https://issues.apache.org/jira/browse/CALCITE-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823890#comment-17823890 ] Alessandro Solimando commented on CALCITE-6301: --- Isn't this equivalent to say that "EMPNAME" is an alternative to the "Must-filter-fields"? In that case I think it's simpler to extend the syntax to provide multiple "Must-filter-fields" lists, something like "[EMPNO, DEPTNO],[ENAME]" (a list of lists, conceptually, where each list is an independent alternative). >From your example it's not clear if multiple by-pass fields values would >require to be there at the same time (very much like "Must-filter-fields"), or >if any of them alone would do. For instance, for Bypass-fields: [ENAME, EMPNO], would the following queries be valid or not? SELECT * FROM EMP WHERE ENAME = ’name’ SELECT * FROM EMP WHERE EMPNO = 1 In any case, the multiple "Must-filter-fields" syntax would allow to express both pretty naturally. You can even probably re-use the same machinery you coded for "Must-filter-fields" more easily. WDYT? > Extend ‘Must-filter’ columns to support a conditional bypass list > - > > Key: CALCITE-6301 > URL: https://issues.apache.org/jira/browse/CALCITE-6301 > Project: Calcite > Issue Type: Improvement >Reporter: Oliver Lee >Assignee: Oliver Lee >Priority: Major > > In [CALCITE-6219] we introduced SemanticTable, where tables that implement > this interface can define fields to be ‘must-filter’, and a query without > those filters in any of its WHERE or HAVING clauses, it will throw a > validation error. > > I would like to extend this functionality to support a by-pass list of fields > such that if any field from this secondary list is present in a WHERE / > HAVING clause, then the must-filter fields can be ignored and will not raise > an exception if not filtered on. > > Ex. > > EMP table specifies the following: > Must-filter-fields: [EMPNO, DEPTNO] > Bypass-fields: [ENAME] > > > SELECT * FROM EMP WHERE EMPNO = 1 and DEPTNO = 2 -> No error > SELECT * FROM EMP WHERE EMPNO = 1 -> Error > SELECT * FROM EMP WHERE EMPNO = 1 and ENAME = ’name’ -> No error > SELECT * FROM EMP WHERE ENAME = ’name’ -> No error > > > > Again, special considerations are for handling > > * Joins > * CTEs > * Subqueries > > > And a similar exhaustive suite of tests like the one for [CALCITE-6219] > should be employed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6301) Extend ‘Must-filter’ columns to support a conditional bypass list
[ https://issues.apache.org/jira/browse/CALCITE-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823890#comment-17823890 ] Alessandro Solimando edited comment on CALCITE-6301 at 3/6/24 7:42 AM: --- Isn't this equivalent to say that "ENAME" is an alternative to the "Must-filter-fields"? In that case I think it's simpler to extend the syntax to provide multiple "Must-filter-fields" lists, something like "[EMPNO, DEPTNO],[ENAME]" (a list of lists, conceptually, where each list is an independent alternative). >From your example it's not clear if multiple by-pass fields values would >require to be there at the same time (very much like "Must-filter-fields"), or >if any of them alone would do. For instance, for Bypass-fields: [ENAME, EMPNO], would the following queries be valid or not? SELECT * FROM EMP WHERE ENAME = ’name’ SELECT * FROM EMP WHERE EMPNO = 1 In any case, the multiple "Must-filter-fields" syntax would allow to express both pretty naturally. You can even probably re-use the same machinery you coded for "Must-filter-fields" more easily. WDYT? was (Author: asolimando): Isn't this equivalent to say that "EMPNAME" is an alternative to the "Must-filter-fields"? In that case I think it's simpler to extend the syntax to provide multiple "Must-filter-fields" lists, something like "[EMPNO, DEPTNO],[ENAME]" (a list of lists, conceptually, where each list is an independent alternative). >From your example it's not clear if multiple by-pass fields values would >require to be there at the same time (very much like "Must-filter-fields"), or >if any of them alone would do. For instance, for Bypass-fields: [ENAME, EMPNO], would the following queries be valid or not? SELECT * FROM EMP WHERE ENAME = ’name’ SELECT * FROM EMP WHERE EMPNO = 1 In any case, the multiple "Must-filter-fields" syntax would allow to express both pretty naturally. You can even probably re-use the same machinery you coded for "Must-filter-fields" more easily. WDYT? > Extend ‘Must-filter’ columns to support a conditional bypass list > - > > Key: CALCITE-6301 > URL: https://issues.apache.org/jira/browse/CALCITE-6301 > Project: Calcite > Issue Type: Improvement >Reporter: Oliver Lee >Assignee: Oliver Lee >Priority: Major > > In [CALCITE-6219] we introduced SemanticTable, where tables that implement > this interface can define fields to be ‘must-filter’, and a query without > those filters in any of its WHERE or HAVING clauses, it will throw a > validation error. > > I would like to extend this functionality to support a by-pass list of fields > such that if any field from this secondary list is present in a WHERE / > HAVING clause, then the must-filter fields can be ignored and will not raise > an exception if not filtered on. > > Ex. > > EMP table specifies the following: > Must-filter-fields: [EMPNO, DEPTNO] > Bypass-fields: [ENAME] > > > SELECT * FROM EMP WHERE EMPNO = 1 and DEPTNO = 2 -> No error > SELECT * FROM EMP WHERE EMPNO = 1 -> Error > SELECT * FROM EMP WHERE EMPNO = 1 and ENAME = ’name’ -> No error > SELECT * FROM EMP WHERE ENAME = ’name’ -> No error > > > > Again, special considerations are for handling > > * Joins > * CTEs > * Subqueries > > > And a similar exhaustive suite of tests like the one for [CALCITE-6219] > should be employed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6280) Jetty version number leaked by Avatica http server
[
https://issues.apache.org/jira/browse/CALCITE-6280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-6280.
---
Resolution: Fixed
> Jetty version number leaked by Avatica http server
> --
>
> Key: CALCITE-6280
> URL: https://issues.apache.org/jira/browse/CALCITE-6280
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Affects Versions: 1.24.0
>Reporter: Vaibhav Joshi
>Assignee: Vaibhav Joshi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.25.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Unauthorised access to HTTP server using curl returns the Jerry server
> version. See sample response below
> {code:java}
>
>
>
> Error 401 Unauthorized
>
> HTTP ERROR 401 Unauthorized
>
> URI:/
> STATUS:401
> MESSAGE:Unauthorized
> SERVLET:-
>
> https://eclipse.org/jetty;>Powered by Jetty://
> 9.4.44.v20210927
>
> {code}
>
> For security reason, it's not advisable to return server version in the
> response.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (CALCITE-6280) Jetty version number leaked by Avatica http server
[
https://issues.apache.org/jira/browse/CALCITE-6280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reopened CALCITE-6280:
---
> Jetty version number leaked by Avatica http server
> --
>
> Key: CALCITE-6280
> URL: https://issues.apache.org/jira/browse/CALCITE-6280
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Affects Versions: 1.24.0
>Reporter: Vaibhav Joshi
>Assignee: Vaibhav Joshi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.25.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Unauthorised access to HTTP server using curl returns the Jerry server
> version. See sample response below
> {code:java}
>
>
>
> Error 401 Unauthorized
>
> HTTP ERROR 401 Unauthorized
>
> URI:/
> STATUS:401
> MESSAGE:Unauthorized
> SERVLET:-
>
> https://eclipse.org/jetty;>Powered by Jetty://
> 9.4.44.v20210927
>
> {code}
>
> For security reason, it's not advisable to return server version in the
> response.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820512#comment-17820512 ] Alessandro Solimando edited comment on CALCITE-2040 at 2/25/24 7:33 PM: [~hongyuguo], I have left a (partial) review, there is enough to be looked at already I feel, I will finish the review sometime next week. Since you are the one currently working on it, it might make sense to assign the ticket to yourself and mark it as "in progress", wdyt? was (Author: asolimando): [~hongyuguo], I have left a (partial) review, there is enough to be looked at already I feel, I will finish the review sometime next week. > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820512#comment-17820512 ] Alessandro Solimando commented on CALCITE-2040: --- [~hongyuguo], I have left a (partial) review, there is enough to be looked at already I feel, I will finish the review sometime next week. > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6265) Cannot provide different numeric type as placeholder
[
https://issues.apache.org/jira/browse/CALCITE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817747#comment-17817747
]
Alessandro Solimando commented on CALCITE-6265:
---
In principle I agree, but it seems that the code is failing on the execution at
the "Enumerable" level, for this reason [~tnieradzik]'s PR is trying to apply
coercion when building the Linq expression.
Not sure this is what you mean [~julianhyde], because that's still on the
Calcite-side if I am not wrong, while you seem to suggest to look into the
Avatica-side.
> Cannot provide different numeric type as placeholder
>
>
> Key: CALCITE-6265
> URL: https://issues.apache.org/jira/browse/CALCITE-6265
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Tim Nieradzik
>Assignee: Tim Nieradzik
>Priority: Major
> Labels: pull-request-available
>
> The _empid_ column is of type {_}INT{_}. When providing a _short_ value as a
> placeholder, a _ClassCastException_ is thrown.
> *Test case:*
> {{ final String sql =}}
> {{ "select \"empid\" from \"hr\".\"emps\" where \"empid\" in (?,
> ?)";}}{{ CalciteAssert.hr()}}
> {{ .query(sql)}}
> {{ .consumesPreparedStatement(p -> {}}
> {{ p.setShort(1, (short) 100);}}
> {{ p.setShort(2, (short) 110);}}
> {{ })}}
> {{ .returnsUnordered("empid=100", "empid=110");}}
> *Stack trace:*
> {{java.lang.ClassCastException: class java.lang.Short cannot be cast to class
> java.lang.Integer (java.lang.Short and java.lang.Integer are in module
> java.base of loader 'bootstrap')}}
> {{ at Baz$1$1.moveNext(Unknown Source)}}
> {{ at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:679)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6265) Cannot provide different numeric type as placeholder
[
https://issues.apache.org/jira/browse/CALCITE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817547#comment-17817547
]
Alessandro Solimando commented on CALCITE-6265:
---
[~tnieradzik] I think it's worth mentioning in the ticket title that we are
dealing with a prepared statement, I didn't get what placeholder meant until I
saw the example, in the release notes we don't have them so it's better to
improve the title too.
> Cannot provide different numeric type as placeholder
>
>
> Key: CALCITE-6265
> URL: https://issues.apache.org/jira/browse/CALCITE-6265
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Tim Nieradzik
>Assignee: Tim Nieradzik
>Priority: Major
> Labels: pull-request-available
>
> The _empid_ column is of type {_}INT{_}. When providing a _short_ value as a
> placeholder, a _ClassCastException_ is thrown.
> *Test case:*
> {{ final String sql =}}
> {{ "select \"empid\" from \"hr\".\"emps\" where \"empid\" in (?,
> ?)";}}{{ CalciteAssert.hr()}}
> {{ .query(sql)}}
> {{ .consumesPreparedStatement(p -> {}}
> {{ p.setShort(1, (short) 100);}}
> {{ p.setShort(2, (short) 110);}}
> {{ })}}
> {{ .returnsUnordered("empid=100", "empid=110");}}
> *Stack trace:*
> {{java.lang.ClassCastException: class java.lang.Short cannot be cast to class
> java.lang.Integer (java.lang.Short and java.lang.Integer are in module
> java.base of loader 'bootstrap')}}
> {{ at Baz$1$1.moveNext(Unknown Source)}}
> {{ at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.(Linq4j.java:679)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6243) Upgrade Cassandra to 4.1.3 and DataStax driver for Cassandra to 4.17.0
[ https://issues.apache.org/jira/browse/CALCITE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816336#comment-17816336 ] Alessandro Solimando commented on CALCITE-6243: --- Fixed via [8fd9518|https://github.com/apache/calcite/commit/8fd9518887302af43e7e74cdb155474dcebfc9f5], thanks [~jiajunbernoulli] for the review. > Upgrade Cassandra to 4.1.3 and DataStax driver for Cassandra to 4.17.0 > -- > > Key: CALCITE-6243 > URL: https://issues.apache.org/jira/browse/CALCITE-6243 > Project: Calcite > Issue Type: Task > Components: cassandra-adapter >Affects Versions: 1.36.0 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: security > Fix For: 1.37.0 > > > Upgrading the following dependencies: > * the Cassandra version from 4.0.1 to 4.1.3 > * the DataStax Apache Cassandra driver from 4.13.0 to 4.17.0 > From 4.1.x Windows support is dropped by Cassandra (CASSANDRA-16956), so > Cassandra tests are now skipped when executing on Windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6243) Upgrade Cassandra to 4.1.3 and DataStax driver for Cassandra to 4.17.0
[ https://issues.apache.org/jira/browse/CALCITE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando resolved CALCITE-6243. --- Resolution: Fixed > Upgrade Cassandra to 4.1.3 and DataStax driver for Cassandra to 4.17.0 > -- > > Key: CALCITE-6243 > URL: https://issues.apache.org/jira/browse/CALCITE-6243 > Project: Calcite > Issue Type: Task > Components: cassandra-adapter >Affects Versions: 1.36.0 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: security > Fix For: 1.37.0 > > > Upgrading the following dependencies: > * the Cassandra version from 4.0.1 to 4.1.3 > * the DataStax Apache Cassandra driver from 4.13.0 to 4.17.0 > From 4.1.x Windows support is dropped by Cassandra (CASSANDRA-16956), so > Cassandra tests are now skipped when executing on Windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6243) Upgrade Cassandra to 4.1.3 and DataStax driver for Cassandra to 4.17.0
Alessandro Solimando created CALCITE-6243: - Summary: Upgrade Cassandra to 4.1.3 and DataStax driver for Cassandra to 4.17.0 Key: CALCITE-6243 URL: https://issues.apache.org/jira/browse/CALCITE-6243 Project: Calcite Issue Type: Task Components: cassandra-adapter Affects Versions: 1.36.0 Reporter: Alessandro Solimando Assignee: Alessandro Solimando Fix For: 1.37.0 Upgrading the following dependencies: * the Cassandra version from 4.0.1 to 4.1.3 * the DataStax Apache Cassandra driver from 4.13.0 to 4.17.0 >From 4.1.x Windows support is dropped by Cassandra (CASSANDRA-16956), so >Cassandra tests are now skipped when executing on Windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6237) Using fractions in LOG function does not return correct results
[
https://issues.apache.org/jira/browse/CALCITE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813889#comment-17813889
]
Alessandro Solimando commented on CALCITE-6237:
---
Calcite is a powerful but complex SW, there will always be weak spots in your
knowledge: see how many times committers and PMC members ask for a second
opinion or admit they are not very familiar with a certain area.
Knowledge will increase over time, but there is much more to it, like
describing your problem clearly and at the right abstraction level, providing
the needed context for others to understand your proposal/bug, backing up your
hypotheses with facts, being able to look up in the history if anything is
related and take it into consideration, splitting complex issues into simpler
ones (and creating minimal reproducers), how to interact with people, and I am
surely forgetting something along the way.
At first, one thinks that only contributing code matters, but it's way easier
to build those skills by a mix of contributions, reviews of work from others
and carefully following discussions on Jira and the ML.
No need to be anxious, all feedback is precious and meant to help you becoming
a better contributor. In a volunteer-based project nobody invests time
providing feedback if they don't see enough effort or the chance to improve, so
don't lose your motivation and keep and keep pushing.
Hth,
Alessandro
> Using fractions in LOG function does not return correct results
> ---
>
> Key: CALCITE-6237
> URL: https://issues.apache.org/jira/browse/CALCITE-6237
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Caican Cai
>Priority: Major
> Fix For: 1.37.0
>
>
> {code:java}
> mysql> SELECT LOG10(2/3) AS result;
> +-+
> | result |
> +-+
> | -0.1760912594899757 |
> +-+
> 1 row in set (0.00 sec)
> mysql> SELECT LOG2(0) AS result;
> ++
> | result |
> ++
> | NULL |
> ++
> 1 row in set, 1 warning (0.01 sec)
> mysql> SELECT LOG2(2/3) AS result;
> +-+
> | result |
> +-+
> | -0.5849625021638512 |
> +-+
> 1 row in set (0.00 sec)
> {code}
> When using Calcite's Log2 and Log10 functions, if the parameter is a
> fraction, the integer digits will be retained by default. I think this is
> wrong. Several decimal places (such as tens) should be retained by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6236) EnumerableBatchNestedLoopJoin uses wrong row count for cost calculation
[
https://issues.apache.org/jira/browse/CALCITE-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813303#comment-17813303
]
Alessandro Solimando commented on CALCITE-6236:
---
Equivalent in terms of cardinality (rows) yes, but cost not so sure, otherwise
there would be no interest at all in CBO, as the way you get to the same result
(the same number of rows, the cardinality) matters in terms of cost.
If we have a situation were "EnumerableBatchNestedLoopJoin" is always preferred
to "JdbcJoin" due to the extra filter the rewrite adds, one could either look
into improve the cardinality estimation for the filter by using data
statistics, or check if there isn't any problems with the cost model we use
(e.g., maybe the added cost of "JdbcToEnumerableConverter" should be higher?).
Do you guys have any running example for the same problem happening with
de-correlation so that we can reason on that too at the same time without
focusing too much on this specific example?
> EnumerableBatchNestedLoopJoin uses wrong row count for cost calculation
> ---
>
> Key: CALCITE-6236
> URL: https://issues.apache.org/jira/browse/CALCITE-6236
> Project: Calcite
> Issue Type: Bug
>Reporter: Ulrich Kramer
>Priority: Major
> Labels: pull-request-available
>
> {{EnumerableBatchNestedLoopJoin}} always adds a {{Filter}} on the right
> relation.
> This filter reduces the number of rows by it's selectivity (in our case by a
> factor of 4).
> Therefore, {{RelMdUtil.getJoinRowCount}} returns a value 4 times lower
> compared to the one returned for a {{JdbcJoin}}.
> This leads to the fact that in most cases {{EnumerableBatchNestedLoopJoin}}
> is preferred over {{JdbcJoin}}.
> This is an example for the different costs
> {code}
> EnumerableProject rows=460.0 self_costs=460.0 cumulative_costs=1465.0
> EnumerableBatchNestedLoopJoin rows=460.0 self_costs=687.5
> cumulative_costs=1005.0
> JdbcToEnumerableConverter rows=100.0 self_costs=10.0
> cumulative_costs=190.0
> JdbcProject rows=100.0 self_costs=80.0 cumulative_costs=180.0
> JdbcTableScan rows=100.0 self_costs=100.0 cumulative_costs=100.0
> JdbcToEnumerableConverter rows=25.0 self_costs=2.5 cumulative_costs=127.5
> JdbcFilter rows=25.0 self_costs=25.0 cumulative_costs=125.0
> JdbcTableScan rows=100.0 self_costs=100.0 cumulative_costs=100.0
> {code}
> vs.
> {code}
> JdbcToEnumerableConverter rows=1585.0 self_costs=158.5 cumulative_costs=2023.5
> JdbcJoin rows=1585.0 self_costs=1585.0 cumulative_costs=1865.0
> JdbcProject rows=100.0 self_costs=80.0 cumulative_costs=180.0
> JdbcTableScan rows=100.0 self_costs=100.0 cumulative_costs=100.0
> JdbcTableScan rows=100.0 self_costs=100.0 cumulative_costs=100.0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6230) Jira case collections
[ https://issues.apache.org/jira/browse/CALCITE-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811192#comment-17811192 ] Alessandro Solimando edited comment on CALCITE-6230 at 1/26/24 9:57 AM: [~hongyuguo], what's the purpose of this ticket? If multiple tickets can be grouped into a single coherent topic (probably more fine-grained than the labels you used), you can make an umbrella ticket and have them as sub-tickets. In general any Jira ticket ideally ends up in our release notes describing an improvement or bug-fix, but I don't see how this ticket can fit into this model. EDIT: maybe you can propose to add some extra labels if you want to categorize those tickets (like: "dialect") and see what the community thinks of the proposal. was (Author: asolimando): [~hongyuguo], what's the purpose of this ticket? If multiple tickets can be grouped into a single coherent topic (probably more fine-grained than the labels you used), you can make an umbrella ticket and have them as sub-tickets. In general any Jira ticket ideally ends up in our release notes describing an improvement or bug-fix, but I don't see how this ticket can fit into this model. > Jira case collections > - > > Key: CALCITE-6230 > URL: https://issues.apache.org/jira/browse/CALCITE-6230 > Project: Calcite > Issue Type: Task >Reporter: hongyu guo >Priority: Minor > > syntax: > CALCITE-5852: MERGE INTO > CALCITE-5301: AT TIME ZONE > CALCITE-5386: LIKE (ANY | SOME | ALL) > CALCITE-5216: Cannot parse parenthesized nested WITH clause > CALCITE-5205: Supports hint option as string and numeric literal > CALCITE-5168: Allow AS after parenthesized JOIN > CALCITE-5084: Support ROWS syntax with TABLESAMPLE > CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) > CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ > CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY > statement > CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax > CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands > data type: > CALCITE-5346: type aliases > CALCITE-4918: Add a VARIANT data type > rule and optimization: > CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= > CALCITE-4052: Enable Top-down Optimization > dialect: > CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) > function: > CALCITE-5809: enable already available functions in Apache Spark Library > CALCITE-5087: Support bitwise operators > CALCITE-4521: Support User Defined Table-valued Function > CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not > unique > CALCITE-3683: Enhanced MATH Function > CALCITE-3646: MySQL compression functions > CALCITE-2031: Implement more OpenGIS functions > CALCITE-2871: Implement JSON_TABLE table function -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6230) Jira case collections
[ https://issues.apache.org/jira/browse/CALCITE-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811192#comment-17811192 ] Alessandro Solimando commented on CALCITE-6230: --- [~hongyuguo], what's the purpose of this ticket? If multiple tickets can be grouped into a single coherent topic (probably more fine-grained than the labels you used), you can make an umbrella ticket and have them as sub-tickets. In general any Jira ticket ideally ends up in our release notes describing an improvement or bug-fix, but I don't see how this ticket can fit into this model. > Jira case collections > - > > Key: CALCITE-6230 > URL: https://issues.apache.org/jira/browse/CALCITE-6230 > Project: Calcite > Issue Type: Task >Reporter: hongyu guo >Priority: Minor > > syntax: > CALCITE-5852: MERGE INTO > CALCITE-5301: AT TIME ZONE > CALCITE-5386: LIKE (ANY | SOME | ALL) > CALCITE-5216: Cannot parse parenthesized nested WITH clause > CALCITE-5205: Supports hint option as string and numeric literal > CALCITE-5168: Allow AS after parenthesized JOIN > CALCITE-5084: Support ROWS syntax with TABLESAMPLE > CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) > CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ > CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY > statement > CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax > CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands > data type: > CALCITE-5346: type aliases > CALCITE-4918: Add a VARIANT data type > rule and optimization: > CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= > CALCITE-4052: Enable Top-down Optimization > dialect: > CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) > function: > CALCITE-5809: enable already available functions in Apache Spark Library > CALCITE-5087: Support bitwise operators > CALCITE-4521: Support User Defined Table-valued Function > CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not > unique > CALCITE-3683: Enhanced MATH Function > CALCITE-3646: MySQL compression functions > CALCITE-2031: Implement more OpenGIS functions > CALCITE-2871: Implement JSON_TABLE table function -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6195) SqlValidator validateParameterizedExpression is not supporting qualified paths
[
https://issues.apache.org/jira/browse/CALCITE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805212#comment-17805212
]
Alessandro Solimando edited comment on CALCITE-6195 at 1/10/24 4:20 PM:
I think it's just a matter of finding the right combination of casing + quoting.
I suggest to try to write the same query with _sqline_ and use qualified names,
and see what works there, there is a good chance it will work even in your code
example.
For instance I had a small toy example for querying postgres which was as
follows:
{noformat}
final String postgresQuery = "select postgres.\"table1\".\"id\",
postgres.\"table1\".\"field1\", postgres.\"table2\".\"field1\" " +
"from postgres.\"table1\" join postgres.\"table2\" " +
" on postgres.\"table1\".\"id\" = postgres.\"table2\".\"id1\""
+
"where postgres.\"table1\".\"id\" < 5";{noformat}
Try also possibly to set relevant properties for quoting and casing, like in
the following (non-exhaustive) example:
{noformat}
final Properties info = new Properties();
info.setProperty("lex", "POSTGRES");
info.put(InternalProperty.CASE_SENSITIVE, true);
info.put(InternalProperty.UNQUOTED_CASING, Casing.TO_LOWER);
info.put(InternalProperty.QUOTED_CASING, Casing.UNCHANGED);{noformat}
was (Author: asolimando):
I think it's just a matter of finding the right combination of casing + quoting.
I suggest to try to write the same query with _sqline_ and use qualified names,
and see what works there, there is a good chance it will work even in your code
example, for instance:
{noformat}
final String postgresQuery = "select postgres.\"table1\".\"id\",
postgres.\"table1\".\"field1\", postgres.\"table2\".\"field1\" " +
"from postgres.\"table1\" join postgres.\"table2\" " +
" on postgres.\"table1\".\"id\" = postgres.\"table2\".\"id1\""
+
"where postgres.\"table1\".\"id\" < 5";{noformat}
Try also possibly to set relevant properties for quoting and casing, like in
the following (non-exhaustive) example:
{noformat}
final Properties info = new Properties();
info.setProperty("lex", "POSTGRES");
info.put(InternalProperty.CASE_SENSITIVE, true);
info.put(InternalProperty.UNQUOTED_CASING, Casing.TO_LOWER);
info.put(InternalProperty.QUOTED_CASING, Casing.UNCHANGED);{noformat}
> SqlValidator validateParameterizedExpression is not supporting qualified paths
> --
>
> Key: CALCITE-6195
> URL: https://issues.apache.org/jira/browse/CALCITE-6195
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: David Dali Susanibar Arce
>Priority: Major
>
> The SQLValidator.validateParameterizedExpression method can be used to
> validate parameterized expressions.
> It is possible to register
> [validateParameterizedExpression|https://github.com/apache/calcite/blob/1b11d99e65d03a15ae4b25c47250b6918ce9aa10/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11421:L11435]:
> {code:java}
> final Map nameToTypeMap = new HashMap<>();
> nameToTypeMap.put("A", intType);
> nameToTypeMap.put("B", intTypeNull);
> final String expr = "a + b";
> final SqlNode validated = validator.validateParameterizedExpression(sqlNode,
> nameToTypeMap);{code}
>
> Problems appear if we are trying to register qualified paths:
> {code:java}
> final Map nameToTypeMap = new HashMap<>();
> nameToTypeMap.put("EMP.A", intType);
> nameToTypeMap.put("EMP.B", intTypeNull);
> final String expr = "EMP.a + EMP.b";
> final SqlNode validated = validator.validateParameterizedExpression(sqlNode,
> nameToTypeMap);
> Error:
> org.apache.calcite.runtime.CalciteContextException: From line 1, column 1 to
> line 1, column 3: Unknown identifier 'EMP'
> {code}
> I would appreciate if you could provide me with some advice regarding how to
> resolve this error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6195) SqlValidator validateParameterizedExpression is not supporting qualified paths
[
https://issues.apache.org/jira/browse/CALCITE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805212#comment-17805212
]
Alessandro Solimando edited comment on CALCITE-6195 at 1/10/24 4:19 PM:
I think it's just a matter of finding the right combination of casing + quoting.
I suggest to try to write the same query with _sqline_ and use qualified names,
and see what works there, there is a good chance it will work even in your code
example, for instance:
{noformat}
final String postgresQuery = "select postgres.\"table1\".\"id\",
postgres.\"table1\".\"field1\", postgres.\"table2\".\"field1\" " +
"from postgres.\"table1\" join postgres.\"table2\" " +
" on postgres.\"table1\".\"id\" = postgres.\"table2\".\"id1\""
+
"where postgres.\"table1\".\"id\" < 5";{noformat}
Try also possibly to set relevant properties for quoting and casing, like in
the following (non-exhaustive) example:
{noformat}
final Properties info = new Properties();
info.setProperty("lex", "POSTGRES");
info.put(InternalProperty.CASE_SENSITIVE, true);
info.put(InternalProperty.UNQUOTED_CASING, Casing.TO_LOWER);
info.put(InternalProperty.QUOTED_CASING, Casing.UNCHANGED);{noformat}
was (Author: asolimando):
I think it's just a matter of finding the right combination of casing + quoting.
I suggest to try to write the same query with _sqline_ and use __ qualified
names, and see what works there, there is a good chance it will work even in
your code example, for instance:
{noformat}
final String postgresQuery = "select postgres.\"table1\".\"id\",
postgres.\"table1\".\"field1\", postgres.\"table2\".\"field1\" " +
"from postgres.\"table1\" join postgres.\"table2\" " +
" on postgres.\"table1\".\"id\" = postgres.\"table2\".\"id1\""
+
"where postgres.\"table1\".\"id\" < 5";{noformat}
Try also possibly to set relevant properties for quoting and casing, like in
the following (non-exhaustive) example:
{noformat}
final Properties info = new Properties();
info.setProperty("lex", "POSTGRES");
info.put(InternalProperty.CASE_SENSITIVE, true);
info.put(InternalProperty.UNQUOTED_CASING, Casing.TO_LOWER);
info.put(InternalProperty.QUOTED_CASING, Casing.UNCHANGED);{noformat}
> SqlValidator validateParameterizedExpression is not supporting qualified paths
> --
>
> Key: CALCITE-6195
> URL: https://issues.apache.org/jira/browse/CALCITE-6195
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: David Dali Susanibar Arce
>Priority: Major
>
> The SQLValidator.validateParameterizedExpression method can be used to
> validate parameterized expressions.
> It is possible to register
> [validateParameterizedExpression|https://github.com/apache/calcite/blob/1b11d99e65d03a15ae4b25c47250b6918ce9aa10/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11421:L11435]:
> {code:java}
> final Map nameToTypeMap = new HashMap<>();
> nameToTypeMap.put("A", intType);
> nameToTypeMap.put("B", intTypeNull);
> final String expr = "a + b";
> final SqlNode validated = validator.validateParameterizedExpression(sqlNode,
> nameToTypeMap);{code}
>
> Problems appear if we are trying to register qualified paths:
> {code:java}
> final Map nameToTypeMap = new HashMap<>();
> nameToTypeMap.put("EMP.A", intType);
> nameToTypeMap.put("EMP.B", intTypeNull);
> final String expr = "EMP.a + EMP.b";
> final SqlNode validated = validator.validateParameterizedExpression(sqlNode,
> nameToTypeMap);
> Error:
> org.apache.calcite.runtime.CalciteContextException: From line 1, column 1 to
> line 1, column 3: Unknown identifier 'EMP'
> {code}
> I would appreciate if you could provide me with some advice regarding how to
> resolve this error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6195) SqlValidator validateParameterizedExpression is not supporting qualified paths
[
https://issues.apache.org/jira/browse/CALCITE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805212#comment-17805212
]
Alessandro Solimando commented on CALCITE-6195:
---
I think it's just a matter of finding the right combination of casing + quoting.
I suggest to try to write the same query with _sqline_ and use __ qualified
names, and see what works there, there is a good chance it will work even in
your code example, for instance:
{noformat}
final String postgresQuery = "select postgres.\"table1\".\"id\",
postgres.\"table1\".\"field1\", postgres.\"table2\".\"field1\" " +
"from postgres.\"table1\" join postgres.\"table2\" " +
" on postgres.\"table1\".\"id\" = postgres.\"table2\".\"id1\""
+
"where postgres.\"table1\".\"id\" < 5";{noformat}
Try also possibly to set relevant properties for quoting and casing, like in
the following (non-exhaustive) example:
{noformat}
final Properties info = new Properties();
info.setProperty("lex", "POSTGRES");
info.put(InternalProperty.CASE_SENSITIVE, true);
info.put(InternalProperty.UNQUOTED_CASING, Casing.TO_LOWER);
info.put(InternalProperty.QUOTED_CASING, Casing.UNCHANGED);{noformat}
> SqlValidator validateParameterizedExpression is not supporting qualified paths
> --
>
> Key: CALCITE-6195
> URL: https://issues.apache.org/jira/browse/CALCITE-6195
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: David Dali Susanibar Arce
>Priority: Major
>
> The SQLValidator.validateParameterizedExpression method can be used to
> validate parameterized expressions.
> It is possible to register
> [validateParameterizedExpression|https://github.com/apache/calcite/blob/1b11d99e65d03a15ae4b25c47250b6918ce9aa10/core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java#L11421:L11435]:
> {code:java}
> final Map nameToTypeMap = new HashMap<>();
> nameToTypeMap.put("A", intType);
> nameToTypeMap.put("B", intTypeNull);
> final String expr = "a + b";
> final SqlNode validated = validator.validateParameterizedExpression(sqlNode,
> nameToTypeMap);{code}
>
> Problems appear if we are trying to register qualified paths:
> {code:java}
> final Map nameToTypeMap = new HashMap<>();
> nameToTypeMap.put("EMP.A", intType);
> nameToTypeMap.put("EMP.B", intTypeNull);
> final String expr = "EMP.a + EMP.b";
> final SqlNode validated = validator.validateParameterizedExpression(sqlNode,
> nameToTypeMap);
> Error:
> org.apache.calcite.runtime.CalciteContextException: From line 1, column 1 to
> line 1, column 3: Unknown identifier 'EMP'
> {code}
> I would appreciate if you could provide me with some advice regarding how to
> resolve this error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6145) Function 'TRIM' without parameters throw NullPointerException
[
https://issues.apache.org/jira/browse/CALCITE-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800275#comment-17800275
]
Alessandro Solimando commented on CALCITE-6145:
---
[~simonalexs] no need to be a committer for the ML and I am not aware of any
restrictions regarding email domains (I think I have seen QQ emails used here
in the past).
Apache MLs are moderated and require you to subscribe though (see
[https://calcite.apache.org/community/#mailing-lists|https://calcite.apache.org/community/#mailing-lists]
), your previous email is probably awaiting for manual approval.
If you register you should be able to send and receive emails, if that's not
the case let us know.
> Function 'TRIM' without parameters throw NullPointerException
> -
>
> Key: CALCITE-6145
> URL: https://issues.apache.org/jira/browse/CALCITE-6145
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: SimonAlexs
>Priority: Major
> Labels: newbie
>
> When use sql as follow, trim without parameters, it throws
> java.lang.NullPointerException.
> {code:java}
> select trim(){code}
> Error position is org.apache.calcite.sql.SqlOperator#constructArgNameList.
> Source code is below. Line 5 "operand.getKind()" results in this exception.
>
> {code:java}
> protected @Nullable List constructArgNameList(SqlCall call) {
> // If any arguments are named, construct a map.
> final ImmutableList.Builder nameBuilder = ImmutableList.builder();
> for (SqlNode operand : call.getOperandList()) {
> if (operand.getKind() == SqlKind.ARGUMENT_ASSIGNMENT) {
> final List operandList = ((SqlCall) operand).getOperandList();
> nameBuilder.add(((SqlIdentifier) operandList.get(1)).getSimple());
> }
> }
> ImmutableList argNames = nameBuilder.build();
> if (argNames.isEmpty()) {
> return null;
> } else {
> return argNames;
> }
> } {code}
>
> Should it throw an exact description?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6145) Function 'TRIM' without parameters throw NullPointerException
[
https://issues.apache.org/jira/browse/CALCITE-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798891#comment-17798891
]
Alessandro Solimando commented on CALCITE-6145:
---
[~simonalexs], that kind conversations usually happen in the project mailing
list (see [https://calcite.apache.org/develop/#getting-started] for details),
it's pretty active and Q is common there, it's also a rich source of
information about the project, so definitely worth checking it out.
If there is agreement in the discussion in the ML that there is a limitation or
an issue, a Jira ticket usually follows to track it, and to continue the
high-level discussion, if necessary.
Implementation-level discussions happen on GitHub once a PR is finally
available, if any.
HTH!
> Function 'TRIM' without parameters throw NullPointerException
> -
>
> Key: CALCITE-6145
> URL: https://issues.apache.org/jira/browse/CALCITE-6145
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: SimonAlexs
>Priority: Major
> Labels: newbie
>
> When use sql as follow, trim without parameters, it throws
> java.lang.NullPointerException.
> {code:java}
> select trim(){code}
> Error position is org.apache.calcite.sql.SqlOperator#constructArgNameList.
> Source code is below. Line 5 "operand.getKind()" results in this exception.
>
> {code:java}
> protected @Nullable List constructArgNameList(SqlCall call) {
> // If any arguments are named, construct a map.
> final ImmutableList.Builder nameBuilder = ImmutableList.builder();
> for (SqlNode operand : call.getOperandList()) {
> if (operand.getKind() == SqlKind.ARGUMENT_ASSIGNMENT) {
> final List operandList = ((SqlCall) operand).getOperandList();
> nameBuilder.add(((SqlIdentifier) operandList.get(1)).getSimple());
> }
> }
> ImmutableList argNames = nameBuilder.build();
> if (argNames.isEmpty()) {
> return null;
> } else {
> return argNames;
> }
> } {code}
>
> Should it throw an exact description?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-6115) Interval type specifier with zero fractional second precision does not pass validation
[
https://issues.apache.org/jira/browse/CALCITE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-6115.
---
Resolution: Fixed
> Interval type specifier with zero fractional second precision does not pass
> validation
> --
>
> Key: CALCITE-6115
> URL: https://issues.apache.org/jira/browse/CALCITE-6115
> Project: Calcite
> Issue Type: Bug
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Major
> Labels: pull-request-available
>
> Consider interval expression
> {code:java}
> interval '1' second(1, 0) {code}
> Calcite SQL validator considers it as not correct, since it uses following
> lower bound for fractional seconds precision:
> {code:java}
> public static final int MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION = 1;{code}
> In order to reproduce this issue one can add following test cast to
> SqlValidatorTest.java:
> {code:java}
> @Test void testSecondIntervalExpression() {
> expr("interval '1' second(1, 0)").columnType("INTERVAL SECOND(1, 0) NOT
> NULL");
> } {code}
> and get an error:
> {code:java}
> Interval fractional second precision '0' out of range for INTERVAL SECOND(1,
> 0) {code}
> However, SQL standard say:
> {code:java}
> An , if specified, shall be greater
> than or equal to 0 (zero)
> and shall not be greater than the implementation-defined maximum. If SECOND
> is specified
> and is not specified, then an
> precision> of 6 is implicit. {code}
> Consequently, MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION should be equal to 0
> to make Calcite behavior consistent with SQL specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6115) Interval type specifier with zero fractional second precision does not pass validation
[
https://issues.apache.org/jira/browse/CALCITE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-6115:
--
Affects Version/s: 1.36.0
> Interval type specifier with zero fractional second precision does not pass
> validation
> --
>
> Key: CALCITE-6115
> URL: https://issues.apache.org/jira/browse/CALCITE-6115
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Major
> Labels: pull-request-available
>
> Consider interval expression
> {code:java}
> interval '1' second(1, 0) {code}
> Calcite SQL validator considers it as not correct, since it uses following
> lower bound for fractional seconds precision:
> {code:java}
> public static final int MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION = 1;{code}
> In order to reproduce this issue one can add following test cast to
> SqlValidatorTest.java:
> {code:java}
> @Test void testSecondIntervalExpression() {
> expr("interval '1' second(1, 0)").columnType("INTERVAL SECOND(1, 0) NOT
> NULL");
> } {code}
> and get an error:
> {code:java}
> Interval fractional second precision '0' out of range for INTERVAL SECOND(1,
> 0) {code}
> However, SQL standard say:
> {code:java}
> An , if specified, shall be greater
> than or equal to 0 (zero)
> and shall not be greater than the implementation-defined maximum. If SECOND
> is specified
> and is not specified, then an
> precision> of 6 is implicit. {code}
> Consequently, MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION should be equal to 0
> to make Calcite behavior consistent with SQL specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6115) Interval type specifier with zero fractional second precision does not pass validation
[
https://issues.apache.org/jira/browse/CALCITE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-6115:
--
Component/s: core
> Interval type specifier with zero fractional second precision does not pass
> validation
> --
>
> Key: CALCITE-6115
> URL: https://issues.apache.org/jira/browse/CALCITE-6115
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Major
> Labels: pull-request-available
>
> Consider interval expression
> {code:java}
> interval '1' second(1, 0) {code}
> Calcite SQL validator considers it as not correct, since it uses following
> lower bound for fractional seconds precision:
> {code:java}
> public static final int MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION = 1;{code}
> In order to reproduce this issue one can add following test cast to
> SqlValidatorTest.java:
> {code:java}
> @Test void testSecondIntervalExpression() {
> expr("interval '1' second(1, 0)").columnType("INTERVAL SECOND(1, 0) NOT
> NULL");
> } {code}
> and get an error:
> {code:java}
> Interval fractional second precision '0' out of range for INTERVAL SECOND(1,
> 0) {code}
> However, SQL standard say:
> {code:java}
> An , if specified, shall be greater
> than or equal to 0 (zero)
> and shall not be greater than the implementation-defined maximum. If SECOND
> is specified
> and is not specified, then an
> precision> of 6 is implicit. {code}
> Consequently, MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION should be equal to 0
> to make Calcite behavior consistent with SQL specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6115) Interval type specifier with zero fractional second precision does not pass validation
[
https://issues.apache.org/jira/browse/CALCITE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-6115:
--
Fix Version/s: 1.37.0
> Interval type specifier with zero fractional second precision does not pass
> validation
> --
>
> Key: CALCITE-6115
> URL: https://issues.apache.org/jira/browse/CALCITE-6115
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> Consider interval expression
> {code:java}
> interval '1' second(1, 0) {code}
> Calcite SQL validator considers it as not correct, since it uses following
> lower bound for fractional seconds precision:
> {code:java}
> public static final int MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION = 1;{code}
> In order to reproduce this issue one can add following test cast to
> SqlValidatorTest.java:
> {code:java}
> @Test void testSecondIntervalExpression() {
> expr("interval '1' second(1, 0)").columnType("INTERVAL SECOND(1, 0) NOT
> NULL");
> } {code}
> and get an error:
> {code:java}
> Interval fractional second precision '0' out of range for INTERVAL SECOND(1,
> 0) {code}
> However, SQL standard say:
> {code:java}
> An , if specified, shall be greater
> than or equal to 0 (zero)
> and shall not be greater than the implementation-defined maximum. If SECOND
> is specified
> and is not specified, then an
> precision> of 6 is implicit. {code}
> Consequently, MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION should be equal to 0
> to make Calcite behavior consistent with SQL specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6115) Interval type specifier with zero fractional second precision does not pass validation
[
https://issues.apache.org/jira/browse/CALCITE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796800#comment-17796800
]
Alessandro Solimando commented on CALCITE-6115:
---
Fixed via
[{{e4bd21e}}|https://github.com/apache/calcite/commit/e4bd21ec3bd04f9afeacc155ad50c843ba67f343],
thanks [~lchistov1987] for your contribution, and [~mbudiu] for the review!
> Interval type specifier with zero fractional second precision does not pass
> validation
> --
>
> Key: CALCITE-6115
> URL: https://issues.apache.org/jira/browse/CALCITE-6115
> Project: Calcite
> Issue Type: Bug
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Major
> Labels: pull-request-available
>
> Consider interval expression
> {code:java}
> interval '1' second(1, 0) {code}
> Calcite SQL validator considers it as not correct, since it uses following
> lower bound for fractional seconds precision:
> {code:java}
> public static final int MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION = 1;{code}
> In order to reproduce this issue one can add following test cast to
> SqlValidatorTest.java:
> {code:java}
> @Test void testSecondIntervalExpression() {
> expr("interval '1' second(1, 0)").columnType("INTERVAL SECOND(1, 0) NOT
> NULL");
> } {code}
> and get an error:
> {code:java}
> Interval fractional second precision '0' out of range for INTERVAL SECOND(1,
> 0) {code}
> However, SQL standard say:
> {code:java}
> An , if specified, shall be greater
> than or equal to 0 (zero)
> and shall not be greater than the implementation-defined maximum. If SECOND
> is specified
> and is not specified, then an
> precision> of 6 is implicit. {code}
> Consequently, MIN_INTERVAL_FRACTIONAL_SECOND_PRECISION should be equal to 0
> to make Calcite behavior consistent with SQL specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CALCITE-5981) TIMESTAMPDIFF function returns incorrect result
[
https://issues.apache.org/jira/browse/CALCITE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-5981:
-
Assignee: Mihai Budiu (was: Alessandro Solimando)
> TIMESTAMPDIFF function returns incorrect result
>
>
> Key: CALCITE-5981
> URL: https://issues.apache.org/jira/browse/CALCITE-5981
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Affects Versions: 1.23.0
>Reporter: Mihai Budiu
>Assignee: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.24.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test fails, when added to SqlOperatorTest:
> {code:java}
> f.checkScalar("timestampdiff(month, DATE '2004-02-29', DATE '2005-02-28')",
> "11", "INTEGER NOT NULL");
> {code}
> The result returned by the expression is 12. However, MySQL returns 11.
> The semantics of this function is not described clearly in the documentation,
> but according to several prior issues [1] [2] [3] the intended semantics
> should be the same as in MySQL.
> A corresponding MySQL test: [4]
> The implementation seems to be in
> StandardConvertletTable.TimestampDiffConvertlet.
> [1] https://issues.apache.org/jira/browse/CALCITE-1827,
> [2] https://issues.apache.org/jira/browse/CALCITE-3529,
> [3] https://issues.apache.org/jira/browse/CALCITE-1124.
> [4]
> [https://github.com/mysql/mysql-server/blob/ea1efa9822d81044b726aab20c857d5e1b7e046a/mysql-test/r/func_time.result#L1151]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5981) TIMESTAMPDIFF function returns incorrect result
[
https://issues.apache.org/jira/browse/CALCITE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5981:
--
Affects Version/s: 1.23.0
(was: 1.35.0)
> TIMESTAMPDIFF function returns incorrect result
>
>
> Key: CALCITE-5981
> URL: https://issues.apache.org/jira/browse/CALCITE-5981
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Mihai Budiu
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test fails, when added to SqlOperatorTest:
> {code:java}
> f.checkScalar("timestampdiff(month, DATE '2004-02-29', DATE '2005-02-28')",
> "11", "INTEGER NOT NULL");
> {code}
> The result returned by the expression is 12. However, MySQL returns 11.
> The semantics of this function is not described clearly in the documentation,
> but according to several prior issues [1] [2] [3] the intended semantics
> should be the same as in MySQL.
> A corresponding MySQL test: [4]
> The implementation seems to be in
> StandardConvertletTable.TimestampDiffConvertlet.
> [1] https://issues.apache.org/jira/browse/CALCITE-1827,
> [2] https://issues.apache.org/jira/browse/CALCITE-3529,
> [3] https://issues.apache.org/jira/browse/CALCITE-1124.
> [4]
> [https://github.com/mysql/mysql-server/blob/ea1efa9822d81044b726aab20c857d5e1b7e046a/mysql-test/r/func_time.result#L1151]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5981) TIMESTAMPDIFF function returns incorrect result
[
https://issues.apache.org/jira/browse/CALCITE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5981:
--
Component/s: avatica
(was: core)
> TIMESTAMPDIFF function returns incorrect result
>
>
> Key: CALCITE-5981
> URL: https://issues.apache.org/jira/browse/CALCITE-5981
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Affects Versions: 1.23.0
>Reporter: Mihai Budiu
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.24.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test fails, when added to SqlOperatorTest:
> {code:java}
> f.checkScalar("timestampdiff(month, DATE '2004-02-29', DATE '2005-02-28')",
> "11", "INTEGER NOT NULL");
> {code}
> The result returned by the expression is 12. However, MySQL returns 11.
> The semantics of this function is not described clearly in the documentation,
> but according to several prior issues [1] [2] [3] the intended semantics
> should be the same as in MySQL.
> A corresponding MySQL test: [4]
> The implementation seems to be in
> StandardConvertletTable.TimestampDiffConvertlet.
> [1] https://issues.apache.org/jira/browse/CALCITE-1827,
> [2] https://issues.apache.org/jira/browse/CALCITE-3529,
> [3] https://issues.apache.org/jira/browse/CALCITE-1124.
> [4]
> [https://github.com/mysql/mysql-server/blob/ea1efa9822d81044b726aab20c857d5e1b7e046a/mysql-test/r/func_time.result#L1151]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5981) TIMESTAMPDIFF function returns incorrect result
[
https://issues.apache.org/jira/browse/CALCITE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5981:
--
Fix Version/s: 1.24.0
> TIMESTAMPDIFF function returns incorrect result
>
>
> Key: CALCITE-5981
> URL: https://issues.apache.org/jira/browse/CALCITE-5981
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.23.0
>Reporter: Mihai Budiu
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.24.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test fails, when added to SqlOperatorTest:
> {code:java}
> f.checkScalar("timestampdiff(month, DATE '2004-02-29', DATE '2005-02-28')",
> "11", "INTEGER NOT NULL");
> {code}
> The result returned by the expression is 12. However, MySQL returns 11.
> The semantics of this function is not described clearly in the documentation,
> but according to several prior issues [1] [2] [3] the intended semantics
> should be the same as in MySQL.
> A corresponding MySQL test: [4]
> The implementation seems to be in
> StandardConvertletTable.TimestampDiffConvertlet.
> [1] https://issues.apache.org/jira/browse/CALCITE-1827,
> [2] https://issues.apache.org/jira/browse/CALCITE-3529,
> [3] https://issues.apache.org/jira/browse/CALCITE-1124.
> [4]
> [https://github.com/mysql/mysql-server/blob/ea1efa9822d81044b726aab20c857d5e1b7e046a/mysql-test/r/func_time.result#L1151]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5981) TIMESTAMPDIFF function returns incorrect result
[
https://issues.apache.org/jira/browse/CALCITE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5981.
---
Resolution: Fixed
Fixed via
[{{247fecd}}|https://github.com/apache/calcite-avatica/commit/247fecdab7d2e949995210da13bb042fbdc8fb34],
thanks [~mbudiu] for your contribution!
> TIMESTAMPDIFF function returns incorrect result
>
>
> Key: CALCITE-5981
> URL: https://issues.apache.org/jira/browse/CALCITE-5981
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test fails, when added to SqlOperatorTest:
> {code:java}
> f.checkScalar("timestampdiff(month, DATE '2004-02-29', DATE '2005-02-28')",
> "11", "INTEGER NOT NULL");
> {code}
> The result returned by the expression is 12. However, MySQL returns 11.
> The semantics of this function is not described clearly in the documentation,
> but according to several prior issues [1] [2] [3] the intended semantics
> should be the same as in MySQL.
> A corresponding MySQL test: [4]
> The implementation seems to be in
> StandardConvertletTable.TimestampDiffConvertlet.
> [1] https://issues.apache.org/jira/browse/CALCITE-1827,
> [2] https://issues.apache.org/jira/browse/CALCITE-3529,
> [3] https://issues.apache.org/jira/browse/CALCITE-1124.
> [4]
> [https://github.com/mysql/mysql-server/blob/ea1efa9822d81044b726aab20c857d5e1b7e046a/mysql-test/r/func_time.result#L1151]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CALCITE-5981) TIMESTAMPDIFF function returns incorrect result
[
https://issues.apache.org/jira/browse/CALCITE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-5981:
-
Assignee: Alessandro Solimando (was: Mihai Budiu)
> TIMESTAMPDIFF function returns incorrect result
>
>
> Key: CALCITE-5981
> URL: https://issues.apache.org/jira/browse/CALCITE-5981
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Assignee: Alessandro Solimando
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test fails, when added to SqlOperatorTest:
> {code:java}
> f.checkScalar("timestampdiff(month, DATE '2004-02-29', DATE '2005-02-28')",
> "11", "INTEGER NOT NULL");
> {code}
> The result returned by the expression is 12. However, MySQL returns 11.
> The semantics of this function is not described clearly in the documentation,
> but according to several prior issues [1] [2] [3] the intended semantics
> should be the same as in MySQL.
> A corresponding MySQL test: [4]
> The implementation seems to be in
> StandardConvertletTable.TimestampDiffConvertlet.
> [1] https://issues.apache.org/jira/browse/CALCITE-1827,
> [2] https://issues.apache.org/jira/browse/CALCITE-3529,
> [3] https://issues.apache.org/jira/browse/CALCITE-1124.
> [4]
> [https://github.com/mysql/mysql-server/blob/ea1efa9822d81044b726aab20c857d5e1b7e046a/mysql-test/r/func_time.result#L1151]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CALCITE-5981) TIMESTAMPDIFF function returns incorrect result
[
https://issues.apache.org/jira/browse/CALCITE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-5981:
-
Assignee: Mihai Budiu
> TIMESTAMPDIFF function returns incorrect result
>
>
> Key: CALCITE-5981
> URL: https://issues.apache.org/jira/browse/CALCITE-5981
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Assignee: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test fails, when added to SqlOperatorTest:
> {code:java}
> f.checkScalar("timestampdiff(month, DATE '2004-02-29', DATE '2005-02-28')",
> "11", "INTEGER NOT NULL");
> {code}
> The result returned by the expression is 12. However, MySQL returns 11.
> The semantics of this function is not described clearly in the documentation,
> but according to several prior issues [1] [2] [3] the intended semantics
> should be the same as in MySQL.
> A corresponding MySQL test: [4]
> The implementation seems to be in
> StandardConvertletTable.TimestampDiffConvertlet.
> [1] https://issues.apache.org/jira/browse/CALCITE-1827,
> [2] https://issues.apache.org/jira/browse/CALCITE-3529,
> [3] https://issues.apache.org/jira/browse/CALCITE-1124.
> [4]
> [https://github.com/mysql/mysql-server/blob/ea1efa9822d81044b726aab20c857d5e1b7e046a/mysql-test/r/func_time.result#L1151]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5889) Add the RelRule that converts Minus into UNION ALL..GROUP BY...WHERE
[
https://issues.apache.org/jira/browse/CALCITE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5889.
---
Resolution: Fixed
Fixed via
[{{9c75894}}|https://github.com/apache/calcite/commit/9c758942ab51af22a1095087ea2daeabc7d692ea],
thanks [~shenlang] for your contribution, thanks [~nobigo], [~tanclary] and
[~Runking] for your reviews!
> Add the RelRule that converts Minus into UNION ALL..GROUP BY...WHERE
>
>
> Key: CALCITE-5889
> URL: https://issues.apache.org/jira/browse/CALCITE-5889
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Reporter: LakeShen
>Assignee: LakeShen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> There are many projects that implement optimizers based on Calcite,our
> optimizer is also based on Calcite.
> Calcite has a lot of good rules in CoreRules.It has UnionToDistinctRule and
> IntersectToDistinctRule RelRule ,UnionToDistinctRule is that converts
> Union(all = false) to Union(all=true) + Aggregate,IntersectToDistinctRule is
> that converts Intersect(all=false) to Union(all=true) + Aggregate +
> Filter.None of these rules translate Minus to other RelNode combinations.
> Normally, a computation engine does not have a Minus operator, so it is
> common to convert Minus to some other relational algebra combination in the
> optimizer.For example,in presto,it has the ImplementIntersectAndExceptAsUnion
> PlanOptimizer that converts Minus to into UNION ALL..GROUP BY...WHERE. In
> flink,it has the ReplaceMinusWithAntiJoinRule RelOptRule. In dremio-oss,it
> has MinusToJoin RelOptRule.All of this rules,converts the Minus to other
> composition of relational algebra.
> Since there are no optimization rules for dealing with minus in calcite,
> users of calcite generally need to write their own optimization rules for
> dealing with Minus.
> I think it makes sense to add the RelRule that converts Minus to other
> composition of relational algebranto ,like UNION ALL.. GROUP BY... WHERE. So
> for users of calcite,they don't have to go through the extra work of writing
> rules for Minus to convert it to something else.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5972) cannot execute a sql like 'insert into A select * from B'
[
https://issues.apache.org/jira/browse/CALCITE-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761712#comment-17761712
]
Alessandro Solimando commented on CALCITE-5972:
---
For now I suggest to keep it open and decide what to do based on the discussion
on the ML (resolve it, improve description if confirmed, duplicate if already
existing, etc.)
> cannot execute a sql like 'insert into A select * from B'
> -
>
> Key: CALCITE-5972
> URL: https://issues.apache.org/jira/browse/CALCITE-5972
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: Moccy.T
>Priority: Blocker
>
> I have put the
> {code:java}
> org.apache.calcite.adapter.csv.CsvScannableTable{code}
> and its related classes (Factory, Reader.., etc.) into my test project.
> When I tried to execute a sql like
> {code:java}
> insert into mysql.calcite_test (select * from csv) {code}
> an NPE was thrown. where `mysql` is recognised by JdbcSchema#Factory. When I
> debugged, I found that the bind method of the dynamically generated Bindable
> code called the JdbcTable's getModifiableCollection() method, which returned
> null directly. the bind method's subsequent call to collection.size() threw
> an NPE. this code is identified by the This code is generated by
> EnumerableTableModify.class.
> After that, I tried to execute
> {code:java}
> insert into mysql.calcite(select * from mysql2.calcite2){code}
> There are two JdbcSchema#Factory recognised data sources here, but I found
> that the code that was eventually generated by
> JdbcToEnumerableConverter.class executed
> {code:java}
> insert into calcite(select * from mysql2.calcite){code}
> in `mysql2` without performing the insert in the `mysql` library.
> Are these issues intentional with calcite or are they bugs?
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5861) ReduceExpressionsRule rules should constant-fold expressions in window bounds
[
https://issues.apache.org/jira/browse/CALCITE-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5861.
---
Resolution: Fixed
Merged via
[{{cbf857e}}|https://github.com/apache/calcite/commit/cbf857e319a695fe96b1f3f5f38a779821c92cb3],
thanks [~mbudiu] for your contribution, and thanks to [~nobigo] and
[~julianhyde] for reviewing!
> ReduceExpressionsRule rules should constant-fold expressions in window bounds
> -
>
> Key: CALCITE-5861
> URL: https://issues.apache.org/jira/browse/CALCITE-5861
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.34.0, 1.35.0
>Reporter: Mihai Budiu
>Assignee: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> This bug is a bit more complicated to describe, and there is a possibility
> that I am doing something wrong.
> Consider the following test that can be added to RelOptRulesTest:
> {code:java}
> @Test void testExpressionPreceding() {
> HepProgramBuilder preBuilder = new HepProgramBuilder();
> preBuilder.addRuleInstance(CoreRules.WINDOW_REDUCE_EXPRESSIONS);
>
> preBuilder.addRuleInstance(CoreRules.PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW);
> HepProgramBuilder builder = new HepProgramBuilder();
> builder.addRuleInstance(CoreRules.PROJECT_REDUCE_EXPRESSIONS);
> HepPlanner hepPlanner = new HepPlanner(builder.build());
> final String sql =
> "select COUNT(*) over (\n"
> + "ORDER BY empno\n"
> + "ROWS BETWEEN 5 + 5 PRECEDING AND 1 PRECEDING) AS w_avg\n"
> + " from emp\n";
> sql(sql)
> .withPre(preBuilder.build())
> .withPlanner(hepPlanner)
> .check();
> }
> {code}
> The plan before looks like this:
> {code:java}
> LogicalProject($0=[$2])
> LogicalWindow(window#0=[window(order by [0] rows between $1 PRECEDING and
> $2 PRECEDING aggs [COUNT()])])
> LogicalProject(EMPNO=[$0], $1=[+(5, 5)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The problem is that the expression "5+5", which appears in the window bounds,
> has not been reduced to a constant by the rule WINDOW_REDUCE_EXPRESSIONS.
> Moreover, the next optimization rule PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW
> has pushed this expression into the LogicalProject. So it appears locally
> that the LogicalWindow no longer has a constant bound, which is required by
> the SQL language spec (it is constant, but that is no longer apparent in the
> query). (At least our code generator is upset by this state of affairs.)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CALCITE-5861) ReduceExpressionsRule rules should constant-fold expressions in window bounds
[
https://issues.apache.org/jira/browse/CALCITE-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-5861:
-
Assignee: Mihai Budiu
> ReduceExpressionsRule rules should constant-fold expressions in window bounds
> -
>
> Key: CALCITE-5861
> URL: https://issues.apache.org/jira/browse/CALCITE-5861
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.34.0, 1.35.0
>Reporter: Mihai Budiu
>Assignee: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> This bug is a bit more complicated to describe, and there is a possibility
> that I am doing something wrong.
> Consider the following test that can be added to RelOptRulesTest:
> {code:java}
> @Test void testExpressionPreceding() {
> HepProgramBuilder preBuilder = new HepProgramBuilder();
> preBuilder.addRuleInstance(CoreRules.WINDOW_REDUCE_EXPRESSIONS);
>
> preBuilder.addRuleInstance(CoreRules.PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW);
> HepProgramBuilder builder = new HepProgramBuilder();
> builder.addRuleInstance(CoreRules.PROJECT_REDUCE_EXPRESSIONS);
> HepPlanner hepPlanner = new HepPlanner(builder.build());
> final String sql =
> "select COUNT(*) over (\n"
> + "ORDER BY empno\n"
> + "ROWS BETWEEN 5 + 5 PRECEDING AND 1 PRECEDING) AS w_avg\n"
> + " from emp\n";
> sql(sql)
> .withPre(preBuilder.build())
> .withPlanner(hepPlanner)
> .check();
> }
> {code}
> The plan before looks like this:
> {code:java}
> LogicalProject($0=[$2])
> LogicalWindow(window#0=[window(order by [0] rows between $1 PRECEDING and
> $2 PRECEDING aggs [COUNT()])])
> LogicalProject(EMPNO=[$0], $1=[+(5, 5)])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> {code}
> The problem is that the expression "5+5", which appears in the window bounds,
> has not been reduced to a constant by the rule WINDOW_REDUCE_EXPRESSIONS.
> Moreover, the next optimization rule PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW
> has pushed this expression into the LogicalProject. So it appears locally
> that the LogicalWindow no longer has a constant bound, which is required by
> the SQL language spec (it is constant, but that is no longer apparent in the
> query). (At least our code generator is upset by this state of affairs.)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5972) cannot execute a sql like 'insert into A select * from B'
[
https://issues.apache.org/jira/browse/CALCITE-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761252#comment-17761252
]
Alessandro Solimando commented on CALCITE-5972:
---
[~moccy] this seems more a question for the dev list rather than a Jira ticket,
I suggest to write to [d...@calcite.apache.org|mailto:d...@calcite.apache.org]
to get some input
> cannot execute a sql like 'insert into A select * from B'
> -
>
> Key: CALCITE-5972
> URL: https://issues.apache.org/jira/browse/CALCITE-5972
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.35.0
>Reporter: Moccy.T
>Priority: Blocker
>
> I have put the
> {code:java}
> org.apache.calcite.adapter.csv.CsvScannableTable{code}
> and its related classes (Factory, Reader.., etc.) into my test project.
> When I tried to execute a sql like
> {code:java}
> insert into mysql.calcite_test (select * from csv) {code}
> an NPE was thrown. where `mysql` is recognised by JdbcSchema#Factory. When I
> debugged, I found that the bind method of the dynamically generated Bindable
> code called the JdbcTable's getModifiableCollection() method, which returned
> null directly. the bind method's subsequent call to collection.size() threw
> an NPE. this code is identified by the This code is generated by
> EnumerableTableModify.class.
> After that, I tried to execute
> {code:java}
> insert into mysql.calcite(select * from mysql2.calcite2){code}
> There are two JdbcSchema#Factory recognised data sources here, but I found
> that the code that was eventually generated by
> JdbcToEnumerableConverter.class executed
> {code:java}
> insert into calcite(select * from mysql2.calcite){code}
> in `mysql2` without performing the insert in the `mysql` library.
> Are these issues intentional with calcite or are they bugs?
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5907) Unexpected boolean expression simplification for And expression
[
https://issues.apache.org/jira/browse/CALCITE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5907.
---
Resolution: Not A Problem
> Unexpected boolean expression simplification for And expression
> ---
>
> Key: CALCITE-5907
> URL: https://issues.apache.org/jira/browse/CALCITE-5907
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Yunhong Zheng
>Priority: Major
> Fix For: 1.36.0
>
>
> As FLINK-27402 shown. If we have a table MyTable(a Int, b Boolean, c
> String). Calcite will not simplify this case ( c is Varchar type while
> SqlLiteral is boolean):
> {code:java}
> SELECT * FROM MyTable WHERE c = true;{code}
> As the logical plan is :
> {code:java}
> LogicalSink(table=[*anonymous_collect$1*], fields=[a, b, c])
> +- LogicalProject(inputs=[0..2])
> +- LogicalFilter(condition=[=($2, true)])
> +- LogicalTableScan(table=[[default_catalog, default_database,
> MyTable]]){code}
> However, Calcite will simplify this case while simplifyAnd :
> {code:java}
> SELECT * FROM MyTable WHERE b = true and c = true;{code}
> As the logical plan is shown below: 'b = true' and 'c = true' both were
> simplified to 'b' and 'c':
> {code:java}
> LogicalSink(table=[*anonymous_collect$1*], fields=[a, b, c])
> +- LogicalProject(inputs=[0..2])
> +- LogicalFilter(condition=[AND($1, $2)])
> +- LogicalTableScan(table=[[default_catalog, default_database,
> MyTable]]){code}
> This may cause error because of filter condition is a Varchar type literal.
>
> After reading Calcite's code, I found that. The logic of
> RexSimplify.implify() and RexSimplify.implifyAnd() is different, where the
> logic of RexSimplify.implifyAnd() is problematic:
> {code:java}
> // Simplify BOOLEAN expressions if possible
> while (term.getKind() == SqlKind.EQUALS) {
> RexCall call = (RexCall) term;
> if (call.getOperands().get(0).isAlwaysTrue()) {
> term = call.getOperands().get(1);
> terms.set(i, term);
> continue;
> } else if (call.getOperands().get(1).isAlwaysTrue()) {
> term = call.getOperands().get(0);
> terms.set(i, term);
> continue;
> }
> break;
> } {code}
> The above code cannot make such a simple judgment, as there may not be an
> implicit conversion to ensure that the types on both sides of the condition
> are consistent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5894) Add SortRemoveRedundantRule to remove redundant sort fields if sort fields contains unique key
[
https://issues.apache.org/jira/browse/CALCITE-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751001#comment-17751001
]
Alessandro Solimando commented on CALCITE-5894:
---
[~libenchao], I see your point but since Calcite it's a general-purpose
library, I am not sure we can make assumptions on the physical implementation,
some can be less mature than others and still benefit from this optimization.
I don't have supporting examples in mind, but removing sort might unlock the
application of further simplifications, it's hard to think of all the
implications but it's reasonable that we are missing some further optimizations
because of (extra) sort operators, at the end of the day we have syntactical
matching.
In summary, I think this rule would be nice to have, even if we decide it's not
part of the standard ruleset.
What do you think?
> Add SortRemoveRedundantRule to remove redundant sort fields if sort fields
> contains unique key
> --
>
> Key: CALCITE-5894
> URL: https://issues.apache.org/jira/browse/CALCITE-5894
> Project: Calcite
> Issue Type: New Feature
>Reporter: JingDas
>Assignee: JingDas
>Priority: Minor
>
> In some scene, Sort fields can be reduct, if sort fields contain unique key
> For Example
> {code:java}
> SELECT name, Emp.salary FROM Emp
> order by empno, ename{code}
> where `empno` is a key, `ename` is redundant since `empno` alone is
> sufficient to determine the order of any two records.
> So the SQL can be optimized as following:
> {code:java}
> SELECT name, Emp.salary FROM Emp
> order by empno{code}
> Sorting is an expensive operation, however. Therefore, it is imperative that
> sorting
> is optimized to avoid unnecessary sort field.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5880) When the association condition of the association subquery is 'is not distinct from', the join condition becomes '=' after decorrelation.
[
https://issues.apache.org/jira/browse/CALCITE-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750789#comment-17750789
]
Alessandro Solimando commented on CALCITE-5880:
---
[~leepb], it seems we need some help reproducing the problem, we can all agree
that the plans you share seem wrong under the context you describe (nullable
fields), but we need to double check the details, then possibly debug it, and
we can't do it if we can't reproduce the issue.
What [~shenlang] tried seems reasonable to me, but it does not seem to be
enough, I think that sharing a (minimal) reproducer would help moving the
discussion forward.
If you have one in your own downstream fork, you might consider even sharing
that if it's complicated to paste the code here.
> When the association condition of the association subquery is 'is not
> distinct from', the join condition becomes '=' after decorrelation.
> -
>
> Key: CALCITE-5880
> URL: https://issues.apache.org/jira/browse/CALCITE-5880
> Project: Calcite
> Issue Type: Bug
>Reporter: libopeng
>Priority: Major
> Attachments: image-2023-08-01-15-20-22-105.png,
> image-2023-08-02-10-15-00-455.png
>
>
> {code:java}
> select EMPNO
> from emp
> where EXISTS (select DEPTNO from dept where emp.EMPNO is not DISTINCT from
> dept.DEPTNO){code}
> before decorrelation
> {code:java}
> LogicalProject(EMPNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($9)])
> LogicalCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{0}])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{}], agg#0=[MIN($0)])
> LogicalProject($f0=[true])
> LogicalFilter(condition=[OR(AND(IS NULL($cor0.EMPNO), IS NULL($0)),
> IS TRUE(=($cor0.EMPNO, $0)))])<--- here is 't1.x is not distinct from
> t2.x'
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> {code}
> after decorrelation
> {code:java}
> LogicalProject(EMPNO=[$0])
> LogicalFilter(condition=[IS NOT NULL($9)])
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4],
> SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8], $f0=[$13])
> LogicalJoin(condition=[AND(=($0, $11), =($9, $12))], joinType=[left])
> <---here is 't1.x=t2.x and (t1.x is null = t2.x is null)'
> LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3],
> HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8], $f9=[false],
> EMPNO0=[$0])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]])
> LogicalAggregate(group=[{0, 1}], agg#0=[MIN($2)])
> LogicalProject(EMPNO=[$2], $f9=[$3], $f0=[true])
> LogicalJoin(condition=[=($2, $0)], joinType=[inner])
> LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
> LogicalProject(EMPNO=[$0], $f9=[false])
> LogicalTableScan(table=[[CATALOG, SALES, EMP]]) {code}
>
> {code:java}
> EMP | DEPT
> +-+ | +-+
> | EMPNO || | DEPTNO |
> +-+ | +-+
> | null | | | null |
>{code}
> expect result
> {code:java}
> EMPNO
> +---+
> | null |
> +---+{code}
> actual result
> {code:java}
> EMPNO
> +---+
> +---+ {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5884) Type Inference rule for ARRAY_TO_STRING is incorrect
[
https://issues.apache.org/jira/browse/CALCITE-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750595#comment-17750595
]
Alessandro Solimando commented on CALCITE-5884:
---
[~mbudiu], I am a bit confused, the title states that the type inference is
incorrect, but the ticket details cover a wrong result in presence of NULL
value for one of the parameters, the two things don't seem to match, I'd rather
say that the ticket is about "ARRAY_TO_STRING should not return NULL when the
null_string parameter is null", am I missing something?
Another part of the discussion (more general than the current ticket) is how
RexToLixTranslator and NullPolicy interact with ReturnType from functions, I'd
like to hear more about that too, I don't have a clear picture either at the
moment.
> Type Inference rule for ARRAY_TO_STRING is incorrect
>
>
> Key: CALCITE-5884
> URL: https://issues.apache.org/jira/browse/CALCITE-5884
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.35.0
>Reporter: Mihai Budiu
>Priority: Trivial
> Labels: pull-request-available
>
> This is the current definition of the function ARRAY_TO_STRING in
> SqlLibraryOperators:
> {code:java}
> /** The "ARRAY_TO_STRING(array, delimiter [, nullText ])" function. */
> @LibraryOperator(libraries = {BIG_QUERY})
> public static final SqlFunction ARRAY_TO_STRING =
> SqlBasicFunction.create(SqlKind.ARRAY_TO_STRING,
> ReturnTypes.VARCHAR_NULLABLE,
> OperandTypes.STRING_ARRAY_CHARACTER_OPTIONAL_CHARACTER);
> {code}
> So the result is nullable if any of the arguments is nullable. However, the
> nullability of the last argument does not influence the result nullabillity:
> a NULL value for the third optional argument will not cause a NULL value to
> be output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5859) Compile-time evaluation of LEFT(NULL, n) should not throw RuntimeException
[
https://issues.apache.org/jira/browse/CALCITE-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5859.
---
Resolution: Fixed
Merged via
[{{7098677}}|https://github.com/apache/calcite/commit/70986773b80f1941a979c793c4c385dffb2da7cd],
thanks [~mbudiu] for your contribution, and [~Chunwei Lei] for the review!
> Compile-time evaluation of LEFT(NULL, n) should not throw RuntimeException
> --
>
> Key: CALCITE-5859
> URL: https://issues.apache.org/jira/browse/CALCITE-5859
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0, 1.35.0
>Reporter: Mihai Budiu
>Assignee: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> {code:sql}
> SELECT LEFT(NULL, 100)
> {code}
> causes this exception:
> {code}
> Caused by: java.lang.RuntimeException: while resolving method 'valueOf[class
> java.lang.String]' in class class java.lang.Void
> at org.apache.calcite.linq4j.tree.Types.lookupMethod(Types.java:318)
> at org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:449)
> at org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:461)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:396)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:339)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (CALCITE-5859) Compile-time evaluation of LEFT(NULL, n) should not throw RuntimeException
[
https://issues.apache.org/jira/browse/CALCITE-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reassigned CALCITE-5859:
-
Assignee: Mihai Budiu
> Compile-time evaluation of LEFT(NULL, n) should not throw RuntimeException
> --
>
> Key: CALCITE-5859
> URL: https://issues.apache.org/jira/browse/CALCITE-5859
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0, 1.35.0
>Reporter: Mihai Budiu
>Assignee: Mihai Budiu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> {code:sql}
> SELECT LEFT(NULL, 100)
> {code}
> causes this exception:
> {code}
> Caused by: java.lang.RuntimeException: while resolving method 'valueOf[class
> java.lang.String]' in class class java.lang.Void
> at org.apache.calcite.linq4j.tree.Types.lookupMethod(Types.java:318)
> at org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:449)
> at org.apache.calcite.linq4j.tree.Expressions.call(Expressions.java:461)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:396)
> at
> org.apache.calcite.adapter.enumerable.EnumUtils.convert(EnumUtils.java:339)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5837) RexUtil#pullFactors output's order should be deterministic even when the RexNode kind is OR
[
https://issues.apache.org/jira/browse/CALCITE-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5837.
---
Resolution: Fixed
Fixed via
[11546c|https://github.com/apache/calcite/commit/11546ce518492c25ed582d47b2ea7b44a56f67d4],
thanks [~shenlang] for your contribution, [~libenchao] and [~jiajunxie] for
your reviews!
> RexUtil#pullFactors output's order should be deterministic even when the
> RexNode kind is OR
> ---
>
> Key: CALCITE-5837
> URL: https://issues.apache.org/jira/browse/CALCITE-5837
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0
>Reporter: LakeShen
>Assignee: LakeShen
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.36.0
>
> Attachments: image-2023-08-02-10-08-06-390.png,
> image-2023-08-02-10-08-41-599.png
>
>
> In my rule, I used the RexUtil.pullFactors to simplify the Filter's condition
> and I tested the rule with TPC-H q19.sql.
> When I print the plan tree with RelOptUtil.toString,sometimes the plan is :
> {code:java}
> LogicalAggregate(group=[{}], revenue=[SUM($0)])
> LogicalProject($f0=[*($5, -(1, $6))])
> LogicalFilter(condition=[AND(=($16, $1), SEARCH($14, Sarg['AIR':CHAR(7),
> 'AIR REG']:CHAR(7)), =($13, 'DELIVER IN PERSON'), OR(AND(=($19, 'Brand#12'),
> SEARCH($22, Sarg['SM BOX':CHAR(7), 'SM CASE', 'SM PACK', 'SM
> PKG':CHAR(7)]:CHAR(7)), >=($4, 1), <=($4, +(1, 10)), SEARCH($21,
> Sarg[[1..5]])), AND(=($19, 'Brand#23'), SEARCH($22, Sarg['MED BAG':CHAR(8),
> 'MED BOX':CHAR(8), 'MED PACK', 'MED PKG':CHAR(8)]:CHAR(8)), >=($4, 10),
> <=($4, +(10, 10)), SEARCH($21, Sarg[[1..10]])), AND(=($19, 'Brand#34'),
> SEARCH($22, Sarg['LG BOX':CHAR(7), 'LG CASE', 'LG PACK', 'LG
> PKG':CHAR(7)]:CHAR(7)), >=($4, 20), <=($4, +(20, 10)), SEARCH($21,
> Sarg[[1..15]]])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[tpch, LINEITEM]])
> LogicalTableScan(table=[[tpch, PART]])
> {code}
> and sometimes the plan is :
> {code:java}
> LogicalAggregate(group=[{}], revenue=[SUM($0)])
> LogicalProject($f0=[*($5, -(1, $6))])
> LogicalFilter(condition=[AND(=($16, $1), =($13, 'DELIVER IN PERSON'),
> SEARCH($14, Sarg['AIR':CHAR(7), 'AIR REG']:CHAR(7)), OR(AND(=($19,
> 'Brand#12'), SEARCH($22, Sarg['SM BOX':CHAR(7), 'SM CASE', 'SM PACK', 'SM
> PKG':CHAR(7)]:CHAR(7)), >=($4, 1), <=($4, +(1, 10)), SEARCH($21,
> Sarg[[1..5]])), AND(=($19, 'Brand#23'), SEARCH($22, Sarg['MED BAG':CHAR(8),
> 'MED BOX':CHAR(8), 'MED PACK', 'MED PKG':CHAR(8)]:CHAR(8)), >=($4, 10),
> <=($4, +(10, 10)), SEARCH($21, Sarg[[1..10]])), AND(=($19, 'Brand#34'),
> SEARCH($22, Sarg['LG BOX':CHAR(7), 'LG CASE', 'LG PACK', 'LG
> PKG':CHAR(7)]:CHAR(7)), >=($4, 20), <=($4, +(20, 10)), SEARCH($21,
> Sarg[[1..15]]])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[tpch, LINEITEM]])
> LogicalTableScan(table=[[tpch, PART]])
> {code}
> The above two SQL Plan semantics are the same, the only difference is that
> the Filter conditions are in different order.
> Although this has no effect on the SQL execution results, it is difficult for
> me to monitor my plan because of the variability of the plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5837) RexUtil#pullFactors output's order should be deterministic even when the RexNode kind is OR
[
https://issues.apache.org/jira/browse/CALCITE-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749941#comment-17749941
]
Alessandro Solimando commented on CALCITE-5837:
---
Just a general tip [~shenlang], you can assign tickets to yourself if you plan
to work on it (and nobody else is), and move it to "In progress"/"Open" to
reflect if you are working on it or not, this helps people following the ticket
and avoids somebody else spending time thinking it's not acted upon.
> RexUtil#pullFactors output's order should be deterministic even when the
> RexNode kind is OR
> ---
>
> Key: CALCITE-5837
> URL: https://issues.apache.org/jira/browse/CALCITE-5837
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0
>Reporter: LakeShen
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.36.0
>
>
> In my rule, I used the RexUtil.pullFactors to simplify the Filter's condition
> and I tested the rule with TPC-H q19.sql.
> When I print the plan tree with RelOptUtil.toString,sometimes the plan is :
> {code:java}
> LogicalAggregate(group=[{}], revenue=[SUM($0)])
> LogicalProject($f0=[*($5, -(1, $6))])
> LogicalFilter(condition=[AND(=($16, $1), SEARCH($14, Sarg['AIR':CHAR(7),
> 'AIR REG']:CHAR(7)), =($13, 'DELIVER IN PERSON'), OR(AND(=($19, 'Brand#12'),
> SEARCH($22, Sarg['SM BOX':CHAR(7), 'SM CASE', 'SM PACK', 'SM
> PKG':CHAR(7)]:CHAR(7)), >=($4, 1), <=($4, +(1, 10)), SEARCH($21,
> Sarg[[1..5]])), AND(=($19, 'Brand#23'), SEARCH($22, Sarg['MED BAG':CHAR(8),
> 'MED BOX':CHAR(8), 'MED PACK', 'MED PKG':CHAR(8)]:CHAR(8)), >=($4, 10),
> <=($4, +(10, 10)), SEARCH($21, Sarg[[1..10]])), AND(=($19, 'Brand#34'),
> SEARCH($22, Sarg['LG BOX':CHAR(7), 'LG CASE', 'LG PACK', 'LG
> PKG':CHAR(7)]:CHAR(7)), >=($4, 20), <=($4, +(20, 10)), SEARCH($21,
> Sarg[[1..15]]])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[tpch, LINEITEM]])
> LogicalTableScan(table=[[tpch, PART]])
> {code}
> and sometimes the plan is :
> {code:java}
> LogicalAggregate(group=[{}], revenue=[SUM($0)])
> LogicalProject($f0=[*($5, -(1, $6))])
> LogicalFilter(condition=[AND(=($16, $1), =($13, 'DELIVER IN PERSON'),
> SEARCH($14, Sarg['AIR':CHAR(7), 'AIR REG']:CHAR(7)), OR(AND(=($19,
> 'Brand#12'), SEARCH($22, Sarg['SM BOX':CHAR(7), 'SM CASE', 'SM PACK', 'SM
> PKG':CHAR(7)]:CHAR(7)), >=($4, 1), <=($4, +(1, 10)), SEARCH($21,
> Sarg[[1..5]])), AND(=($19, 'Brand#23'), SEARCH($22, Sarg['MED BAG':CHAR(8),
> 'MED BOX':CHAR(8), 'MED PACK', 'MED PKG':CHAR(8)]:CHAR(8)), >=($4, 10),
> <=($4, +(10, 10)), SEARCH($21, Sarg[[1..10]])), AND(=($19, 'Brand#34'),
> SEARCH($22, Sarg['LG BOX':CHAR(7), 'LG CASE', 'LG PACK', 'LG
> PKG':CHAR(7)]:CHAR(7)), >=($4, 20), <=($4, +(20, 10)), SEARCH($21,
> Sarg[[1..15]]])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[tpch, LINEITEM]])
> LogicalTableScan(table=[[tpch, PART]])
> {code}
> The above two SQL Plan semantics are the same, the only difference is that
> the Filter conditions are in different order.
> Although this has no effect on the SQL execution results, it is difficult for
> me to monitor my plan because of the variability of the plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5872) JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not null"
[ https://issues.apache.org/jira/browse/CALCITE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749259#comment-17749259 ] Alessandro Solimando edited comment on CALCITE-5872 at 7/31/23 4:22 PM: The query does not look like a minimal reproducer, reducing it further will help you debug it more easily, or alternatively it will make it easier (therefore more probable) that someone else will pick it up and fix it. With the input provided so far, we can't really see what is the problem, or even confirm that there is one, just to check that it would require someone to set things up, which consumes time. Therefore, I also suggest to provide (possibly in the ticket description rather than as a reply comment) at least the original plan (what's translated from SQL directly, more or less), an intermediate plan which still looks OK, and one plan which is not OK (ideally the first one which looks bad to you). For an easy way to activate extra debug logs during query planning you can refer to this talk [https://calcite.apache.org/community/#debugging-planning-issues-using-calcites-built-in-loggers] EDIT: almost forgot, please set also which version of Calcite is affected by this bug by using the "Affects Version/s" Jira field. was (Author: asolimando): The query does not look like a minimal reproducer, reducing it further will help you debug it more easily, or alternatively it will make it easier (therefore more probable) that someone else will pick it up and fix it. With the input provided so far, we can't really see what is the problem, or even confirm that there is one, just to check that it would require someone to set things up, which consumes time. Therefore, I also suggest to provide (possibly in the ticket description rather than as a reply comment) at least the original plan (what's translated from SQL directly, more or less), an intermediate plan which still looks OK, and one plan which is not OK (ideally the first one which looks bad to you). For an easy way to activate extra debug logs during query planning you can refer to this talk [https://calcite.apache.org/community/#debugging-planning-issues-using-calcites-built-in-loggers] > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" > --- > > Key: CALCITE-5872 > URL: https://issues.apache.org/jira/browse/CALCITE-5872 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: grandfisher >Priority: Major > > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" and will always optimize by `VolcanoPlanner` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5872) JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not null"
[ https://issues.apache.org/jira/browse/CALCITE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749259#comment-17749259 ] Alessandro Solimando commented on CALCITE-5872: --- The query does not look like a minimal reproducer, reducing it further will help you debug it more easily, or alternatively it will make it easier (therefore more probable) that someone else will pick it up and fix it. With the input provided so far, we can't really see what is the problem, or even confirm that there is one, just to check that it would require someone to set things up, which consumes time. Therefore, I also suggest to provide (possibly in the ticket description rather than as a reply comment) at least the original plan (what's translated from SQL directly, more or less), an intermediate plan which still looks OK, and one plan which is not OK (ideally the first one which looks bad to you). For an easy way to activate extra debug logs during query planning you can refer to this talk [https://calcite.apache.org/community/#debugging-planning-issues-using-calcites-built-in-loggers] > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" > --- > > Key: CALCITE-5872 > URL: https://issues.apache.org/jira/browse/CALCITE-5872 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: grandfisher >Priority: Major > > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" and will always optimize by `VolcanoPlanner` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5756) Expand ProjectJoinRemoveRule to support inner join removal by using the foreign-unique constraints
[
https://issues.apache.org/jira/browse/CALCITE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748462#comment-17748462
]
Alessandro Solimando commented on CALCITE-5756:
---
I agree with you, [~jingda], it's good to split the contribution into two, the
current scope is too big now.
I suggest to open a separate ticket for the RelMetadataQuery#getForeignKeys
contribution, mark this ticket as "depending on" or "blocked by" (or "relates
to", at the very least), and resume this ticket once the other contribution has
been merged.
WDYT?
> Expand ProjectJoinRemoveRule to support inner join removal by using the
> foreign-unique constraints
> --
>
> Key: CALCITE-5756
> URL: https://issues.apache.org/jira/browse/CALCITE-5756
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: JingDas
>Assignee: JingDas
>Priority: Major
> Labels: pull-request-available
>
> Join elimination is a useful optmize improvement.
> Consider a query that joins the two tables but does not make use of the Dept
> columns:
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp, Dept
> WHERE Emp.deptno = Dept.dno {code}
> Assuming Emp.deptno is the foreign-key and is non-null, Dept.dno is the
> unique-key. The sql above can be rewritten as following. remove the Dept
> table without affecting the resultset.
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp {code}
> Without redundant join elimination, this query execution may perform poorly.
> The optimize improvement is also available in SQL Server, Oracle and
> Snowflake and so on.
> In Calcite, i think that is also useful. The infrastructure that join
> elimination depend on is already available.
> The main steps are as follows:
> 1. Analyse the column used by project, and then split them to left and right
> side.
> 2. Acccording to the project info above and outer join type, bail out in some
> scene.
> 3. Get join info such as join keys.
> 4. For inner join check foreign and unique keys, these may use
> RelMetadataQuery#getForeignKeys(newly add, similar to
> RelMetadataQuery#getUniqueKeys),
> RelOptTable#getReferentialConstraints.
> 5. Check removing side join keys are areColumnsUnique both for outer join and
> inner join.
> 6. If all done, calculate the fianl project and transform.
> Please help me to check the improvement whether is useful or not.
> And i would like to add this improvement to Calcite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5872) JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not null"
[ https://issues.apache.org/jira/browse/CALCITE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747043#comment-17747043 ] Alessandro Solimando edited comment on CALCITE-5872 at 7/25/23 3:17 PM: [~grandfisher], can you add to the description an example of schema, SQL query, and relevant plans to illustrate the problem more clearly? In the past I have worked on a similar issue, the root cause wasn't in the _JoinPushTransitivePredicatesRule,_ but __ in a rule equivalent to [_JoinDeriveIsNotNullFilterRule_|https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rel/rules/JoinDeriveIsNotNullFilterRule.java] which was creating the filter although an equivalent one was already there. was (Author: asolimando): [~grandfisher], can you add to the description an example of schema, SQL query, and relevant plans to illustrate the problem more clearly? > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" > --- > > Key: CALCITE-5872 > URL: https://issues.apache.org/jira/browse/CALCITE-5872 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: grandfisher >Priority: Major > > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" and will always optimize by `VolcanoPlanner` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5872) JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not null"
[ https://issues.apache.org/jira/browse/CALCITE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747043#comment-17747043 ] Alessandro Solimando commented on CALCITE-5872: --- [~grandfisher], can you add to the description an example of schema, SQL query, and relevant plans to illustrate the problem more clearly? > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" > --- > > Key: CALCITE-5872 > URL: https://issues.apache.org/jira/browse/CALCITE-5872 > Project: Calcite > Issue Type: Bug > Components: core >Reporter: grandfisher >Priority: Major > > JoinPushTransitivePredicatesRule will repeatedly insert the condition "is not > null" and will always optimize by `VolcanoPlanner` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5837) After RexUtil.pullFactors method,sometimes the condition order of RexNode is not the same as before
[
https://issues.apache.org/jira/browse/CALCITE-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741893#comment-17741893
]
Alessandro Solimando commented on CALCITE-5837:
---
+1 from me too, we should always strive for fully deterministic methods in such
cases, even if the plans are semantically equivalent we gain nothing from the
filters shuffling
> After RexUtil.pullFactors method,sometimes the condition order of RexNode is
> not the same as before
> ---
>
> Key: CALCITE-5837
> URL: https://issues.apache.org/jira/browse/CALCITE-5837
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0
>Reporter: LakeShen
>Priority: Minor
> Fix For: 1.36.0
>
>
> In my rule, I used the RexUtil.pullFactors to simplify the Filter's condition
> and I tested the rule with TPC-H q19.sql.
> When I print the plan tree with RelOptUtil.toString,sometimes the plan is :
> {code:java}
> LogicalAggregate(group=[{}], revenue=[SUM($0)])
> LogicalProject($f0=[*($5, -(1, $6))])
> LogicalFilter(condition=[AND(=($16, $1), SEARCH($14, Sarg['AIR':CHAR(7),
> 'AIR REG']:CHAR(7)), =($13, 'DELIVER IN PERSON'), OR(AND(=($19, 'Brand#12'),
> SEARCH($22, Sarg['SM BOX':CHAR(7), 'SM CASE', 'SM PACK', 'SM
> PKG':CHAR(7)]:CHAR(7)), >=($4, 1), <=($4, +(1, 10)), SEARCH($21,
> Sarg[[1..5]])), AND(=($19, 'Brand#23'), SEARCH($22, Sarg['MED BAG':CHAR(8),
> 'MED BOX':CHAR(8), 'MED PACK', 'MED PKG':CHAR(8)]:CHAR(8)), >=($4, 10),
> <=($4, +(10, 10)), SEARCH($21, Sarg[[1..10]])), AND(=($19, 'Brand#34'),
> SEARCH($22, Sarg['LG BOX':CHAR(7), 'LG CASE', 'LG PACK', 'LG
> PKG':CHAR(7)]:CHAR(7)), >=($4, 20), <=($4, +(20, 10)), SEARCH($21,
> Sarg[[1..15]]])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[tpch, LINEITEM]])
> LogicalTableScan(table=[[tpch, PART]])
> {code}
> and sometimes the plan is :
> {code:java}
> LogicalAggregate(group=[{}], revenue=[SUM($0)])
> LogicalProject($f0=[*($5, -(1, $6))])
> LogicalFilter(condition=[AND(=($16, $1), =($13, 'DELIVER IN PERSON'),
> SEARCH($14, Sarg['AIR':CHAR(7), 'AIR REG']:CHAR(7)), OR(AND(=($19,
> 'Brand#12'), SEARCH($22, Sarg['SM BOX':CHAR(7), 'SM CASE', 'SM PACK', 'SM
> PKG':CHAR(7)]:CHAR(7)), >=($4, 1), <=($4, +(1, 10)), SEARCH($21,
> Sarg[[1..5]])), AND(=($19, 'Brand#23'), SEARCH($22, Sarg['MED BAG':CHAR(8),
> 'MED BOX':CHAR(8), 'MED PACK', 'MED PKG':CHAR(8)]:CHAR(8)), >=($4, 10),
> <=($4, +(10, 10)), SEARCH($21, Sarg[[1..10]])), AND(=($19, 'Brand#34'),
> SEARCH($22, Sarg['LG BOX':CHAR(7), 'LG CASE', 'LG PACK', 'LG
> PKG':CHAR(7)]:CHAR(7)), >=($4, 20), <=($4, +(20, 10)), SEARCH($21,
> Sarg[[1..15]]])
> LogicalJoin(condition=[true], joinType=[inner])
> LogicalTableScan(table=[[tpch, LINEITEM]])
> LogicalTableScan(table=[[tpch, PART]])
> {code}
> The above two SQL Plan semantics are the same, the only difference is that
> the Filter conditions are in different order.
> Although this has no effect on the SQL execution results, it is difficult for
> me to monitor my plan because of the variability of the plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5756) Expand ProjectJoinRemoveRule to support inner join removal by using the foreign-unique constraints
[
https://issues.apache.org/jira/browse/CALCITE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738176#comment-17738176
]
Alessandro Solimando commented on CALCITE-5756:
---
I have removed the fixVersion previously set to 1.3.5 as the release will
happen soon, and there are still some open discussions around the validity of
the approach
> Expand ProjectJoinRemoveRule to support inner join removal by using the
> foreign-unique constraints
> --
>
> Key: CALCITE-5756
> URL: https://issues.apache.org/jira/browse/CALCITE-5756
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: winds
>Assignee: winds
>Priority: Major
> Labels: pull-request-available
>
> Join elimination is a useful optmize improvement.
> Consider a query that joins the two tables but does not make use of the Dept
> columns:
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp, Dept
> WHERE Emp.deptno = Dept.dno {code}
> Assuming Emp.deptno is the foreign-key and is non-null, Dept.dno is the
> unique-key. The sql above can be rewritten as following. remove the Dept
> table without affecting the resultset.
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp {code}
> Without redundant join elimination, this query execution may perform poorly.
> The optimize improvement is also available in SQL Server, Oracle and
> Snowflake and so on.
> In Calcite, i think that is also useful. The infrastructure that join
> elimination depend on is already available.
> The main steps are as follows:
> 1. Analyse the column used by project, and then split them to left and right
> side.
> 2. Acccording to the project info above and outer join type, bail out in some
> scene.
> 3. Get join info such as join keys.
> 4. For inner join check foreign and unique keys, these may use
> RelMetadataQuery#getForeignKeys(newly add, similar to
> RelMetadataQuery#getUniqueKeys),
> RelOptTable#getReferentialConstraints.
> 5. Check removing side join keys are areColumnsUnique both for outer join and
> inner join.
> 6. If all done, calculate the fianl project and transform.
> Please help me to check the improvement whether is useful or not.
> And i would like to add this improvement to Calcite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5756) Expand ProjectJoinRemoveRule to support inner join removal by using the foreign-unique constraints
[
https://issues.apache.org/jira/browse/CALCITE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5756:
--
Fix Version/s: (was: 1.35.0)
> Expand ProjectJoinRemoveRule to support inner join removal by using the
> foreign-unique constraints
> --
>
> Key: CALCITE-5756
> URL: https://issues.apache.org/jira/browse/CALCITE-5756
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: winds
>Assignee: winds
>Priority: Major
> Labels: pull-request-available
>
> Join elimination is a useful optmize improvement.
> Consider a query that joins the two tables but does not make use of the Dept
> columns:
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp, Dept
> WHERE Emp.deptno = Dept.dno {code}
> Assuming Emp.deptno is the foreign-key and is non-null, Dept.dno is the
> unique-key. The sql above can be rewritten as following. remove the Dept
> table without affecting the resultset.
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp {code}
> Without redundant join elimination, this query execution may perform poorly.
> The optimize improvement is also available in SQL Server, Oracle and
> Snowflake and so on.
> In Calcite, i think that is also useful. The infrastructure that join
> elimination depend on is already available.
> The main steps are as follows:
> 1. Analyse the column used by project, and then split them to left and right
> side.
> 2. Acccording to the project info above and outer join type, bail out in some
> scene.
> 3. Get join info such as join keys.
> 4. For inner join check foreign and unique keys, these may use
> RelMetadataQuery#getForeignKeys(newly add, similar to
> RelMetadataQuery#getUniqueKeys),
> RelOptTable#getReferentialConstraints.
> 5. Check removing side join keys are areColumnsUnique both for outer join and
> inner join.
> 6. If all done, calculate the fianl project and transform.
> Please help me to check the improvement whether is useful or not.
> And i would like to add this improvement to Calcite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5756) Expand ProjectJoinRemoveRule to support inner join removal by using the foreign-unique constraints
[
https://issues.apache.org/jira/browse/CALCITE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5756:
--
Summary: Expand ProjectJoinRemoveRule to support inner join removal by
using the foreign-unique constraints (was: Expand ProjectJoinRemoveRule to
support inner join remove by the foreign-unique constraint in the catalog)
> Expand ProjectJoinRemoveRule to support inner join removal by using the
> foreign-unique constraints
> --
>
> Key: CALCITE-5756
> URL: https://issues.apache.org/jira/browse/CALCITE-5756
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: winds
>Assignee: winds
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.35.0
>
>
> Join elimination is a useful optmize improvement.
> Consider a query that joins the two tables but does not make use of the Dept
> columns:
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp, Dept
> WHERE Emp.deptno = Dept.dno {code}
> Assuming Emp.deptno is the foreign-key and is non-null, Dept.dno is the
> unique-key. The sql above can be rewritten as following. remove the Dept
> table without affecting the resultset.
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp {code}
> Without redundant join elimination, this query execution may perform poorly.
> The optimize improvement is also available in SQL Server, Oracle and
> Snowflake and so on.
> In Calcite, i think that is also useful. The infrastructure that join
> elimination depend on is already available.
> The main steps are as follows:
> 1. Analyse the column used by project, and then split them to left and right
> side.
> 2. Acccording to the project info above and outer join type, bail out in some
> scene.
> 3. Get join info such as join keys.
> 4. For inner join check foreign and unique keys, these may use
> RelMetadataQuery#getForeignKeys(newly add, similar to
> RelMetadataQuery#getUniqueKeys),
> RelOptTable#getReferentialConstraints.
> 5. Check removing side join keys are areColumnsUnique both for outer join and
> inner join.
> 6. If all done, calculate the fianl project and transform.
> Please help me to check the improvement whether is useful or not.
> And i would like to add this improvement to Calcite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5756) Expand ProjectJoinRemoveRule to support inner join remove by the foreign-unique constraint in the catalog
[
https://issues.apache.org/jira/browse/CALCITE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5756:
--
Fix Version/s: 1.35.0
> Expand ProjectJoinRemoveRule to support inner join remove by the
> foreign-unique constraint in the catalog
> -
>
> Key: CALCITE-5756
> URL: https://issues.apache.org/jira/browse/CALCITE-5756
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: winds
>Assignee: winds
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.35.0
>
>
> Join elimination is a useful optmize improvement.
> Consider a query that joins the two tables but does not make use of the Dept
> columns:
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp, Dept
> WHERE Emp.deptno = Dept.dno {code}
> Assuming Emp.deptno is the foreign-key and is non-null, Dept.dno is the
> unique-key. The sql above can be rewritten as following. remove the Dept
> table without affecting the resultset.
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp {code}
> Without redundant join elimination, this query execution may perform poorly.
> The optimize improvement is also available in SQL Server, Oracle and
> Snowflake and so on.
> In Calcite, i think that is also useful. The infrastructure that join
> elimination depend on is already available.
> The main steps are as follows:
> 1. Analyse the column used by project, and then split them to left and right
> side.
> 2. Acccording to the project info above and outer join type, bail out in some
> scene.
> 3. Get join info such as join keys.
> 4. For inner join check foreign and unique keys, these may use
> RelMetadataQuery#getForeignKeys(newly add, similar to
> RelMetadataQuery#getUniqueKeys),
> RelOptTable#getReferentialConstraints.
> 5. Check removing side join keys are areColumnsUnique both for outer join and
> inner join.
> 6. If all done, calculate the fianl project and transform.
> Please help me to check the improvement whether is useful or not.
> And i would like to add this improvement to Calcite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5756) Expand ProjectJoinRemoveRule to support inner join remove by the foreign-unique constraint in the catalog
[
https://issues.apache.org/jira/browse/CALCITE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5756:
--
Priority: Major (was: Minor)
> Expand ProjectJoinRemoveRule to support inner join remove by the
> foreign-unique constraint in the catalog
> -
>
> Key: CALCITE-5756
> URL: https://issues.apache.org/jira/browse/CALCITE-5756
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: winds
>Assignee: winds
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.35.0
>
>
> Join elimination is a useful optmize improvement.
> Consider a query that joins the two tables but does not make use of the Dept
> columns:
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp, Dept
> WHERE Emp.deptno = Dept.dno {code}
> Assuming Emp.deptno is the foreign-key and is non-null, Dept.dno is the
> unique-key. The sql above can be rewritten as following. remove the Dept
> table without affecting the resultset.
> {code:java}
> SELECT Emp.name, Emp.salary
> FROM Emp {code}
> Without redundant join elimination, this query execution may perform poorly.
> The optimize improvement is also available in SQL Server, Oracle and
> Snowflake and so on.
> In Calcite, i think that is also useful. The infrastructure that join
> elimination depend on is already available.
> The main steps are as follows:
> 1. Analyse the column used by project, and then split them to left and right
> side.
> 2. Acccording to the project info above and outer join type, bail out in some
> scene.
> 3. Get join info such as join keys.
> 4. For inner join check foreign and unique keys, these may use
> RelMetadataQuery#getForeignKeys(newly add, similar to
> RelMetadataQuery#getUniqueKeys),
> RelOptTable#getReferentialConstraints.
> 5. Check removing side join keys are areColumnsUnique both for outer join and
> inner join.
> 6. If all done, calculate the fianl project and transform.
> Please help me to check the improvement whether is useful or not.
> And i would like to add this improvement to Calcite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5720) improve FilterJoinRule to push filter into both join sides
[
https://issues.apache.org/jira/browse/CALCITE-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727441#comment-17727441
]
Alessandro Solimando commented on CALCITE-5720:
---
{quote}`SELECT * FROM MyTable1 join MyTable2 ON a1 = a2 AND a1 = 2`
`SELECT * FROM MyTable1, MyTable2 WHERE a1 = a2 AND a1 = 2`
{quote}
The kind of enhancements proposed in this ticket looks a lot like "inferring
predicates from one side of a join and pushing to the other side", and
therefore, ideally, I would not make _FilterJoinRule_ more complex by adding
support for more inference patterns, but rather use
_JoinPushTransitivePredicatesRule_ and eventually fix its existing issues if
needed (even with some heuristics at first, like limiting the number of times
the rule can be applied to avoid the cases where the fix-point cannot be
reached).
On the other hand, the line separating the two rules is a bit fuzzy, and I also
understand that in the short-term enhancing _FilterJoinRule_ brings more
immediate value, although it will make the situation more complex in the
mid/long run.
In summary, I only have a slight preference but not a very strong opinion, I
see value on both ways of moving forward.
The discussion is interesting IMO and it's quite general, about how to deal
with short vs long term benefits, maybe in the ML we could gather more valuable
opinions.
> improve FilterJoinRule to push filter into both join sides
> --
>
> Key: CALCITE-5720
> URL: https://issues.apache.org/jira/browse/CALCITE-5720
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Reporter: Suhan Mao
>Priority: Major
>
> In flink source code,
> [FlinkFilterJoinRule|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/FlinkFilterJoinRule.java],
> the FilterJoinRule is improved to handle more cases:
> for the above filter of inner/left/right join or the join condition of inner
> join, the predicate which field references are all from one side join
> condition can be pushed into another join side. Such as:
> `SELECT * FROM MyTable1 join MyTable2 ON a1 = a2 AND a1 = 2`
> `SELECT * FROM MyTable1, MyTable2 WHERE a1 = a2 AND a1 = 2 `
>
> I would like to add the improvement to exist FilterJoinRule class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5720) improve FilterJoinRule to push filter into both join sides
[ https://issues.apache.org/jira/browse/CALCITE-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725690#comment-17725690 ] Alessandro Solimando commented on CALCITE-5720: --- [~maosuhan], thanks for sharing! [~libenchao], from past experience working with the Hive's equivalent rule ([HiveJoinPushTransitivePredicatesRule|https://github.com/apache/hive/blob/7583142cbffcb3958a546a9aaa15700bbc243df9/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java]), the rule is very sensitive to how effective _RexSimplify_ is to reach a fixpoint. When simplification is missing some opportunities, it can lead to infinite loops where predicates are pushed and pulled from one side of the join to the other without (and OOMs). You can see an example in the Calcite meetup talk of last March (https://calcite.apache.org/community/#debugging-planning-issues-using-calcites-built-in-loggers), especially slides 44 to 51 from here: https://www.slideshare.net/StamatisZampetakis/debugging-planning-issues-using-calcites-builtin-loggers This said, the rule is powerful and useful, I don't know what are the reasons behind the exclusion from the core ruleset, maybe you want to bring this discussion to the ML? > improve FilterJoinRule to push filter into both join sides > -- > > Key: CALCITE-5720 > URL: https://issues.apache.org/jira/browse/CALCITE-5720 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Suhan Mao >Priority: Major > > In flink source code, > [FlinkFilterJoinRule|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/FlinkFilterJoinRule.java], > the FilterJoinRule is improved to handle more cases: > for the above filter of inner/left/right join or the join condition of inner > join, the predicate which field references are all from one side join > condition can be pushed into another join side. Such as: > `SELECT * FROM MyTable1 join MyTable2 ON a1 = a2 AND a1 = 2` > `SELECT * FROM MyTable1, MyTable2 WHERE a1 = a2 AND a1 = 2 ` > > I would like to add the improvement to exist FilterJoinRule class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5720) improve FilterJoinRule to push filter into both join sides
[ https://issues.apache.org/jira/browse/CALCITE-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725277#comment-17725277 ] Alessandro Solimando commented on CALCITE-5720: --- Have you checked [JoinPushTransitivePredicatesRule|https://github.com/apache/calcite/blob/2dba40e7a0a5651eac5a30d9e0a72f178bd9bff2/core/src/main/java/org/apache/calcite/rel/rules/JoinPushTransitivePredicatesRule.java]? > improve FilterJoinRule to push filter into both join sides > -- > > Key: CALCITE-5720 > URL: https://issues.apache.org/jira/browse/CALCITE-5720 > Project: Calcite > Issue Type: Improvement > Components: core >Reporter: Suhan Mao >Priority: Major > > In flink source code, > [FlinkFilterJoinRule|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/FlinkFilterJoinRule.java], > the FilterJoinRule is improved to handle more cases: > for the above filter of inner/left/right join or the join condition of inner > join, the predicate which field references are all from one side join > condition can be pushed into another join side. Such as: > `SELECT * FROM MyTable1 join MyTable2 ON a1 = a2 AND a1 = 2` > `SELECT * FROM MyTable1, MyTable2 WHERE a1 = a2 AND a1 = 2 ` > > I would like to add the improvement to exist FilterJoinRule class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5639) RexSimplify should remove IS NOT NULL check when LIKE comparison is present
[
https://issues.apache.org/jira/browse/CALCITE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5639:
--
Component/s: core
> RexSimplify should remove IS NOT NULL check when LIKE comparison is present
> ---
>
> Key: CALCITE-5639
> URL: https://issues.apache.org/jira/browse/CALCITE-5639
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.34.0
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.35.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Consider query like
> {code:java}
> select r_reason_desc from reason
> where r_reason_desc is not null and r_reason_desc like '%child%'
> {code}
> where "r_reason_desc" is a nullable field.
> When RexSimplify::simplifyFilterPredicates is called on that conjunction of
> expressions, expression is not simplified, meaning that redundant "is not
> null" check is not removed.
> In the same time, if query like
> {code:java}
> select r_reason_desc from reason
> where r_reason_desc is not null and r_reason_desc >= 'A'{code}
> is passed to optimizer, redundant "is not null" check is eliminated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5639) RexSimplify should remove IS NOT NULL check when LIKE comparison is present
[
https://issues.apache.org/jira/browse/CALCITE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5639.
---
Fix Version/s: 1.35.0
Resolution: Fixed
Fixed via
[0305067|https://github.com/apache/calcite/commit/03050674594152ea785af69517f33960d7e27dd5],
thanks [~lchistov1987] for the patch and [~julianhyde] [~Chunwei Lei] and
[~libenchao] for your comments and reivews!
> RexSimplify should remove IS NOT NULL check when LIKE comparison is present
> ---
>
> Key: CALCITE-5639
> URL: https://issues.apache.org/jira/browse/CALCITE-5639
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.34.0
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.35.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Consider query like
> {code:java}
> select r_reason_desc from reason
> where r_reason_desc is not null and r_reason_desc like '%child%'
> {code}
> where "r_reason_desc" is a nullable field.
> When RexSimplify::simplifyFilterPredicates is called on that conjunction of
> expressions, expression is not simplified, meaning that redundant "is not
> null" check is not removed.
> In the same time, if query like
> {code:java}
> select r_reason_desc from reason
> where r_reason_desc is not null and r_reason_desc >= 'A'{code}
> is passed to optimizer, redundant "is not null" check is eliminated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5639) RexSimplify should remove IS NOT NULL check when LIKE comparison is present
[
https://issues.apache.org/jira/browse/CALCITE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712316#comment-17712316
]
Alessandro Solimando commented on CALCITE-5639:
---
I have reviewed the PR and LGTM module few minor things, but if someone else
feels like taking a look of course they are welcome
> RexSimplify should remove IS NOT NULL check when LIKE comparison is present
> ---
>
> Key: CALCITE-5639
> URL: https://issues.apache.org/jira/browse/CALCITE-5639
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.34.0
>Reporter: Leonid Chistov
>Assignee: Leonid Chistov
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Consider query like
> {code:java}
> select r_reason_desc from reason
> where r_reason_desc is not null and r_reason_desc like '%child%'
> {code}
> where "r_reason_desc" is a nullable field.
> When RexSimplify::simplifyFilterPredicates is called on that conjunction of
> expressions, expression is not simplified, meaning that redundant "is not
> null" check is not removed.
> In the same time, if query like
> {code:java}
> select r_reason_desc from reason
> where r_reason_desc is not null and r_reason_desc >= 'A'{code}
> is passed to optimizer, redundant "is not null" check is eliminated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687461#comment-17687461 ] Alessandro Solimando commented on CALCITE-5501: --- I think that will help with CALCITE-3517, but the issue we have here would still exist, since each test (and associated golden file) would still be shared. As I was saying in a previous comment, the race condition will be rarer but still happening if the same test is executed at the same time by both classes. This said I think it might be a good improvement and worth trying, since these kind of tests are widespread and time consuming, so if it helps with CALCITE-3517 we will make our tests run way faster! > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Fix For: 1.34.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando updated CALCITE-5501: -- Fix Version/s: 1.34.0 > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Fix For: 1.34.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[
https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5501.
---
Resolution: Fixed
Fixed via
[{{03a0b9d}}|https://github.com/apache/calcite/commit/03a0b9d93774ad6d0a7c0909fa33bc9640a3a02a],
thanks [~clesaec] for your review, and to [~zabetak] and [~julianhyde] for the
discussion and for reporting the issue.
> SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in
> Jenkins CI
> -
>
> Key: CALCITE-5501
> URL: https://issues.apache.org/jira/browse/CALCITE-5501
> Project: Calcite
> Issue Type: Bug
> Components: tests
>Affects Versions: 1.32.0
>Reporter: Stamatis Zampetakis
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Sample runs that failed due to
> SqlToRelConverterTest.checkActualAndReferenceFiles:
> *
> https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console
> *
> https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (CALCITE-5253) NATURAL join and USING should fail if join columns are not unique - expression validation partially broken
[
https://issues.apache.org/jira/browse/CALCITE-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando reopened CALCITE-5253:
---
> NATURAL join and USING should fail if join columns are not unique -
> expression validation partially broken
> --
>
> Key: CALCITE-5253
> URL: https://issues.apache.org/jira/browse/CALCITE-5253
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.31.0
>Reporter: Evgeny Stanilovsky
>Assignee: Evgeny Stanilovsky
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.33.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Such issue is not possible for now:
> {code:java}
> CREATE TABLE t1(a INTEGER, b INTEGER, c INTEGER);
> CREATE TABLE t2(b INTEGER, c INTEGER, d INTEGER);
> CREATE TABLE t3(c INTEGER, d INTEGER, e INTEGER);
> SELECT t1.c, t2.d, t1.b, t1.a, t3.e FROM t1 natural join t2 natural join t3;
> {code}
> cause:
> {noformat}
> SqlValidatorException: Column name 'C' in NATURAL join or USING clause is not
> unique on one side of join
> {noformat}
> This validation is correct for example for case :
>
> {noformat}
> select e.ename, d.name from dept as d natural join (select ename, sal as
> deptno, deptno from emp) as e
> {noformat}
> but fails as described above.
> Was broken by:
> [1] https://issues.apache.org/jira/browse/CALCITE-5171
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-5253) NATURAL join and USING should fail if join columns are not unique - expression validation partially broken
[
https://issues.apache.org/jira/browse/CALCITE-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando resolved CALCITE-5253.
---
Resolution: Fixed
> NATURAL join and USING should fail if join columns are not unique -
> expression validation partially broken
> --
>
> Key: CALCITE-5253
> URL: https://issues.apache.org/jira/browse/CALCITE-5253
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.31.0
>Reporter: Evgeny Stanilovsky
>Assignee: Evgeny Stanilovsky
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.33.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Such issue is not possible for now:
> {code:java}
> CREATE TABLE t1(a INTEGER, b INTEGER, c INTEGER);
> CREATE TABLE t2(b INTEGER, c INTEGER, d INTEGER);
> CREATE TABLE t3(c INTEGER, d INTEGER, e INTEGER);
> SELECT t1.c, t2.d, t1.b, t1.a, t3.e FROM t1 natural join t2 natural join t3;
> {code}
> cause:
> {noformat}
> SqlValidatorException: Column name 'C' in NATURAL join or USING clause is not
> unique on one side of join
> {noformat}
> This validation is correct for example for case :
>
> {noformat}
> select e.ename, d.name from dept as d natural join (select ename, sal as
> deptno, deptno from emp) as e
> {noformat}
> but fails as described above.
> Was broken by:
> [1] https://issues.apache.org/jira/browse/CALCITE-5171
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5512) why not support FOR SYSTEM_VERSION AS OF
[ https://issues.apache.org/jira/browse/CALCITE-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682908#comment-17682908 ] Alessandro Solimando commented on CALCITE-5512: --- Removing the fixVersion as it's to be set to the next release version once the ticket is resolved, so 1.26.0 won't anyway be viable as it's been released already. > why not support FOR SYSTEM_VERSION AS OF > - > > Key: CALCITE-5512 > URL: https://issues.apache.org/jira/browse/CALCITE-5512 > Project: Calcite > Issue Type: Wish > Components: core >Affects Versions: 1.26.0 >Reporter: YangFei >Priority: Major > > hive support FOR SYSTEM_VERSION AS OF/FOR SYSTEM_TIME AS OF, why calcite > Parser.jj not support FOR SYSTEM_VERSION AS OF -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5512) why not support FOR SYSTEM_VERSION AS OF
[ https://issues.apache.org/jira/browse/CALCITE-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando updated CALCITE-5512: -- Fix Version/s: (was: 1.26.0) > why not support FOR SYSTEM_VERSION AS OF > - > > Key: CALCITE-5512 > URL: https://issues.apache.org/jira/browse/CALCITE-5512 > Project: Calcite > Issue Type: Wish > Components: core >Affects Versions: 1.26.0 >Reporter: YangFei >Priority: Major > > hive support FOR SYSTEM_VERSION AS OF/FOR SYSTEM_TIME AS OF, why calcite > Parser.jj not support FOR SYSTEM_VERSION AS OF -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-5505) JavaCC warns about missing LOOKAHEAD directives in Parser.jj
[
https://issues.apache.org/jira/browse/CALCITE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated CALCITE-5505:
--
Priority: Minor (was: Major)
> JavaCC warns about missing LOOKAHEAD directives in Parser.jj
>
>
> Key: CALCITE-5505
> URL: https://issues.apache.org/jira/browse/CALCITE-5505
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.32.0
>Reporter: Alessandro Solimando
>Priority: Minor
>
> JavaCC is reporting several warnings for insufficient lookahead for
> Parser.jj, an example of log displaying the warnings is here:
> [https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/consoleFull]
>
> Here is the relevant extract from the aforementioned log file:
> {noformat}
> > Task :core:javaCCMain
> Java Compiler Compiler Version 4.0 (Parser Generator)
> (type "javacc" with no arguments for help)
> Reading from file
> /home/jenkins/jenkins-agent/workspace/Calcite_Calcite-sonar_main/core/build/fmpp/fmppMain/javacc/Parser.jj
> . . .
> Warning: Output directory
> "/home/jenkins/jenkins-agent/workspace/Calcite_Calcite-sonar_main/core/build/javacc/javaCCMain/org/apache/calcite/sql/parser/impl"
> does not exist. Creating the directory.
> Note: UNICODE_INPUT option is specified. Please make sure you create the
> parser/lexer using a Reader with the correct character encoding.
> Warning: Choice conflict involving two expansions at
> line 4930, column 5 and line 4956, column 5 respectively.
> A common prefix is: "MICROSECOND"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4931, column 5 and line 4956, column 5 respectively.
> A common prefix is: "MILLISECOND"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4936, column 5 and line 4956, column 5 respectively.
> A common prefix is: "DOW"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4937, column 5 and line 4956, column 5 respectively.
> A common prefix is: "DOY"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4938, column 5 and line 4956, column 5 respectively.
> A common prefix is: "ISODOW"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4939, column 5 and line 4956, column 5 respectively.
> A common prefix is: "ISOYEAR"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4940, column 5 and line 4956, column 5 respectively.
> A common prefix is: "WEEK"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4950, column 5 and line 4956, column 5 respectively.
> A common prefix is: "QUARTER"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4952, column 5 and line 4956, column 5 respectively.
> A common prefix is: "EPOCH"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4953, column 5 and line 4956, column 5 respectively.
> A common prefix is: "DECADE"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4954, column 5 and line 4956, column 5 respectively.
> A common prefix is: "CENTURY"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 4955, column 5 and line 4956, column 5 respectively.
> A common prefix is: "MILLENNIUM"
> Consider using a lookahead of 2 for earlier expansion.
> Warning: Choice conflict involving two expansions at
> line 6549, column 9 and line 6551, column 9 respectively.
> A common prefix is: "WEEK" "("
> Consider using a lookahead of 3 or more for earlier expansion.
> File "TokenMgrError.java" does not exist. Will create one.
> File "ParseException.java" does not exist. Will create one.
> File "Token.java" does not exist. Will create one.
> File "SimpleCharStream.java" does not exist. Will create one.
> Parser generated with 0 errors and 14 warnings.{noformat}
> We are probably missing one or two LOOKAHEAD directives in the parser.
--
This
[jira] [Created] (CALCITE-5505) JavaCC warns about missing LOOKAHEAD directives in Parser.jj
Alessandro Solimando created CALCITE-5505:
-
Summary: JavaCC warns about missing LOOKAHEAD directives in
Parser.jj
Key: CALCITE-5505
URL: https://issues.apache.org/jira/browse/CALCITE-5505
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.32.0
Reporter: Alessandro Solimando
JavaCC is reporting several warnings for insufficient lookahead for Parser.jj,
an example of log displaying the warnings is here:
[https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/consoleFull]
Here is the relevant extract from the aforementioned log file:
{noformat}
> Task :core:javaCCMain
Java Compiler Compiler Version 4.0 (Parser Generator)
(type "javacc" with no arguments for help)
Reading from file
/home/jenkins/jenkins-agent/workspace/Calcite_Calcite-sonar_main/core/build/fmpp/fmppMain/javacc/Parser.jj
. . .
Warning: Output directory
"/home/jenkins/jenkins-agent/workspace/Calcite_Calcite-sonar_main/core/build/javacc/javaCCMain/org/apache/calcite/sql/parser/impl"
does not exist. Creating the directory.
Note: UNICODE_INPUT option is specified. Please make sure you create the
parser/lexer using a Reader with the correct character encoding.
Warning: Choice conflict involving two expansions at
line 4930, column 5 and line 4956, column 5 respectively.
A common prefix is: "MICROSECOND"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4931, column 5 and line 4956, column 5 respectively.
A common prefix is: "MILLISECOND"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4936, column 5 and line 4956, column 5 respectively.
A common prefix is: "DOW"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4937, column 5 and line 4956, column 5 respectively.
A common prefix is: "DOY"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4938, column 5 and line 4956, column 5 respectively.
A common prefix is: "ISODOW"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4939, column 5 and line 4956, column 5 respectively.
A common prefix is: "ISOYEAR"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4940, column 5 and line 4956, column 5 respectively.
A common prefix is: "WEEK"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4950, column 5 and line 4956, column 5 respectively.
A common prefix is: "QUARTER"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4952, column 5 and line 4956, column 5 respectively.
A common prefix is: "EPOCH"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4953, column 5 and line 4956, column 5 respectively.
A common prefix is: "DECADE"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4954, column 5 and line 4956, column 5 respectively.
A common prefix is: "CENTURY"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 4955, column 5 and line 4956, column 5 respectively.
A common prefix is: "MILLENNIUM"
Consider using a lookahead of 2 for earlier expansion.
Warning: Choice conflict involving two expansions at
line 6549, column 9 and line 6551, column 9 respectively.
A common prefix is: "WEEK" "("
Consider using a lookahead of 3 or more for earlier expansion.
File "TokenMgrError.java" does not exist. Will create one.
File "ParseException.java" does not exist. Will create one.
File "Token.java" does not exist. Will create one.
File "SimpleCharStream.java" does not exist. Will create one.
Parser generated with 0 errors and 14 warnings.{noformat}
We are probably missing one or two LOOKAHEAD directives in the parser.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681535#comment-17681535 ] Alessandro Solimando edited comment on CALCITE-5501 at 1/28/23 7:53 AM: >From your description I think you are referring to CALCITE-3517. I think that even writing the file once per test class, we would still be having a race condition, because one of the two test classes might be writing when the other one is comparing. Of course it would limit sensibly the chances of the race condition to happen (once per class vs once per test) but we would still need a lock on _SqlToRelConverterTest.xml_ which remains shared, so I think that my proposed patch would still be needed. I therefore suggest to get this in (once the release is over) and tackle the performance improvement separately as part of CALCITE-3517, if that makes sense to you too. was (Author: asolimando): >From your description I think you are referring to CALCITE-3517. I think that even writing the file once per test class, we would still be having a race condition, because one of the two test classes might be writing when the other one is comparing. Of course it would limit sensibly the chances of the race condition to happen (once per class vs once per test) but we would still need a lock on _SqlToRelConverterTest.xml_ which remains shared, so I think that my propose patch would still be needed. I therefore suggest to get this in (once the release is over) and tackle the performance improvement separately as part of CALCITE-3517, if that makes sense to you too. > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681535#comment-17681535 ] Alessandro Solimando commented on CALCITE-5501: --- >From your description I think you are referring to CALCITE-3517. I think that even writing the file once per test class, we would still be having a race condition, because one of the two test classes might be writing when the other one is comparing. Of course it would limit sensibly the chances of the race condition to happen (once per class vs once per test) but we would still need a lock on _SqlToRelConverterTest.xml_ which remains shared, so I think that my propose patch would still be needed. I therefore suggest to get this in (once the release is over) and tackle the performance improvement separately as part of CALCITE-3517, if that makes sense to you too. > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681468#comment-17681468 ] Alessandro Solimando edited comment on CALCITE-5501 at 1/28/23 7:42 AM: [@ResourceLock|https://junit.org/junit5/docs/current/api/org.junit.jupiter.api/org/junit/jupiter/api/parallel/ResourceLock.html] from JUnit5 to the rescue here, I have updated my [PR|https://github.com/apache/calcite/pull/3049] by making _SqlToRelConverterExtendedTest_ and _SqlToRelConverterTest_ classes run sequentially, while keeping their execution w.r.t. the other classes and for the test cases within them, parallel. Running "./gradlew assemble --no-build-cache cleanTest :core:test" repeatedly on the main branch and with the PR on my machine did not provide noticeable difference in the total execution time, because the tests from the two classes are still executed in parallel with all the rest of the tests, the two classes simply don't step on each other toes anymore. I will check in the ML if there is still time to fix this before the release, or if it will wait for main to be open again. was (Author: asolimando): [@ResourceLock|https://junit.org/junit5/docs/current/api/org.junit.jupiter.api/org/junit/jupiter/api/parallel/ResourceLock.html] from JUnit5 to the rescue here, I have updated my [PR|https://github.com/apache/calcite/pull/3049] by making _SqlToRelConverterExtendedTest_ and _SqlToRelConverterTest_ classes __ run sequentially, while __ keeping their execution w.r.t. the other classes and for the test cases within them, parallel. Running "./gradlew assemble --no-build-cache cleanTest :core:test" repeatedly on the main branch and with the PR on my machine did not provide noticeable difference in the total execution time, because the tests from the two classes are still executed in parallel with all the rest of the tests, the two classes simply don't step on each other toes anymore. I will check in the ML if there is still time to fix this before the release, or if it will wait for main to be open again. > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681468#comment-17681468 ] Alessandro Solimando commented on CALCITE-5501: --- [@ResourceLock|https://junit.org/junit5/docs/current/api/org.junit.jupiter.api/org/junit/jupiter/api/parallel/ResourceLock.html] from JUnit5 to the rescue here, I have updated my [PR|https://github.com/apache/calcite/pull/3049] by making _SqlToRelConverterExtendedTest_ and _SqlToRelConverterTest_ classes __ run sequentially, while __ keeping their execution w.r.t. the other classes and for the test cases within them, parallel. Running "./gradlew assemble --no-build-cache cleanTest :core:test" repeatedly on the main branch and with the PR on my machine did not provide noticeable difference in the total execution time, because the tests from the two classes are still executed in parallel with all the rest of the tests, the two classes simply don't step on each other toes anymore. I will check in the ML if there is still time to fix this before the release, or if it will wait for main to be open again. > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando reassigned CALCITE-5501: - Assignee: Alessandro Solimando > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[
https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681456#comment-17681456
]
Alessandro Solimando commented on CALCITE-5501:
---
Unfortunately _SqlToRelConverterExtendedTest_ needs a _DiffRepository_ too it
seems, so my proposed solution does not work.
I managed to reproduce locally as follows:
{code:java}
./gradlew assemble --no-build-cache cleanTest :core:test --tests
"org.apache.calcite.test.SqlToRelConverterExtendedTest" --tests
"org.apache.calcite.test.SqlToRelConverterTest"{code}
I am trying to see if I can make those two classes run sequentially (either at
the JUnit5 level or via gradle), worst case we go for the extra XML file as
[~clesaec] suggested, if there are no objections.
> SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in
> Jenkins CI
> -
>
> Key: CALCITE-5501
> URL: https://issues.apache.org/jira/browse/CALCITE-5501
> Project: Calcite
> Issue Type: Bug
> Components: tests
>Affects Versions: 1.32.0
>Reporter: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Sample runs that failed due to
> SqlToRelConverterTest.checkActualAndReferenceFiles:
> *
> https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console
> *
> https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[
https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681409#comment-17681409
]
Alessandro Solimando commented on CALCITE-5501:
---
I haven't had time to look in-depth yet but I had the same idea of
[~julianhyde], it smells of a race condition on the output file.
IIUC _SqlToRelConverterExtendedTest_ just tests the "Rel -> Json -> Rel"
round-trip for each test in the base class {_}SqlToRelConverterTest{_}, is it
possible that we don't need to write to an XML at all in the subclass?
In that case, simply removing the _DiffRepository_ from the test fixture in
_SqlToRelConverterExtendedTest_ would do, like this:
{code:java}
@Override public SqlToRelFixture fixture() {
return SqlToRelFixture.DEFAULT;
}{code}
It would be nice to avoid to duplicate the output file because it enforces a
bit more manual work for contributors of new tests (on the bright side the
current machinery at least guarantees that they would stay in sync and
up-to-date).
Of course if there is no other way and what I proposed does not work (or breaks
assumptions over _SqlToRelConverterExtendedTest_ that I missed), I am good with
at least recovering from the actual flaky state, and we can improve further in
a second moment.
> SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in
> Jenkins CI
> -
>
> Key: CALCITE-5501
> URL: https://issues.apache.org/jira/browse/CALCITE-5501
> Project: Calcite
> Issue Type: Bug
> Components: tests
>Affects Versions: 1.32.0
>Reporter: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Sample runs that failed due to
> SqlToRelConverterTest.checkActualAndReferenceFiles:
> *
> https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console
> *
> https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-5501) SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in Jenkins CI
[ https://issues.apache.org/jira/browse/CALCITE-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681023#comment-17681023 ] Alessandro Solimando commented on CALCITE-5501: --- This seems to be happening only on the SonarCloud part, right? Of course we need to fix this even if that's the case, just wondering what can cause the difference in behaviour w.r.t. "regular" CI, understanding that might hint us into the right direction. > SqlToRelConverterTest.checkActualAndReferenceFiles fails intermittently in > Jenkins CI > - > > Key: CALCITE-5501 > URL: https://issues.apache.org/jira/browse/CALCITE-5501 > Project: Calcite > Issue Type: Bug > Components: tests >Affects Versions: 1.32.0 >Reporter: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Sample runs that failed due to > SqlToRelConverterTest.checkActualAndReferenceFiles: > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/15/console > * > https://ci-builds.apache.org/job/Calcite/job/Calcite-sonar/job/main/18/console > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-2989) Use ISO-8601 calendar when converting between java.sql types and unix timestamps
[
https://issues.apache.org/jira/browse/CALCITE-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644259#comment-17644259
]
Alessandro Solimando commented on CALCITE-2989:
---
I have added some comments too, the PR LGTM already, and I agree we can merge
after the requested changes.
> Use ISO-8601 calendar when converting between java.sql types and unix
> timestamps
>
>
> Key: CALCITE-2989
> URL: https://issues.apache.org/jira/browse/CALCITE-2989
> Project: Calcite
> Issue Type: Bug
> Components: avatica
>Reporter: vinoyang
>Assignee: Gregory Hart
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Converting java.sql types to unix timestamps requires extra steps to also
> convert to the correct calendar. Unix timestamps should follow the proleptic
> Gregorian calendar as defined by ISO-8601. Java uses the standard Gregorian
> calendar for java.sql types and switches to the Julian calendar for dates
> before the Gregorian shift.
> If we uses avatica's {{DateTimeUtils}} the dates less than 2299161 will cause
> an error result in Flink table/sql , test code :
> {code:java}
> testAllApis(
> "1500-04-30 12:00:00".cast(Types.SQL_TIMESTAMP),
> "'1500-04-30 12:00:00'.cast(SQL_TIMESTAMP)",
> "CAST('1500-04-30 12:00:00' AS TIMESTAMP)",
> "1500-04-30 12:00:00.0")
> {code}
> result :
> {code:java}
> Expected :1500-04-30 12:00:00.0
> Actual :1500-04-20 12:00:00.0
> {code}
> another case is here :
> https://issues.apache.org/jira/browse/FLINK-11935
> I find a key code snippet has been removed in CALCITE-1884 which caused this
> issue :
> {code:java}
> if (j < 2299161) {
>j = day + (153 * m + 2) / 5 + 365 * y + y / 4 - 32083;
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-5414) Use DateTimeUtils to correctly convert between java.sql types and unix timestamps
[ https://issues.apache.org/jira/browse/CALCITE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando updated CALCITE-5414: -- Affects Version/s: 1.32.0 > Use DateTimeUtils to correctly convert between java.sql types and unix > timestamps > - > > Key: CALCITE-5414 > URL: https://issues.apache.org/jira/browse/CALCITE-5414 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.32.0 >Reporter: Gregory Hart >Assignee: Gregory Hart >Priority: Major > Labels: pull-request-available, pull-requests-available > Time Spent: 20m > Remaining Estimate: 0h > > Converting java.sql types to unix timestamps requires extra steps to also > convert to the correct calendar. Unix timestamps should follow the proleptic > Gregorian calendar as defined by ISO-8601. Java uses the standard Gregorian > calendar for java.sql types and switches to the Julian calendar for dates > before the Gregorian shift. > The DateTimeUtils class in Avatica correctly handles the calendar > conversions. Calcite should use those methods since its own methods do not > currently convert between calendars. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-5406) Support the SELECT DISTINCT ON statement for PostgreSQL dialect
[
https://issues.apache.org/jira/browse/CALCITE-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641209#comment-17641209
]
Alessandro Solimando edited comment on CALCITE-5406 at 11/30/22 10:11 AM:
--
Quoting from [https://www.geekytidbits.com/postgres-distinct-on/]:
{noformat}
There are several ways this could be done as well including using a WHERE IN
clause (assuming there is a single identifier that could be used), a LATERAL
join or a WINDOW function.{noformat}
I haven't given much thought to it but it seems that DISTINCT ON is syntactic
sugar and that we can express it via a Window function, for the runtime part we
might be good by doing this translation somewhere like {_}SqlToRelConverter{_}?
For quickly trying out what you might come up with, you can find
_SqlToRelConverterTest_ where you can add tests (refer to the javadoc to see
how to update the associated XML files where results are stored).
was (Author: asolimando):
Quoting from [https://www.geekytidbits.com/postgres-distinct-on/]:
{noformat}
There are several ways this could be done as well including using a WHERE IN
clause (assuming there is a single identifier that could be used), a LATERAL
join or a WINDOW function.{noformat}
I haven't given much thought but it seems to be syntactic sugar and that we can
express the following using a Window function, for the runtime part we might be
good by doing this translation somewhere like {_}SqlToRelConverter{_}?.
For quickly trying out what you might come up with, you can find
_SqlToRelConverterTest_ where you can add tests (refer to the javadoc to see
how to update the associated XML files where results are stored).
> Support the SELECT DISTINCT ON statement for PostgreSQL dialect
> ---
>
> Key: CALCITE-5406
> URL: https://issues.apache.org/jira/browse/CALCITE-5406
> Project: Calcite
> Issue Type: Bug
>Affects Versions: 1.32.0
>Reporter: Zhengqiang Duan
>Assignee: Zhengqiang Duan
>Priority: Major
>
> I use the following sql statement to test with PostgreSQL dialect:
> {code:java}
> select distinct on (user_id) * from t_order order by user_id, order_id; {code}
> But a sql parsing exception is thrown.
> {code:java}
> org.apache.calcite.sql.parser.SqlParseException: Encountered "on" at line 1,
> column 17.
> Was expecting one of:
> "ABS" ...
> "ARRAY" ...
> "AVG" ...
> "CARDINALITY" ...
> "CASE" ...
> "CAST" ...
> "CEIL" ...
> "CEILING" ...
> "CHAR" ...
> "CHAR_LENGTH" ...
> "CHARACTER_LENGTH" ...
> "CLASSIFIER" ...
> "COALESCE" ...
> "COLLECT" ...
> "CONVERT" ...
> "COUNT" ...
> "COVAR_POP" ...
> "COVAR_SAMP" ...
> "CUME_DIST" ...
> "CURRENT" ...
> "CURRENT_CATALOG" ...
> "CURRENT_DATE" ...
> "CURRENT_DEFAULT_TRANSFORM_GROUP" ...
> "CURRENT_PATH" ...
> "CURRENT_ROLE" ...
> "CURRENT_SCHEMA" ...
> "CURRENT_TIME" ...
> "CURRENT_TIMESTAMP" ...
> "CURRENT_USER" ...
> "CURSOR" ...
> "DATE" ...
> "DENSE_RANK" ...
> "ELEMENT" ...
> "EVERY" ...
> "EXISTS" ...
> "EXP" ...
> "EXTRACT" ...
> "FALSE" ...
> "FIRST_VALUE" ...
> "FLOOR" ...
> "FUSION" ...
> "GROUPING" ...
> "HOUR" ...
> "INTERSECTION" ...
> "INTERVAL" ...
> "JSON_ARRAY" ...
> "JSON_ARRAYAGG" ...
> "JSON_EXISTS" ...
> "JSON_OBJECT" ...
> "JSON_OBJECTAGG" ...
> "JSON_QUERY" ...
> "JSON_VALUE" ...
> "LAG" ...
> "LAST_VALUE" ...
> "LEAD" ...
> "LEFT" ...
> "LN" ...
> "LOCALTIME" ...
> "LOCALTIMESTAMP" ...
> "LOWER" ...
> "MATCH_NUMBER" ...
> "MAX" ...
> "MIN" ...
> "MINUTE" ...
> "MOD" ...
> "MONTH" ...
> "MULTISET" ...
> "NEW" ...
> "NEXT" ...
> "NOT" ...
> "NTH_VALUE" ...
> "NTILE" ...
> "NULL" ...
> "NULLIF" ...
> "OCTET_LENGTH" ...
> "OVERLAY" ...
> "PERCENTILE_CONT" ...
> "PERCENTILE_DISC" ...
> "PERCENT_RANK" ...
> "PERIOD" ...
> "POSITION" ...
> "POWER" ...
> "PREV" ...
> "RANK" ...
> "REGR_COUNT" ...
> "REGR_SXX" ...
> "REGR_SYY" ...
> "RIGHT" ...
> "ROW" ...
> "ROW_NUMBER" ...
> "RUNNING" ...
> "SECOND" ...
> "SESSION_USER" ...
> "SOME" ...
> "SPECIFIC" ...
> "SQRT" ...
> "STDDEV_POP" ...
> "STDDEV_SAMP" ...
> "SUBSTRING" ...
> "SUM" ...
> "SYSTEM_USER" ...
> "TIME" ...
> "TIMESTAMP" ...
> "TRANSLATE" ...
> "TRIM" ...
> "TRUE" ...
> "TRUNCATE" ...
> "UNIQUE" ...
> "UNKNOWN" ...
> "UPPER" ...
> "USER" ...
> "VAR_POP" ...
> "VAR_SAMP" ...
> "YEAR" ...
> ...
> ...
> ...
> ...
> ...
> ...
>
