[jira] [Created] (CALCITE-3061) Query with WITH clause fails when alias is the same as the table with rolled up column
Volodymyr Vysotskyi created CALCITE-3061: Summary: Query with WITH clause fails when alias is the same as the table with rolled up column Key: CALCITE-3061 URL: https://issues.apache.org/jira/browse/CALCITE-3061 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.20.0 {{SqlValidatorImpl}} incorrectly resolves aliases for CTE when checking rolled up columns when a table with the same name is present. It may be observed with the following case: we have table {{emp_r}} which contains rolled up column {{slackingmin}}. The following query: {code:sql} with emp_r as (select 1 as slackingmin) select slackingmin from emp_r {code} should pass since it uses CTE, not table. Error for this case: {noformat} org.apache.calcite.sql.validate.SqlValidatorException: Rolled up column 'SLACKINGMIN' is not allowed in SELECT at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772) at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4825) at org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3438) at org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp(SqlValidatorImpl.java:3447) at org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUpInSelectList(SqlValidatorImpl.java:3373) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3361) at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954) at org.apache.calcite.sql.validate.WithNamespace.validateImpl(WithNamespace.java:57) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateWith(SqlValidatorImpl.java:3787) at org.apache.calcite.sql.SqlWith.validate(SqlWith.java:71) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:929) at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:633) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2745) RexSimplify fails with ClassCastException when the expression contains comparisons of operands with different types
Volodymyr Vysotskyi created CALCITE-2745: Summary: RexSimplify fails with ClassCastException when the expression contains comparisons of operands with different types Key: CALCITE-2745 URL: https://issues.apache.org/jira/browse/CALCITE-2745 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde {{RexSimplify}} fails with {{ClassCastException}} when the expression contains predicates with comparisons of operands with different types. Test to reproduce this issue (placed into {{JdbcAdapterTest}}): {code:sql} @Test public void testSeveralTypesComparison() { CalciteAssert.model(JdbcTest.FOODMART_MODEL) .query("SELECT \"full_name\" FROM \"employee\" WHERE " + "\"employee_id\" = '1' and \"employee_id\" > 0") .returns("full_name=Sheri Nowmer\n"); } {code} It fails with the error: {noformat} Caused by: java.lang.ClassCastException: org.apache.calcite.util.NlsString cannot be cast to java.math.BigDecimal at java.math.BigDecimal.compareTo(BigDecimal.java:220) at org.apache.calcite.rex.RexSimplify.processRange(RexSimplify.java:1780) at org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1242) at org.apache.calcite.rex.RexSimplify.simplifyAnd2ForUnknownAsFalse(RexSimplify.java:1109) at org.apache.calcite.rex.RexSimplify.simplifyAnds(RexSimplify.java:386) at org.apache.calcite.rex.RexSimplify.simplifyFilterPredicates(RexSimplify.java:2028) at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1048) at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1037) at org.apache.calcite.rel.rules.PushProjector.convertProject(PushProjector.java:382) at org.apache.calcite.rel.rules.ProjectFilterTransposeRule.onMatch(ProjectFilterTransposeRule.java:104) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212) ... 41 more {noformat} When the query has any of the {{AND}} operands, or when {{ProjectFilterTransposeRule}} is not applied it is passed. Regarding the comparison of different data types, SQL spec says the following: {quote}8.2 Syntax Rules 3) The declared types of the corresponding fields of the two s shall be comparable. {quote} But it also allows implicit casts from numeric types to chars (6.13 ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2497) Update Janino version to 3.0.9
Volodymyr Vysotskyi created CALCITE-2497: Summary: Update Janino version to 3.0.9 Key: CALCITE-2497 URL: https://issues.apache.org/jira/browse/CALCITE-2497 Project: Calcite Issue Type: Task Affects Versions: 1.17.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.18.0 Update Janino version to 3.0.9 where was fixed [https://github.com/janino-compiler/janino/issues/47], and remove workaround made in CALCITE-2261 because of this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2422) Query with unnest of column from nested subquery fails when dynamic table is used
Volodymyr Vysotskyi created CALCITE-2422: Summary: Query with unnest of column from nested subquery fails when dynamic table is used Key: CALCITE-2422 URL: https://issues.apache.org/jira/browse/CALCITE-2422 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde This test: {code:java} @Test public void testStarDynamicSchemaUnnestNestedSubquery() { String sql3 = "select * \n" + "from (select * from SALES.CUSTOMER) as t1,\n" + "unnest(t1.fake_col) as t2"; sql(sql3).with(getTesterWithDynamicTable()).ok(); } {code} placed in {{SqlToRelConverterTest}} fails with {{AssertionError}}: {noformat} java.lang.AssertionError at org.apache.calcite.sql.SqlUnnestOperator.inferReturnType(SqlUnnestOperator.java:81) at org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:437) at org.apache.calcite.sql.validate.UnnestNamespace.validateImpl(UnnestNamespace.java:67) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3032) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3017) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateJoin(SqlValidatorImpl.java:3069) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3026) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3275) at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:968) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:944) at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:919) at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:629) at org.apache.calcite.test.SqlToRelTestBase$TesterImpl.convertSqlToRel(SqlToRelTestBase.java:642) at org.apache.calcite.test.SqlToRelTestBase$TesterImpl.assertConvertsTo(SqlToRelTestBase.java:754) at org.apache.calcite.test.SqlToRelConverterTest$Sql.convertsTo(SqlToRelConverterTest.java:2905) at org.apache.calcite.test.SqlToRelConverterTest$Sql.ok(SqlToRelConverterTest.java:2897) at org.apache.calcite.test.SqlToRelConverterTest.testStarDynamicSchemaUnnestNestedSubquery(SqlToRelConverterTest.java:2530) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at
[jira] [Created] (CALCITE-2415) Evaluation of predicate "(A or B) and C" is failing for MongoDB adapter
Volodymyr Vysotskyi created CALCITE-2415: Summary: Evaluation of predicate "(A or B) and C" is failing for MongoDB adapter Key: CALCITE-2415 URL: https://issues.apache.org/jira/browse/CALCITE-2415 Project: Calcite Issue Type: Bug Components: elasticsearch-adapter, geode, mongodb Reporter: Andrei Sereda Assignee: Michael Mior h2. Queries that are working {code} select count(*) from "elastic" where _MAP['foo'] in ('1') and true; select count(*) from "elastic" where _MAP['foo'] ='1' and true; select count(*) from "elastic" where _MAP['foo'] in ('1', '2'); {code} h2. Queries that are failing {code} select count(*) from "elastic" where _MAP['foo'] in ('1', '2') and true; select count(*) from "elastic" where true and _MAP['foo'] in ('1', '2'); select count(*) from "elastic" where (_MAP['foo'] ='1' or _MAP['foo'] = '2') and true; {code} h2. Elastic Adapter Exception {quote} java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'), =(ITEM($0, 'foo'), '2')) at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$ Translator.translateMatch2(ElasticsearchFilter.java:234) at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$ Translator.translateAnd(ElasticsearchFilter.java:158) at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$ Translator.translateOr(ElasticsearchFilter.java:115) at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$ Translator.translateMatch(ElasticsearchFilter.java:101) at org.apache.calcite.adapter.elasticsearch.ElasticsearchFilter$ Translator.access$000(ElasticsearchFilter.java:87) at org.apache.calcite.adapter.elasticsearch. ElasticsearchFilter.implement(ElasticsearchFilter.java:80) at org.apache.calcite.adapter.elasticsearch. ElasticsearchRel$Implementor.visitChild(ElasticsearchRel.java:53) at org.apache.calcite.adapter.elasticsearch. ElasticsearchToEnumerableConverter.implement(ElasticsearchToEnumerableConve rter.java:71) at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor. visitChild(EnumerableRelImplementor.java:98) at org.apache.calcite.adapter.enumerable. EnumerableAggregate.implement(EnumerableAggregate.java:106) {quote} h2. Mongo Adapter Exception {quote} select * from "mongo"."col" where _MAP['foo'] in ('1', '2') and true; java.lang.AssertionError: cannot translate OR(=(ITEM($0, 'foo'), '1'), =(ITEM($0, 'foo'), '2')) at org.apache.calcite.adapter.mongodb.MongoFilter$ Translator.translateMatch2(MongoFilter.java:181) at org.apache.calcite.adapter.mongodb.MongoFilter$ Translator.translateAnd(MongoFilter.java:116) at org.apache.calcite.adapter.mongodb.MongoFilter$ Translator.translateOr(MongoFilter.java:98) at org.apache.calcite.adapter.mongodb.MongoFilter$ Translator.translateMatch(MongoFilter.java:91) at org.apache.calcite.adapter.mongodb.MongoFilter$ Translator.access$000(MongoFilter.java:77) at org.apache.calcite.adapter.mongodb.MongoFilter.implement( MongoFilter.java:72) at org.apache.calcite.adapter.mongodb.MongoRel$Implementor. visitChild(MongoRel.java:51) at org.apache.calcite.adapter.mongodb.MongoToEnumerableConverter. implement(MongoToEnumerableConverter.java:84) at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor. implementRoot(EnumerableRelImplementor.java:103) at org.apache.calcite.adapter.enumerable.EnumerableInterpretable. toBindable(EnumerableInterpretable.java:92) {quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2400) Allow standards-compliant column ordering for NATURAL JOIN and JOIN USING when dynamic tables are used
Volodymyr Vysotskyi created CALCITE-2400: Summary: Allow standards-compliant column ordering for NATURAL JOIN and JOIN USING when dynamic tables are used Key: CALCITE-2400 URL: https://issues.apache.org/jira/browse/CALCITE-2400 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde In CALCITE-2227 was allowed standards-compliant column ordering for NATURAL JOIN and JOIN USING for non-dynamic tables. In CALCITE-2392 was prevented permutation of columns for dynamic tables to avoid IOBE. The aim of this Jira is to implement standards-compliant column ordering for NATURAL JOIN and JOIN USING for dynamic tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2391) Query with aggregation of UNNEST column fails with ClassCastException
Volodymyr Vysotskyi created CALCITE-2391: Summary: Query with aggregation of UNNEST column fails with ClassCastException Key: CALCITE-2391 URL: https://issues.apache.org/jira/browse/CALCITE-2391 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde This test: {code:java} @Test public void testAggUnnestColumn() { CalciteAssert.hr() .query("select count(d.\"name\") as c\n" + "from \"hr\".\"depts\" as d,\n" + " UNNEST(d.\"employees\") as e") .returnsUnordered( "C=3"); } {code} added to {{JdbcTest}} class fails with {{ClassCastException}} {noformat} Caused by: java.lang.ClassCastException: org.apache.calcite.sql.validate.AggregatingSelectScope cannot be cast to org.apache.calcite.sql.validate.ListScope at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookupExp(SqlToRelConverter.java:4293) at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3614) at org.apache.calcite.sql2rel.SqlToRelConverter.access$2300(SqlToRelConverter.java:213) at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4692) at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3990) at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:344) at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4554) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2090) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:1957) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2033) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:642) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:623) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3058) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:559) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:264) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:783) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:642) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:612) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:232) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:609) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) ... 28 more {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2389) Drill query fails when file name contains semicolon
Volodymyr Vysotskyi created CALCITE-2389: Summary: Drill query fails when file name contains semicolon Key: CALCITE-2389 URL: https://issues.apache.org/jira/browse/CALCITE-2389 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Queries on the tables which contain semicolon in the name: {code:sql} select * from dfs.`/tmp/q:0` {code} fails with error: {noformat} org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: java.net.URISyntaxException: Relative path in absolute URI: q:0 SQL Query null [Error Id: 34fafee1-8fbe-4fe0-9fcb-ddcc926bb192 on user515050-pc:31010] (java.lang.IllegalArgumentException) java.net.URISyntaxException: Relative path in absolute URI: q:0 org.apache.hadoop.fs.Path.initialize():205 org.apache.hadoop.fs.Path.():171 org.apache.hadoop.fs.Path.():93 org.apache.hadoop.fs.Globber.glob():253 org.apache.hadoop.fs.FileSystem.globStatus():1655 org.apache.drill.exec.store.dfs.DrillFileSystem.globStatus():547 org.apache.drill.exec.store.dfs.FileSelection.create():274 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create():607 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create():408 org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry():96 org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get():90 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable():561 org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.getTable():132 org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable():82 org.apache.calcite.jdbc.CalciteSchema.getTable():257 org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntryFrom():1022 org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntry():979 org.apache.calcite.prepare.CalciteCatalogReader.getTable():123 org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable():650 org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom():260 org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3219 org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60 org.apache.calcite.sql.validate.AbstractNamespace.validate():84 org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():947 org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():928 org.apache.calcite.sql.SqlSelect.validate():226 org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression():903 org.apache.calcite.sql.validate.SqlValidatorImpl.validate():613 org.apache.drill.exec.planner.sql.SqlConverter.validate():190 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():644 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():204 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():176 org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():145 org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():83 org.apache.drill.exec.work.foreman.Foreman.runSQL():567 org.apache.drill.exec.work.foreman.Foreman.run():266 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():748 Caused By (java.net.URISyntaxException) Relative path in absolute URI: q:0 java.net.URI.checkPath():1823 java.net.URI.():745 org.apache.hadoop.fs.Path.initialize():202 org.apache.hadoop.fs.Path.():171 org.apache.hadoop.fs.Path.():93 org.apache.hadoop.fs.Globber.glob():253 org.apache.hadoop.fs.FileSystem.globStatus():1655 org.apache.drill.exec.store.dfs.DrillFileSystem.globStatus():547 org.apache.drill.exec.store.dfs.FileSelection.create():274 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create():607 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create():408 org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry():96 org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get():90 org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable():561 org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.getTable():132 org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable():82 org.apache.calcite.jdbc.CalciteSchema.getTable():257 org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntryFrom():1022 org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntry():979 org.apache.calcite.prepare.CalciteCatalogReader.getTable():123 org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable():650
[jira] [Created] (CALCITE-2379) CVSS dependency-check-maven fails for calcite-spark module
Volodymyr Vysotskyi created CALCITE-2379: Summary: CVSS dependency-check-maven fails for calcite-spark module Key: CALCITE-2379 URL: https://issues.apache.org/jira/browse/CALCITE-2379 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Check for vulnerabilities among dependencies fails for {{calcite-spark}} module. Output for "{{mvn install -Ppedantic -DskipTests=true}}": {noformat} One or more dependencies were identified with known vulnerabilities in Calcite Spark: jackson-databind-2.9.4.jar (com.fasterxml.jackson.core:jackson-databind:2.9.4, cpe:/a:fasterxml:jackson-databind:2.9.4, cpe:/a:fasterxml:jackson:2.9.4) : CVE-2018-7489 protobuf-java-3.3.0.jar (com.google.protobuf:protobuf-java:3.3.0, cpe:/a:google:protobuf:3.3.0) : CVE-2015-5237 commons-beanutils-core-1.8.0.jar (commons-beanutils:commons-beanutils-core:1.8.0, cpe:/a:apache:commons_beanutils:1.8.0) : CVE-2014-0114 commons-beanutils-1.7.0.jar (commons-beanutils:commons-beanutils:1.7.0, cpe:/a:apache:commons_beanutils:1.7.0) : CVE-2014-0114 commons-httpclient-3.1.jar (commons-httpclient:commons-httpclient:3.1, cpe:/a:apache:commons-httpclient:3.1, cpe:/a:apache:httpclient:3.1) : CVE-2015-5262, CVE-2014-3577 javax.annotation-api-1.2.jar (cpe:/a:oracle:glassfish:1.2, javax.annotation:javax.annotation-api:1.2) : CVE-2015-2808, CVE-2013-2566 mail-1.4.7.jar (cpe:/a:mail_project:mail:1.4.7, javax.mail:mail:1.4.7) : CVE-2015-9097 validation-api-1.1.0.Final.jar (cpe:/a:bean_project:bean:7.x-1.1::~~~drupal~~, javax.validation:validation-api:1.1.0.Final) : CVE-2013-4499 jaxb-api-2.2.2.jar (cpe:/a:fish:fish:2.2.2, cpe:/a:oracle:glassfish:2.2.2, javax.xml.bind:jaxb-api:2.2.2) : CVE-2015-2808, CVE-2013-2566 pyrolite-4.13.jar (cpe:/a:pickle:pickle:4.13, net.razorvine:pyrolite:4.13) : CVE-2007-1100 py4j-0.10.4.jar (cpe:/a:python:python:0.10.4, cpe:/a:python_software_foundation:python:0.10.4, net.sf.py4j:py4j:0.10.4) : CVE-2018-130, CVE-2017-18207, CVE-2017-17522, CVE-2017-1000158, CVE-2016-5699, CVE-2016-5636, CVE-2016-1494, CVE-2016-0772, CVE-2015-5652, CVE-2014-7185, CVE-2014-3539, CVE-2013-7440, CVE-2013-7338, CVE-2012-1150, CVE-2012-0845, CVE-2011-4940, CVE-2010-3492, CVE-2008-5983, CVE-2008-3143, CVE-2008-3142, CVE-2008-2315, CVE-2008-1887, CVE-2008-1721, CVE-2008-1679, CVE-2007-4559, CVE-2006-1542, CVE-2002-1119 avro-mapred-1.7.7-hadoop2.jar (cpe:/a:apache:hadoop:1.7.7, org.apache.avro:avro-mapred:1.7.7) : CVE-2017-3162, CVE-2017-3161, CVE-2016-5001 curator-recipes-2.6.0.jar (cpe:/a:apache:zookeeper:2.6.0, org.apache.curator:curator-recipes:2.6.0) : CVE-2016-5017, CVE-2014-0085 api-util-1.0.0-M20.jar (cpe:/a:apache:directory_ldap_api:1.0.0.m30, org.apache.directory.api:api-util:1.0.0-M20) : CVE-2015-3250 xbean-asm5-shaded-4.4.jar (cpe:/a:apache:geronimo:4.4) : CVE-2008-0732 zookeeper-3.4.6.jar (cpe:/a:apache:zookeeper:3.4.6, org.apache.zookeeper:zookeeper:3.4.6) : CVE-2017-5637, CVE-2016-5017, CVE-2014-0085 jackson-xc-1.9.13.jar (cpe:/a:fasterxml:jackson-databind:1.9.13, cpe:/a:fasterxml:jackson:1.9.13, org.codehaus.jackson:jackson-xc:1.9.13) : CVE-2018-5968, CVE-2017-17485 jetty-http-9.2.19.v20160908.jar (cpe:/a:eclipse:jetty:9.2.19.v20160908, cpe:/a:jetty:jetty:9.2.19.v20160908, org.eclipse.jetty:jetty-http:9.2.19.v20160908) : CVE-2017-9735 jetty-util-6.1.26.jar (cpe:/a:jetty:jetty:6.1.26, cpe:/a:mortbay:jetty:6.1.26, cpe:/a:mortbay_jetty:jetty:6.1.26, org.mortbay.jetty:jetty-util:6.1.26) : CVE-2011-4461 unused-1.0.0.jar (cpe:/a:apache:spark:1.0.0, org.spark-project.spark:unused:1.0.0) : CVE-2017-7678 xz-1.0.jar (cpe:/a:tukaani:xz:1.0, org.tukaani:xz:1.0) : CVE-2015-4035 serializer-2.7.1.jar (cpe:/a:apache:xalan-java:2.7.1, xalan:serializer:2.7.1) : CVE-2014-0107 xalan-2.7.1.jar (cpe:/a:apache:xalan-java:2.7.1, xalan:xalan:2.7.1) : CVE-2014-0107 xercesImpl-2.9.1.jar (cpe:/a:apache:xerces2_java:2.9.1, xerces:xercesImpl:2.9.1) : CVE-2012-0881 htrace-core-3.1.0-incubating.jar/META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml (com.fasterxml.jackson.core:jackson-databind:2.4.0, cpe:/a:fasterxml:jackson-databind:2.4.0, cpe:/a:fasterxml:jackson:2.4.0) : CVE-2018-7489, CVE-2018-5968, CVE-2017-7525, CVE-2017-17485, CVE-2017-15095 spark-core_2.10-2.2.0.jar/META-INF/maven/org.eclipse.jetty/jetty-plus/pom.xml (cpe:/a:eclipse:jetty:9.3.11.v20160721, cpe:/a:jetty:jetty:9.3.11.v20160721, org.eclipse.jetty:jetty-plus:9.3.11.v20160721) : CVE-2017-9735 {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2368) Fix misc.iq and scalar.iq quidem unit tests failures on Windows
Volodymyr Vysotskyi created CALCITE-2368: Summary: Fix misc.iq and scalar.iq quidem unit tests failures on Windows Key: CALCITE-2368 URL: https://issues.apache.org/jira/browse/CALCITE-2368 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde misc.iq and scalar.iq quidem unit tests fail on Windows but pass on Linux. *Problem description* Files with quidem tests are discovered in runtime. Some tests, such as misc.iq or scalar.iq require setting up {{THREAD_EXPAND}} to true (see {{CoreQuidemTest.testSqlMisc()}} and {{CoreQuidemTest.testSqlScalar()}} methods). Methods, which should be called for concrete quidem tests to set required variables are discovered using the filename. Code replaces character {{'/'}} in the filepath, but in the case of Windows in the path was path separator char for Windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2337) Release Calcite 1.17.0
Volodymyr Vysotskyi created CALCITE-2337: Summary: Release Calcite 1.17.0 Key: CALCITE-2337 URL: https://issues.apache.org/jira/browse/CALCITE-2337 Project: Calcite Issue Type: Task Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Fix For: 1.17.0 Release Calcite 1.17.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2329) Enhance SubQueryRemoveRule to rewrite IN operator with the constant from the left side more optimally
Volodymyr Vysotskyi created CALCITE-2329: Summary: Enhance SubQueryRemoveRule to rewrite IN operator with the constant from the left side more optimally Key: CALCITE-2329 URL: https://issues.apache.org/jira/browse/CALCITE-2329 Project: Calcite Issue Type: Improvement Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Currently, for queries like this: {code:sql} select sal from emp where 2 IN (select deptno from dept) {code} SubQueryRemoveRule rules expand query plan in such a way: {noformat} LogicalProject(SAL=[$5]) LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8]) LogicalJoin(condition=[=(2, $9)], joinType=[inner]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) LogicalProject(DEPTNO=[$0]) LogicalTableScan(table=[[CATALOG, SALES, DEPT]]) {noformat} Since join condition is actually a filter condition, it will be pushed into the filter during further planning stages and join with the true condition is left. But these types of the queries may be rewritten in the same way as EXISTS queries: {code:sql} select sal from emp where EXISTS (select deptno from dept where deptno=2) {code} with the more optimal plan: {noformat} LogicalProject(SAL=[$5]) LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8]) LogicalJoin(condition=[true], joinType=[inner]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) LogicalAggregate(group=[{0}]) LogicalProject(i=[true]) LogicalFilter(condition=[=($0, 2)]) LogicalTableScan(table=[[CATALOG, SALES, DEPT]]) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2223) ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule
Volodymyr Vysotskyi created CALCITE-2223: Summary: ProjectMergeRule is infinitely matched when is applied after ProjectReduceExpressionsRule Key: CALCITE-2223 URL: https://issues.apache.org/jira/browse/CALCITE-2223 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde For queries like this: {code:sql} select t1.f from (select cast(f as int) f, f from (select cast(f as int) f from (values('1')) t(f))) as t1 {code} OOM is thrown when {{ProjectMergeRule}} is applied before applying {{ProjectReduceExpressionsRule}} in VolcanoPlanner. A simple test to reproduce this issue: {code:java} @Test public void testOomProjectMergeRule() { RelBuilder relBuilder = RelBuilder.create(RelBuilderTest.config().build()); RelNode relNode = relBuilder .values(new String[]{"f"}, "1") .project( relBuilder.alias( relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER), "f")) .project( relBuilder.alias( relBuilder.cast(relBuilder.field(0), SqlTypeName.INTEGER), "f0"), relBuilder.alias(relBuilder.field(0), "f")) .project( relBuilder.alias(relBuilder.field(0), "f")) .build(); RelOptPlanner planner = relNode.getCluster().getPlanner(); RuleSet ruleSet = RuleSets.ofList( ReduceExpressionsRule.PROJECT_INSTANCE, new ProjectMergeRuleWithLongerName(), EnumerableRules.ENUMERABLE_PROJECT_RULE, EnumerableRules.ENUMERABLE_VALUES_RULE); Program program = Programs.of(ruleSet); RelTraitSet toTraits = relNode.getCluster().traitSet() .replace(0, EnumerableConvention.INSTANCE); RelNode output = program.run(planner, relNode, toTraits, ImmutableList.of(), ImmutableList.of()); // check for output } /** * ProjectMergeRule inheritor which has * class name longer than ProjectReduceExpressionsRule class name. * * It is needed for RuleQueue.popMatch() method * to apply this rule before ProjectReduceExpressionsRule. */ private static class ProjectMergeRuleWithLongerName extends ProjectMergeRule { public ProjectMergeRuleWithLongerName() { super(true, RelFactories.LOGICAL_BUILDER); } } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2215) Avatica - Remove unused shaded jars
Volodymyr Vysotskyi created CALCITE-2215: Summary: Avatica - Remove unused shaded jars Key: CALCITE-2215 URL: https://issues.apache.org/jira/browse/CALCITE-2215 Project: Calcite Issue Type: Task Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Currently shaded/core/pom.xml, standalone-server/pom.xml and tck/pom.xml pom files shade jars which are not used anywhere in the project: - com.google.common - org.apache.http - org.apache.commons It may cause problems for projects which shade Avatica and have dependencies to specified libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2201) Allow passing custom RelBuilder into RelDecorrelator and RelStructuredTypeFlattener
Volodymyr Vysotskyi created CALCITE-2201: Summary: Allow passing custom RelBuilder into RelDecorrelator and RelStructuredTypeFlattener Key: CALCITE-2201 URL: https://issues.apache.org/jira/browse/CALCITE-2201 Project: Calcite Issue Type: Task Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde When {{RelDecorrelator.decorrelateQuery()}} method is called, {{RelDecorrelator}} instance with {{RelFactories.LOGICAL_BUILDER}} is created. We should modify this method to allow usage of custom {{RelBuilder}}. Also, {{RelStructuredTypeFlattener}} class should be modified in a similar way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2166) Cumulative cost of RelSubset.best RelNode is increased after calling RelSubset.propagateCostImprovements() for input RelNodes
Volodymyr Vysotskyi created CALCITE-2166: Summary: Cumulative cost of RelSubset.best RelNode is increased after calling RelSubset.propagateCostImprovements() for input RelNodes Key: CALCITE-2166 URL: https://issues.apache.org/jira/browse/CALCITE-2166 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.15.0 Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde After calling {{RelSubset.propagateCostImprovements()}} cumulative cost of {{RelSubset.best}} {{RelNode}} may be increased due to the increase of the non-cumulative cost caused by changing of input best {{RelNode}}. To observe this issue, add this code: {code:java} if (subset.best != null) { RelOptCost bestCost = getCost(subset.best, RelMetadataQuery.instance()); if (!subset.bestCost.equals(bestCost)) { throw new AssertionError( "relSubset [" + subset.getDescription() + "] has wrong best cost " + subset.bestCost + ". Correct cost is " + bestCost); } } {code} into {{VolcanoPlanner.validate()}} method (line 907). List of unit tests which fail with this check: {noformat} Failed tests: MaterializationTest.testJoinMaterializationUKFK9:1823->checkMaterialize:198->checkMaterialize:205->checkThatMaterialize:233 relSubset [rel#226287:Subset#8.ENUMERABLE.[]] has wrong best cost {221.5 rows, 128.25 cpu, 0.0 io}. Correct cost is {233.0 rows, 178.0 cpu, 0.0 io} ScannableTableTest.testPFPushDownProjectFilterAggregateNested:279 relSubset [rel#12950:Subset#5.ENUMERABLE.[]] has wrong best cost {63.8 rows, 62.308 cpu, 0.0 io}. Correct cost is {70.4 rows, 60.404 cpu, 0.0 io} ScannableTableTest.testPFTableRefusesFilterCooperative:221 relSubset [rel#13382:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01 cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io} ScannableTableTest.testProjectableFilterableCooperative:148 relSubset [rel#13611:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01 cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io} ScannableTableTest.testProjectableFilterableNonCooperative:165 relSubset [rel#13754:Subset#2.ENUMERABLE.[]] has wrong best cost {81.0 rows, 181.01 cpu, 0.0 io}. Correct cost is {150.5 rows, 250.505 cpu, 0.0 io} FrameworksTest.testUpdate:336->executeQuery:367 relSubset [rel#22533:Subset#2.ENUMERABLE.any] has wrong best cost {19.5 rows, 37.75 cpu, 0.0 io}. Correct cost is {22.575 rows, 52.58 cpu, 0.0 io} {noformat} For the test {{MaterializationTest.testJoinMaterializationUKFK9}} initial best plan was: {noformat} EnumerableProject(empid0=[$5], empid00=[$5], deptno0=[$7]): rowcount = 15.0, cumulative cost = {15.0 rows, 45.0 cpu, 0.0 io}, id = 3989 EnumerableJoin(subset=[rel#3988:Subset#34.ENUMERABLE.[]], condition=[=($1, $7)], joinType=[inner]): rowcount = 15.0, cumulative cost = {116.0 rows, 0.0 cpu, 0.0 io}, id = 4797 EnumerableFilter(subset=[rel#4274:Subset#47.ENUMERABLE.[0]], condition=[=(CAST($2):VARCHAR CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary", 'Bill')]): rowcount = 1.0, cumulative cost = {1.0 rows, 1.0 cpu, 0.0 io}, id = 16522 EnumerableTableScan(subset=[rel#158:Subset#11.ENUMERABLE.[0]], table=[[hr, m0]]): rowcount = 1.0, cumulative cost = {0.0 rows, 1.0 cpu, 0.0 io}, id = 79 EnumerableTableScan(subset=[rel#115:Subset#5.ENUMERABLE.[]], table=[[hr, depts]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 62 {noformat} Its cumulative cost is \{221.5 rows, 123.75 cpu, 0.0 io} After applying some rules it became: {noformat} EnumerableProject(empid0=[$3], empid00=[$3], deptno0=[$0]): rowcount = 2.25, cumulative cost = {2.25 rows, 6.75 cpu, 0.0 io}, id = 4012 EnumerableFilter(subset=[rel#4007:Subset#41.ENUMERABLE.[]], condition=[=(CAST($2):VARCHAR CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary", 'Bill')]): rowcount = 2.25, cumulative cost = {2.25 rows, 15.0 cpu, 0.0 io}, id = 4811 EnumerableProject(subset=[rel#4203:Subset#61.ENUMERABLE.[]], deptno=[$7], deptno0=[$1], name0=[$2], empid0=[$5]): rowcount = 15.0, cumulative cost = {15.0 rows, 60.0 cpu, 0.0 io}, id = 4206 EnumerableJoin(subset=[rel#4204:Subset#52.ENUMERABLE.[]], condition=[=($1, $7)], joinType=[inner]): rowcount = 15.0, cumulative cost = {116.0 rows, 0.0 cpu, 0.0 io}, id = 4795 EnumerableTableScan(subset=[rel#158:Subset#11.ENUMERABLE.[0]], table=[[hr, m0]]): rowcount = 1.0, cumulative cost = {0.0 rows, 1.0 cpu, 0.0 io}, id = 79 EnumerableTableScan(subset=[rel#115:Subset#5.ENUMERABLE.[]], table=[[hr, depts]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 62 {noformat} Its cumulative cost is {{{233.0 rows, 148.0 cpu, 0.0 io}}}. The new plan does not contain
[jira] [Created] (CALCITE-2158) SubQuery with EXISTS clause creates redundant aggregate call
Volodymyr Vysotskyi created CALCITE-2158: Summary: SubQuery with EXISTS clause creates redundant aggregate call Key: CALCITE-2158 URL: https://issues.apache.org/jira/browse/CALCITE-2158 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde When {{SqlToRelConverter.Config.isExpand()}} returns true, subqueries are expanded in {{SqlToRelConverter}}. Then for the queries, like this: {code:sql} SELECT cs1.sal FROM emp cs1 WHERE EXISTS (SELECT * FROM emp cs2 WHERE cs1.sal = cs2.sal AND cs1.deptno <> cs2.deptno) {code} Calcite returns logical plan with excessive aggregate calls: {noformat} LogicalProject(SAL=[$5]) LogicalFilter(condition=[IS NOT NULL($9)]) LogicalCorrelate(correlation=[$cor0], joinType=[left], requiredColumns=[{5, 7}]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) LogicalAggregate(group=[{}], agg#0=[MIN($0)]) LogicalProject($f0=[true]) LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8]) LogicalFilter(condition=[AND(=($cor0.SAL, $5), <>($cor0.DEPTNO, $7))]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) {noformat} But when {{SqlToRelConverter.Config.isExpand()}} returns false and SubQueryRemoveRule rules are applied to the logical plan with RexSubQuery, the resulting logical plan is correct and does not contain excessive aggregate calls: {noformat} LogicalProject(SAL=[$5]) LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8]) LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{5, 7}]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) LogicalAggregate(group=[{0}]) LogicalProject(i=[true]) LogicalFilter(condition=[AND(=($cor0.SAL, $5), <>($cor0.DEPTNO, $7))]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) {noformat} These cases may be observed using this unit test: {code:java} @Test public void testExistsExpand() { final HepProgram preProgram = HepProgram.builder() .addRuleInstance(SubQueryRemoveRule.FILTER) .addRuleInstance(SubQueryRemoveRule.PROJECT) .addRuleInstance(SubQueryRemoveRule.JOIN) .build(); final HepProgram program = HepProgram.builder() .build(); final String sql = "SELECT cs1.sal\n" + "FROM emp cs1 \n" + "WHEREEXISTS\n" + "(SELECT *\n" + "FROM emp cs2\n" + "WHERE cs1.sal = cs2.sal\n" + "ANDcs1.deptno <> cs2.deptno)"; sql(sql) .withDecorrelation(false) .withTrim(false) .expand(true) // change to false .withPre(preProgram) .with(program) .checkUnchanged(); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CALCITE-2092) Allow passing custom RelBuilder into SqlToRelConverter
Volodymyr Vysotskyi created CALCITE-2092: Summary: Allow passing custom RelBuilder into SqlToRelConverter Key: CALCITE-2092 URL: https://issues.apache.org/jira/browse/CALCITE-2092 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Currently, {{SqlToRelConverter}} uses {{RelBuilder}} which was created in such a way: {code:java} RelFactories.LOGICAL_BUILDER.create(cluster, null) {code} So it is not possible to use custom {{RelBuilder}} in this class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2087) Add new method to ViewExpander interface to allow passing SchemaPlus
Volodymyr Vysotskyi created CALCITE-2087: Summary: Add new method to ViewExpander interface to allow passing SchemaPlus Key: CALCITE-2087 URL: https://issues.apache.org/jira/browse/CALCITE-2087 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Currently, {{ViewExpander}} interface contains single method {{expandView(RelDataType rowType, String queryString, List schemaPath, List viewPath)}} which allows creating a {{RelNode}} instance that corresponds to the desired view. Drill supports impersonation for views and it is implemented in such a way, that its configs such as username, etc are stored in {{SchemaPlus}} instance. So currently it is not possible to pass these configs into this method to create {{CatalogReader}} and allow impersonation work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2071) Query with IN operators in WHERE clause returns wrong result
Volodymyr Vysotskyi created CALCITE-2071: Summary: Query with IN operators in WHERE clause returns wrong result Key: CALCITE-2071 URL: https://issues.apache.org/jira/browse/CALCITE-2071 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Query it this test returns the wrong result: {code:java} @Test @Ignore public void testWhereInOr() throws Exception { CalciteAssert.hr() .query("select \"empid\"\n" + "from \"hr\".\"emps\" t\n" + "where (\"empid\" in (select \"empid\" from \"hr\".\"emps\") \n" + "or \"empid\" in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, " + " 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25) )\n" + " and \"empid\" in (100, 200, 150)") .returns("empid=100\n" + "empid=200\n" + "empid=150\n"); } {code} Without condition in or {code:sql} \"empid\" in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25) {code} it returns right result. The problem appears during converting the query into the logical plan and may be observed more clearly using this test: {code:java} @Test @Ignore public void testWhereInOrPlan() throws Exception { final String sql = "select \n" + "from emp t\n" + "where (t.EMPNO in (select EMPNO from emp) \n" + "or t.EMPNO in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, " + "12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25) )\n" + " and t.EMPNO in (1, 2, 3)"; sql(sql).ok(); } {code} This query returns plan {noformat} LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8]) LogicalFilter(condition=[AND(OR(true, true), OR(=($0, 1), =($0, 2), =($0, 3)))]) LogicalJoin(condition=[=($0, $10)], joinType=[inner]) LogicalJoin(condition=[=($0, $9)], joinType=[inner]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) LogicalAggregate(group=[{0}]) LogicalProject(EMPNO=[$0]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) LogicalAggregate(group=[{0}]) LogicalValues(tuples=[[{ 1 }, { 2 }, { 3 }, { 4 }, { 5 }, { 6 }, { 7 }, { 8 }, { 9 }, { 10 }, { 11 }, { 12 }, { 13 }, { 14 }, { 15 }, { 16 }, { 17 }, { 18 }, { 19 }, { 20 }, { 21 }, { 22 }, { 23 }, { 24 }, { 25 }]]) {noformat} where filter has incorrect condition: {{OR(true, true)}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2069) RexSimplify.removeNullabilityCast() always removes cast for operand with ANY type
Volodymyr Vysotskyi created CALCITE-2069: Summary: RexSimplify.removeNullabilityCast() always removes cast for operand with ANY type Key: CALCITE-2069 URL: https://issues.apache.org/jira/browse/CALCITE-2069 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde When a field is received from Dynamic Table, its type left {{ANY}}, and it is used in the filter condition with the cast, which actually should produce physical cast (for example we are trying to cast varchar to boolean) {{RexSimplify.removeNullabilityCast()}} removes this cast and lefts only field in condition. This test helps to observe this issue: {code:java} @Test public void testFilterCastAny() { final RelBuilder builder = RelBuilder.create(config().build()); final RelDataType intType = builder.getTypeFactory().createSqlType(SqlTypeName.ANY); RelNode root = builder.scan("EMP") .filter( builder.cast( builder.patternField("varchar_field", intType, 0), SqlTypeName.BOOLEAN)) .build(); assertThat(str(root), is("LogicalFilter(condition=[CAST(varchar_field.$0):BOOLEAN NOT NULL])\n" + " LogicalTableScan(table=[[scott, EMP]])\n")); } {code} It happens because {{SqlTypeUtil.equalSansNullability()}} returns true if any of its arguments has {{ANY}} type. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2066) RelOptUtil.splitJoinCondition() could not split condition with case after applying FilterReduceExpressionsRule
Volodymyr Vysotskyi created CALCITE-2066: Summary: RelOptUtil.splitJoinCondition() could not split condition with case after applying FilterReduceExpressionsRule Key: CALCITE-2066 URL: https://issues.apache.org/jira/browse/CALCITE-2066 Project: Calcite Issue Type: Bug Components: core Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Query {code:sql} select * from emp a emp b where coalesce(a.mgr, -1) = coalesce(b.mgr, -1) {code} has logical plan {noformat} LogicalProject(EMPNO=[$0], ENAME=[$1], JOB=[$2], MGR=[$3], HIREDATE=[$4], SAL=[$5], COMM=[$6], DEPTNO=[$7], SLACKER=[$8], EMPNO0=[$9], ENAME0=[$10], JOB0=[$11], MGR0=[$12], HIREDATE0=[$13], SAL0=[$14], COMM0=[$15], DEPTNO0=[$16], SLACKER0=[$17]) LogicalFilter(condition=[=(CASE(IS NOT NULL($3), $3, -1), CASE(IS NOT NULL($12), $12, -1))]) LogicalJoin(condition=[true], joinType=[inner]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) LogicalTableScan(table=[[CATALOG, SALES, EMP]]) {noformat} After applying {{FilterReduceExpressionsRule}} condition in filter is changed to {noformat} CASE(IS NOT NULL($12), CASE(IS NOT NULL($3), =($3, $12), =(-1, $12)), CASE(IS NOT NULL($3), =($3, -1), CAST(true):BOOLEAN)) {noformat} When this condition is passed into {{RelOptUtil.splitJoinCondition()}} method, {{leftJoinKeys}} and {{rightJoinKeys}} lists are left empty and passed condition is returned. *Note:* {{RelOptUtil.splitJoinCondition()}} works correctly for the initial condition without applying {{FilterReduceExpressionsRule}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2055) Add check for year/month/day range
Volodymyr Vysotskyi created CALCITE-2055: Summary: Add check for year/month/day range Key: CALCITE-2055 URL: https://issues.apache.org/jira/browse/CALCITE-2055 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Currently, if the year that is passed into {{DateString}} constructor has five digits, the first digit is trimmed. This trimming happens in the {{DateTimeStringUtils.int4()}} method: {code:sql} private static void int4(StringBuilder buf, int i) { buf.append((char) ('0' + (i / 1000) % 10)); buf.append((char) ('0' + (i / 100) % 10)); buf.append((char) ('0' + (i / 10) % 10)); buf.append((char) ('0' + i % 10)); } {code} The same problem with month and day values. Instead of trimming the value, the correct behaviour is to throw an exception if any of the values are outside the expected range. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2043) Use custom RelBuilder implementation in some rules
Volodymyr Vysotskyi created CALCITE-2043: Summary: Use custom RelBuilder implementation in some rules Key: CALCITE-2043 URL: https://issues.apache.org/jira/browse/CALCITE-2043 Project: Calcite Issue Type: Bug Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Some rules do not have constructors with RelBuilderFactory. List of such rules that are used in Drill: * AggregateRemoveRule * ProjectRemoveRule * SortRemoveRule * ProjectWindowTransposeRule * ExpandConversionRule * ConverterRule * ExpandConversionRule * JoinToMultiJoinRule * ProjectToWindowRule -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2026) AssertionError when ProjectReduceExpressionsRule simplifies project expressions
Volodymyr Vysotskyi created CALCITE-2026: Summary: AssertionError when ProjectReduceExpressionsRule simplifies project expressions Key: CALCITE-2026 URL: https://issues.apache.org/jira/browse/CALCITE-2026 Project: Calcite Issue Type: Bug Components: core Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde Priority: Major When applying ProjectReduceExpressionsRule on project which expression has a non-nullable type, but after simplifying expression is null, the query fails with the assertion error: {noformat} java.lang.AssertionError: Type mismatch: rel rowtype: RecordType(INTEGER EXPR$0) NOT NULL equivRel rowtype: RecordType(INTEGER NOT NULL EXPR$0) NOT NULL at org.apache.calcite.util.Litmus$1.fail(Litmus.java:31) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1868) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:855) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1767) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:268) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650) ~[calcite-core-1.13.0.jar:1.13.0] at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368) ~[calcite-core-1.13.0.jar:1.13.0] {noformat} The similar bug was described in CALCITE-1502, since there was a problem with non-nullable and nullable types in different case branches, but this bug appears when project expression is transformed into the expression with nullable type. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CALCITE-2018) Queries failed with AssertionError: rel has lower cost than best cost of subset
Volodymyr Vysotskyi created CALCITE-2018: Summary: Queries failed with AssertionError: rel has lower cost than best cost of subset Key: CALCITE-2018 URL: https://issues.apache.org/jira/browse/CALCITE-2018 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.13.0 Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde *Problem description* When debug is enabled, unit tests * MaterializationTest.testMaterializationSubstitution2 * MaterializationTest.testJoinMaterializationUKFK8 * MaterializationTest.testJoinMaterializationUKFK6 * JdbcTest.testWhereNot unit tests are failed with error AssertionError: rel has lower cost than best cost of subset. Full stack trace for test {{MaterializationTest.testMaterializationSubstitution2}}: {noformat} java.lang.AssertionError: rel [rel#245:EnumerableUnion.ENUMERABLE.[](input#0=rel#246:Subset#5.ENUMERABLE.[],input#1=rel#239:Subset#6.ENUMERABLE.[0],all=true)] has lower cost {14.0 rows, 19.0 cpu, 0.0 io} than best cost {15.0 rows, 20.0 cpu, 0.0 io} of subset [rel#243:Subset#7.ENUMERABLE.[]] at org.apache.calcite.plan.volcano.VolcanoPlanner.validate(VolcanoPlanner.java:906) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:866) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:101) at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:336) at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1495) at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:863) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:883) at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1766) at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135) at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234) at org.apache.calcite.rel.rules.FilterProjectTransposeRule.onMatch(FilterProjectTransposeRule.java:143) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650) at org.apache.calcite.tools.Programs$5.run(Programs.java:326) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:187) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:318) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:229) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:786) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:640) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:610) at org.apache.calcite.schema.Schemas.prepare(Schemas.java:346) at org.apache.calcite.materialize.MaterializationService$DefaultTableFactory.createTable(MaterializationService.java:374) at org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:137) at org.apache.calcite.materialize.MaterializationService.defineMaterialization(MaterializationService.java:99) at org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:110) at org.apache.calcite.schema.impl.MaterializedViewTable$MaterializedViewTableMacro.(MaterializedViewTable.java:100) at org.apache.calcite.schema.impl.MaterializedViewTable.create(MaterializedViewTable.java:81) at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:364) at org.apache.calcite.model.JsonMaterialization.accept(JsonMaterialization.java:42) at org.apache.calcite.model.JsonSchema.visitChildren(JsonSchema.java:98) at org.apache.calcite.model.JsonMapSchema.visitChildren(JsonMapSchema.java:48) at org.apache.calcite.model.ModelHandler.populateSchema(ModelHandler.java:257) at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:273) at org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:45) at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:196) at org.apache.calcite.model.ModelHandler.(ModelHandler.java:88) at org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:104) at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:139) at
[jira] [Created] (CALCITE-1944) Wrong plan for query with window functions and subquery with star
Volodymyr Vysotskyi created CALCITE-1944: Summary: Wrong plan for query with window functions and subquery with star Key: CALCITE-1944 URL: https://issues.apache.org/jira/browse/CALCITE-1944 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.13.0 Reporter: Volodymyr Vysotskyi Assignee: Julian Hyde *Problem description* Calcite builds the wrong plan for the query with a window function and subquery with a star: {code:sql} SELECT SUM(n_nationkey) OVER w FROM (SELECT * FROM SALES.NATION) subQry WINDOW w AS (PARTITION BY REGION ORDER BY n_nationkey) {code} Plan: {noformat} LogicalProject(EXPR$0=[CASE(>(COUNT(ITEM($0, 'N_NATIONKEY')) OVER (PARTITION BY $0 ORDER BY $0 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), 0), $SUM0(ITEM($0, 'N_NATIONKEY')) OVER (PARTITION BY $0 ORDER BY $0 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), null)]) LogicalProject(**=[$0]) LogicalTableScan(table=[[CATALOG, SALES, NATION]]) {noformat} Columns in PARTITION BY and ORDER BY clauses are {{$0}} but they should be {{ITEM($0, 'REGION')}} and {{ITEM($0, 'N_NATIONKEY')}} respectively. So correct plan should be {noformat} LogicalProject(EXPR$0=[CASE(>(COUNT(ITEM($0, 'N_NATIONKEY')) OVER (PARTITION BY ITEM($0, 'REGION') ORDER BY ITEM($0, 'N_NATIONKEY') RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), 0), $SUM0(ITEM($0, 'N_NATIONKEY')) OVER (PARTITION BY ITEM($0, 'REGION') ORDER BY ITEM($0, 'N_NATIONKEY') RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), null)]) LogicalProject(**=[$0]) LogicalTableScan(table=[[CATALOG, SALES, NATION]]) {noformat} *Root cause* In CALCITE-1150 added dynamic star column and dynamic record type support but [SqlValidatorImpl.validateWindowClause()|https://github.com/apache/calcite/blob/01c5446138d419a85678bff7db06eabb4cd39846/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L3354] method does not call [expand(SqlNode expr, SqlValidatorScope scope)|https://github.com/apache/calcite/blob/01c5446138d419a85678bff7db06eabb4cd39846/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L4795] for columns in partition and order lists. Therefore when executes this line [windowList.validate(this, windowScope);|https://github.com/apache/calcite/blob/01c5446138d419a85678bff7db06eabb4cd39846/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L3392] in the [validateIdentifier(SqlIdentifier id, SqlValidatorScope scope)|https://github.com/apache/calcite/blob/01c5446138d419a85678bff7db06eabb4cd39846/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L2778] method simple field name is replaced by the star field. -- This message was sent by Atlassian JIRA (v6.4.14#64029)