[jira] [Resolved] (CALCITE-6400) MAP_ENTRIES function should throw if a key value is null
[
https://issues.apache.org/jira/browse/CALCITE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6400.
-
Resolution: Fixed
Resloved in
[43c8848|https://github.com/apache/calcite/commit/43c88482904bc3614f4e21eb1804b4f0af76997b].
Thanks [~caicancai].
> MAP_ENTRIES function should throw if a key value is null
>
>
> Key: CALCITE-6400
> URL: https://issues.apache.org/jira/browse/CALCITE-6400
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Caican Cai
>Assignee: Caican Cai
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.38.0
>
>
> map_entries does not allow null as the key value of the map. The null
> mentioned here is a subset of the set, not that the set is empty.
> {code:java}
> scala> val df = spark.sql("select map_entries(map('foo', 1, null, 2.0))")
> df: org.apache.spark.sql.DataFrame = [map_entries(map(foo, 1, NULL, 2.0)):
> array>]
> scala> df.show()
> org.apache.spark.SparkRuntimeException: [NULL_MAP_KEY] Cannot use null as map
> key.
> at
> org.apache.spark.sql.errors.QueryExecutionErrors$.nullAsMapKeyNotAllowedError(QueryExecutionErrors.scala:445)
> at
> org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder.put(ArrayBasedMapBuilder.scala:56)
> at
> org.apache.spark.sql.catalyst.expressions.CreateMap.eval(complexTypeCreator.scala:248)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:80)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.$anonfun$constantFolding$4(expressions.scala:90)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1249)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1248)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:532)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:90)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.$anonfun$applyOrElse$1(expressions.scala:94)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:207)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:207)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:218)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$3(QueryPlan.scala:223)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.immutable.List.map(List.scala:305)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:223)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:228)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:355)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:228)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:94)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:93)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
> at
>
[jira] [Commented] (CALCITE-6400) MAP_ENTRIES allows null as a map key
[
https://issues.apache.org/jira/browse/CALCITE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846131#comment-17846131
]
hongyu guo commented on CALCITE-6400:
-
Agree with Julian
> MAP_ENTRIES allows null as a map key
>
>
> Key: CALCITE-6400
> URL: https://issues.apache.org/jira/browse/CALCITE-6400
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Caican Cai
>Assignee: Caican Cai
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.38.0
>
>
> map_entries does not allow null as the key value of the map. The null
> mentioned here is a subset of the set, not that the set is empty.
> {code:java}
> scala> val df = spark.sql("select map_entries(map('foo', 1, null, 2.0))")
> df: org.apache.spark.sql.DataFrame = [map_entries(map(foo, 1, NULL, 2.0)):
> array>]
> scala> df.show()
> org.apache.spark.SparkRuntimeException: [NULL_MAP_KEY] Cannot use null as map
> key.
> at
> org.apache.spark.sql.errors.QueryExecutionErrors$.nullAsMapKeyNotAllowedError(QueryExecutionErrors.scala:445)
> at
> org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder.put(ArrayBasedMapBuilder.scala:56)
> at
> org.apache.spark.sql.catalyst.expressions.CreateMap.eval(complexTypeCreator.scala:248)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:80)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.$anonfun$constantFolding$4(expressions.scala:90)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1249)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1248)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:532)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:90)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.$anonfun$applyOrElse$1(expressions.scala:94)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:207)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:207)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:218)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$3(QueryPlan.scala:223)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.immutable.List.map(List.scala:305)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:223)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:228)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:355)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:228)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:94)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:93)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
> at
>
[jira] [Commented] (CALCITE-6400) MAP_ENTRIES allows null as a map key
[
https://issues.apache.org/jira/browse/CALCITE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845845#comment-17845845
]
hongyu guo commented on CALCITE-6400:
-
This summary makes me confused, I don't know `map_entries` should or should not
see `null` as a legal map key without reading your code.
> MAP_ENTRIES allows null as a map key
>
>
> Key: CALCITE-6400
> URL: https://issues.apache.org/jira/browse/CALCITE-6400
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Caican Cai
>Assignee: Caican Cai
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.38.0
>
>
> map_entries does not allow null as the key value of the map. The null
> mentioned here is a subset of the set, not that the set is empty.
> {code:java}
> scala> val df = spark.sql("select map_entries(map('foo', 1, null, 2.0))")
> df: org.apache.spark.sql.DataFrame = [map_entries(map(foo, 1, NULL, 2.0)):
> array>]
> scala> df.show()
> org.apache.spark.SparkRuntimeException: [NULL_MAP_KEY] Cannot use null as map
> key.
> at
> org.apache.spark.sql.errors.QueryExecutionErrors$.nullAsMapKeyNotAllowedError(QueryExecutionErrors.scala:445)
> at
> org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder.put(ArrayBasedMapBuilder.scala:56)
> at
> org.apache.spark.sql.catalyst.expressions.CreateMap.eval(complexTypeCreator.scala:248)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:80)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.$anonfun$constantFolding$4(expressions.scala:90)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1249)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1248)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:532)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:90)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.$anonfun$applyOrElse$1(expressions.scala:94)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:207)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:207)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:218)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$3(QueryPlan.scala:223)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.immutable.List.map(List.scala:305)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:223)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:228)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:355)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:228)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:94)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:93)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
> at
>
[jira] [Updated] (CALCITE-6319) Add negative test to POW and POWER functions
[ https://issues.apache.org/jira/browse/CALCITE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo updated CALCITE-6319: Fix Version/s: (was: 1.37.0) > Add negative test to POW and POWER functions > > > Key: CALCITE-6319 > URL: https://issues.apache.org/jira/browse/CALCITE-6319 > Project: Calcite > Issue Type: Test >Affects Versions: 1.36.0 >Reporter: Caican Cai >Priority: Minor > Labels: pull-request-available > > Add negative test to pow and power functions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6319) Add negative test to POW and POWER functions
[ https://issues.apache.org/jira/browse/CALCITE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827317#comment-17827317 ] hongyu guo commented on CALCITE-6319: - Hi [~caicancai]. Are you still working on this jira? If not, can you remove 1.37.0 of fix version? > Add negative test to POW and POWER functions > > > Key: CALCITE-6319 > URL: https://issues.apache.org/jira/browse/CALCITE-6319 > Project: Calcite > Issue Type: Test >Affects Versions: 1.36.0 >Reporter: Caican Cai >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > > Add negative test to pow and power functions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6138) Parser does not accept TIMESTAMP WITH TIME ZONE as a data type
[ https://issues.apache.org/jira/browse/CALCITE-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827316#comment-17827316 ] hongyu guo commented on CALCITE-6138: - Has this bug been fixed? we should mark this ticket as `Fixed` status if does > Parser does not accept TIMESTAMP WITH TIME ZONE as a data type > -- > > Key: CALCITE-6138 > URL: https://issues.apache.org/jira/browse/CALCITE-6138 > Project: Calcite > Issue Type: Bug > Components: core >Affects Versions: 1.36.0 >Reporter: Mihai Budiu >Assignee: Mihai Budiu >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > > The current SQL grammar accepts either TIMESTAMP WITHOUT TIME ZONE or > TIMESTAMP WITH LOCAL TIME ZONE. It does not accept the version without LOCAL: > TIMESTAMP WITH TIME ZONE. This is a legal type in other SQL dialects, and it > appears in the SQL 92 standard. > The Calcite documentation also lists the rejected type as a legal scalar > type: [https://calcite.apache.org/docs/reference.html#scalar-types] > So there is a bug: either the compiler is wrong, or the documentation is > wrong. The main question is "which one?" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-6299) Support JOIN in Arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6299:
Summary: Support JOIN in Arrow adapter (was: Support JOIN in arrow adapter)
> Support JOIN in Arrow adapter
> -
>
> Key: CALCITE-6299
> URL: https://issues.apache.org/jira/browse/CALCITE-6299
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> In unit test org.apache.calcite.adapter.arrow.ArrowAdapterTest#testJoin
> {code:java}
> select t1."intField", t2."intField" from arrowdata t1 join arrowdata t2 on
> t1."intField" = t2."intField" {code}
> should return
> {code:java}
> 1
> 2
> 3
> 4
> 5
> {code}
> but return
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6295) Support IS NOT NULL in Arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6295:
Summary: Support IS NOT NULL in Arrow adapter (was: Support IS NOT NULL in
arrow adapter)
> Support IS NOT NULL in Arrow adapter
>
>
> Key: CALCITE-6295
> URL: https://issues.apache.org/jira/browse/CALCITE-6295
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is not null
> order by "intField"
> limit 1
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NOT NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6296) Support IS NULL in Arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6296:
Summary: Support IS NULL in Arrow adapter (was: Support IS NULL in arrow
adapter)
> Support IS NULL in Arrow adapter
>
>
> Key: CALCITE-6296
> URL: https://issues.apache.org/jira/browse/CALCITE-6296
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is null
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6298) Support UNION in Arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6298:
Summary: Support UNION in Arrow adapter (was: Support UNION in arrow
adapter)
> Support UNION in Arrow adapter
> --
>
> Key: CALCITE-6298
> URL: https://issues.apache.org/jira/browse/CALCITE-6298
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> Query
> {code:sql}
> (select "intField"
> from arrowdata
> where "intField" = 2)
> union
> (select "intField"
> from arrowdata
> where "intField" = 1)
> {code}
> should return
> {code}
> 1
> 2
> {code}
> but return
> {code:java}
> 1
> {code}
> See org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowUnion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6293) Support OR condition in Arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6293:
Summary: Support OR condition in Arrow adapter (was: Support OR condition
in arrow adapter)
> Support OR condition in Arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
> will throw an exception:
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6294) Support IN filter in Arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6294:
Summary: Support IN filter in Arrow adapter (was: Support IN filter in
arrow adapter)
> Support IN filter in Arrow adapter
> --
>
> Key: CALCITE-6294
> URL: https://issues.apache.org/jira/browse/CALCITE-6294
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:java}
> select "intField", "stringField"
> from arrowdata
> where "intField" in (0, 1, 2)
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
> See
> org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6292) Support more Arrow type
[
https://issues.apache.org/jira/browse/CALCITE-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6292:
Summary: Support more Arrow type (was: Support more arrow type)
> Support more Arrow type
> ---
>
> Key: CALCITE-6292
> URL: https://issues.apache.org/jira/browse/CALCITE-6292
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> All arrow type:
> {code:java}
> public static enum ArrowTypeID {
> Null(Type.Null),
> Struct(Type.Struct_),
> List(Type.List),
> LargeList(Type.LargeList),
> FixedSizeList(Type.FixedSizeList),
> Union(Type.Union),
> Map(Type.Map),
> Int(Type.Int),
> FloatingPoint(Type.FloatingPoint),
> Utf8(Type.Utf8),
> LargeUtf8(Type.LargeUtf8),
> Binary(Type.Binary),
> LargeBinary(Type.LargeBinary),
> FixedSizeBinary(Type.FixedSizeBinary),
> Bool(Type.Bool),
> Decimal(Type.Decimal),
> Date(Type.Date),
> Time(Type.Time),
> Timestamp(Type.Timestamp),
> Interval(Type.Interval),
> Duration(Type.Duration),
> NONE(Type.NONE);
> }
> {code}
> we support now:
> {code:java}
> public static ArrowFieldType of(ArrowType arrowType) {
> switch (arrowType.getTypeID()) {
> case Int:
> int bitWidth = ((ArrowType.Int) arrowType).getBitWidth();
> switch (bitWidth) {
> case 64:
> return LONG;
> case 32:
> return INT;
> case 16:
> return SHORT;
> case 8:
> return BYTE;
> default:
> throw new IllegalArgumentException("Unsupported Int bit width: " +
> bitWidth);
> }
> case Bool:
> return BOOLEAN;
> case Utf8:
> return STRING;
> case FloatingPoint:
> FloatingPointPrecision precision = ((ArrowType.FloatingPoint)
> arrowType).getPrecision();
> switch (precision) {
> case SINGLE:
> return FLOAT;
> case DOUBLE:
> return DOUBLE;
> default:
> throw new IllegalArgumentException("Unsupported Floating point
> precision: " + precision);
> }
> case Date:
> return DATE;
> case Decimal:
> return DECIMAL;
> default:
> throw new IllegalArgumentException("Unsupported type: " + arrowType);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6300) Function MAP_VALUES/MAP_KEYS gives exception when mapVauleType and mapKeyType not equals map Biggest mapKeytype or mapValueType
[
https://issues.apache.org/jira/browse/CALCITE-6300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6300:
Summary: Function MAP_VALUES/MAP_KEYS gives exception when mapVauleType and
mapKeyType not equals map Biggest mapKeytype or mapValueType (was:
[CALCITE-6300] Function MAP_VALUES/MAP_KEYS gives exception when mapVauleType
and mapKeyType not equals map Biggest mapKeytype or mapValueType )
> Function MAP_VALUES/MAP_KEYS gives exception when mapVauleType and mapKeyType
> not equals map Biggest mapKeytype or mapValueType
>
>
> Key: CALCITE-6300
> URL: https://issues.apache.org/jira/browse/CALCITE-6300
> Project: Calcite
> Issue Type: Bug
> Components: core
>Affects Versions: 1.36.0
>Reporter: Caican Cai
>Assignee: Caican Cai
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> If we run the expression below in calcite, it will cause exception:
> {code:java}
> map_values(map('foo', cast (2 as tinyint), 'bar', 2)) {code}
> {code:java}
> java.lang.ClassCastException: java.lang.Byte cannot be cast to
> java.lang.Integer
> at
> org.apache.calcite.avatica.util.AbstractCursor$IntAccessor.getInt(AbstractCursor.java:522)
>at
> org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1396)
> at
> org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1377)
> at
> org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1432)
> at
> org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getString(AbstractCursor.java:1444)
> at
> org.apache.calcite.avatica.AvaticaResultSet.getString(AvaticaResultSet.java:241)
> at org.apache.calcite.util.JdbcTypeImpl$10.get(JdbcTypeImpl.java:112)
> at org.apache.calcite.util.JdbcTypeImpl$10.get(JdbcTypeImpl.java:109) at
> org.apache.calcite.sql.test.ResultCheckers.compareResultSetWithMatcher(ResultCheckers.java:248)
> at
> org.apache.calcite.sql.test.ResultCheckers$MatcherResultChecker.checkResult(ResultCheckers.java:321)
> at
> org.apache.calcite.test.SqlOperatorTest$TesterImpl.check(SqlOperatorTest.java:14300)
> at org.apache.calcite.sql.test.SqlTester.check(SqlTester.java:160) at
> org.apache.calcite.test.SqlOperatorFixtureImpl.lambda$checkScalar$2(SqlOperatorFixtureImpl.java:226)
> at
> org.apache.calcite.sql.test.AbstractSqlTester.forEachQuery(AbstractSqlTester.java:450)
>at
> org.apache.calcite.test.SqlOperatorFixtureImpl.checkScalar(SqlOperatorFixtureImpl.java:225)
> at
> org.apache.calcite.sql.test.SqlOperatorFixture.checkScalar(SqlOperatorFixture.java:229)
> at
> org.apache.calcite.test.SqlOperatorTest.testMapValuesFunc(SqlOperatorTest.java:7276)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
>
[jira] [Resolved] (CALCITE-6278) Add REGEXP, REGEXP_LIKE function (enabled in Spark library)
[ https://issues.apache.org/jira/browse/CALCITE-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo resolved CALCITE-6278. - Fix Version/s: 1.37.0 Assignee: EveyWu Resolution: Fixed Fixed in [d8804b46|https://github.com/apache/calcite/commit/d8804b46870d9d1ba2bcfafc0646ce206647625c]. Thanks [~eveywu]. > Add REGEXP, REGEXP_LIKE function (enabled in Spark library) > > > Key: CALCITE-6278 > URL: https://issues.apache.org/jira/browse/CALCITE-6278 > Project: Calcite > Issue Type: Improvement >Reporter: EveyWu >Assignee: EveyWu >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: image-2024-03-07-09-32-27-002.png, > image-2024-03-09-11-13-49-064.png, image-2024-03-09-11-37-27-816.png, > image-2024-03-09-11-38-08-797.png > > > Add Spark functions that have been implemented but have different > OperandTypes/Returns. > Add Function > [REGEXP|https://spark.apache.org/docs/latest/api/sql/index.html#regexp], > [REGEXP_LIKE|https://spark.apache.org/docs/latest/api/sql/index.html#regexp_like] > # Since this function has the same implementation as the Spark > [RLIKE|https://spark.apache.org/docs/latest/api/sql/index.html#rlike] > function, the implementation can be directly reused. > # -Since Spark 2.0, string literals (including regex patterns) are unescaped > in SQL parser, also fix this bug in calcite.- > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6306) JDBC adapter should not generate FILTER (WHERE) in MySQL and StarRocks dialect
[
https://issues.apache.org/jira/browse/CALCITE-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6306.
-
Fix Version/s: 1.37.0
Assignee: hongyu guo
Resolution: Fixed
Fixed in
[2ce30626|https://github.com/apache/calcite/commit/2ce306263327a30255bda858ce437c184546ca9b].
> JDBC adapter should not generate FILTER (WHERE) in MySQL and StarRocks dialect
> --
>
> Key: CALCITE-6306
> URL: https://issues.apache.org/jira/browse/CALCITE-6306
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Assignee: hongyu guo
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> {code:sql}
> mysql> select sum(x) filter (where x = 1) from t;
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near '(where x = 1) from t' at line 1 {code}
> See details in [https://modern-sql.com/feature/filter]
> Calcite should not generate the agg() FILTER clause for the MySQL dialect.
> CALCITE-4321 introduced a way to convert the FILTER clause to an aggregation
> function with CASE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825134#comment-17825134 ] hongyu guo edited comment on CALCITE-2040 at 3/11/24 2:47 AM: -- Fixed in [d4e8830|https://github.com/apache/calcite/commit/d4e88302e367b7f5a3b3da9d2e0f734320cef413]. Thanks all contributors! was (Author: JIRAUSER300840): Fixed in [d4e8830|[https://github.com/apache/calcite/commit/d4e88302e367b7f5a3b3da9d2e0f734320cef413].] Thanks all contributors! > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo resolved CALCITE-2040. - Resolution: Fixed Fixed in [d4e8830|[https://github.com/apache/calcite/commit/d4e88302e367b7f5a3b3da9d2e0f734320cef413].] Thanks all contributors! > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-6306) JDBC adapter should not generate FILTER (WHERE) in MySQL and StarRocks dialect
[
https://issues.apache.org/jira/browse/CALCITE-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6306:
Summary: JDBC adapter should not generate FILTER (WHERE) in MySQL and
StarRocks dialect (was: FILTER clauses for aggregate functions are not
supported in MySQL, MariaDB and StarRocks)
> JDBC adapter should not generate FILTER (WHERE) in MySQL and StarRocks dialect
> --
>
> Key: CALCITE-6306
> URL: https://issues.apache.org/jira/browse/CALCITE-6306
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Minor
> Labels: pull-request-available
>
> {code:sql}
> mysql> select sum(x) filter (where x = 1) from t;
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near '(where x = 1) from t' at line 1 {code}
> See details in [https://modern-sql.com/feature/filter]
> Calcite should not generate the agg() FILTER clause for the MySQL dialect.
> CALCITE-4321 introduced a way to convert the FILTER clause to an aggregation
> function with CASE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6306) FILTER clauses for aggregate functions are not supported in MySQL, MariaDB and StarRocks
[
https://issues.apache.org/jira/browse/CALCITE-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6306:
Summary: FILTER clauses for aggregate functions are not supported in MySQL,
MariaDB and StarRocks (was: FILTER clauses for aggregate functions are not
supported in MySQL, MariaDB and Starrocks)
> FILTER clauses for aggregate functions are not supported in MySQL, MariaDB
> and StarRocks
>
>
> Key: CALCITE-6306
> URL: https://issues.apache.org/jira/browse/CALCITE-6306
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Minor
> Labels: pull-request-available
>
> {code:sql}
> mysql> select sum(x) filter (where x = 1) from t;
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near '(where x = 1) from t' at line 1 {code}
> See details in [https://modern-sql.com/feature/filter]
> Calcite should not generate the agg() FILTER clause for the MySQL dialect.
> CALCITE-4321 introduced a way to convert the FILTER clause to an aggregation
> function with CASE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6306) FILTER clauses for aggregate functions are not supported in MySQL, MariaDB and Starrocks
[
https://issues.apache.org/jira/browse/CALCITE-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6306:
Description:
{code:sql}
mysql> select sum(x) filter (where x = 1) from t;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'(where x = 1) from t' at line 1 {code}
See details in [https://modern-sql.com/feature/filter]
Calcite should not generate the agg() FILTER clause for the MySQL dialect.
CALCITE-4321 introduced a way to convert the FILTER clause to an aggregation
function with CASE.
was:
{code:sql}
mysql> select sum(x) filter (where x = 1) from t;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'(where x = 1) from t' at line 1 {code}
See details in [https://modern-sql.com/feature/filter]
> FILTER clauses for aggregate functions are not supported in MySQL, MariaDB
> and Starrocks
>
>
> Key: CALCITE-6306
> URL: https://issues.apache.org/jira/browse/CALCITE-6306
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Minor
> Labels: pull-request-available
>
> {code:sql}
> mysql> select sum(x) filter (where x = 1) from t;
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near '(where x = 1) from t' at line 1 {code}
> See details in [https://modern-sql.com/feature/filter]
> Calcite should not generate the agg() FILTER clause for the MySQL dialect.
> CALCITE-4321 introduced a way to convert the FILTER clause to an aggregation
> function with CASE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6306) FILTER clauses for aggregate functions are not supported in MySQL, MariaDB and Starrocks
[
https://issues.apache.org/jira/browse/CALCITE-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6306:
Description:
{code:sql}
mysql> select sum(x) filter (where x = 1) from t;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'(where x = 1) from t' at line 1 {code}
See details in [https://modern-sql.com/feature/filter]
was:
{code:sql}
mysql> select sum(x) filter (where x = 1) from t;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'(where x = 1) from t' at line 1 {code}
See details in https://modern-sql.com/feature/filter
> FILTER clauses for aggregate functions are not supported in MySQL, MariaDB
> and Starrocks
>
>
> Key: CALCITE-6306
> URL: https://issues.apache.org/jira/browse/CALCITE-6306
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Minor
> Labels: pull-request-available
>
> {code:sql}
> mysql> select sum(x) filter (where x = 1) from t;
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near '(where x = 1) from t' at line 1 {code}
> See details in [https://modern-sql.com/feature/filter]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-2040) Create adapter for Apache Arrow
[
https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824357#comment-17824357
]
hongyu guo commented on CALCITE-2040:
-
[~zabetak] I have addad 5 co-authors, based on the PR2133 and PR2810.
{code:java}
[CALCITE-2040] Create adapter for Apache Arrow
Co-authored-by: Alessandro Solimando
Co-authored-by: Jonathan Swenson
Co-authored-by: Julian Hyde
Co-authored-by: Karshit Shah
Co-authored-by: Michael Mior {code}
> Create adapter for Apache Arrow
> ---
>
> Key: CALCITE-2040
> URL: https://issues.apache.org/jira/browse/CALCITE-2040
> Project: Calcite
> Issue Type: Bug
>Reporter: Julian Hyde
>Assignee: hongyu guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
> Attachments: arrow_data.py
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would
> allow people to execute SQL statements, via JDBC or ODBC, on data stored in
> Arrow in-memory format.
> Since Arrow is an in-memory format, it is not as straightforward as reading,
> say, CSV files using the file adapter: an Arrow data set does not have a URL.
> (Unless we use Arrow's
> [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/]
> format, or use an in-memory file system such as Alluxio.) So we would need
> to devise a way of addressing Arrow data sets.
> Also, since Arrow is an extremely efficient format for processing data, it
> would also be good to have Arrow as a calling convention. That is,
> implementations of relational operators such as Filter, Project, Aggregate in
> addition to just TableScan.
> Lastly, when we have an Arrow convention, if we build adapters for file
> formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in
> CALCITE-2025) it would make a lot of sense to translate those formats
> directly into Arrow (applying simple projects and filters first if
> applicable). Those adapters would belong as a "contrib" module in the Arrow
> project better than in Calcite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6306) FILTER clauses for aggregate functions are not supported in MySQL, MariaDB and Starrocks
hongyu guo created CALCITE-6306:
---
Summary: FILTER clauses for aggregate functions are not supported
in MySQL, MariaDB and Starrocks
Key: CALCITE-6306
URL: https://issues.apache.org/jira/browse/CALCITE-6306
Project: Calcite
Issue Type: Bug
Reporter: hongyu guo
{code:sql}
mysql> select sum(x) filter (where x = 1) from t;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near
'(where x = 1) from t' at line 1 {code}
See details in https://modern-sql.com/feature/filter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6305) Increase coverage of unit tests for arrow adapter
[ https://issues.apache.org/jira/browse/CALCITE-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo updated CALCITE-6305: Description: Add more unit tests for arrow adapter. for example * casts, including lossy casts * complex filter condition * complex SQL * and more... Note: Calcite's support for the arrow feather format is not complete, so before adding the testing process, we may need to implement new features first. was: Add more unit tests for arrow adapter. for example * casts, including lossy casts * complex filter condition * complex SQL Note: Calcite's support for the arrow feather format is not complete, so before adding the testing process, we may need to implement new features first. > Increase coverage of unit tests for arrow adapter > - > > Key: CALCITE-6305 > URL: https://issues.apache.org/jira/browse/CALCITE-6305 > Project: Calcite > Issue Type: Sub-task >Reporter: hongyu guo >Priority: Major > > Add more unit tests for arrow adapter. > for example > * casts, including lossy casts > * complex filter condition > * complex SQL > * and more... > Note: Calcite's support for the arrow feather format is not complete, so > before adding the testing process, we may need to implement new features > first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6305) Increase coverage of unit tests for arrow adapter
hongyu guo created CALCITE-6305: --- Summary: Increase coverage of unit tests for arrow adapter Key: CALCITE-6305 URL: https://issues.apache.org/jira/browse/CALCITE-6305 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo Add more unit tests for arrow adapter. for example * casts, including lossy casts * complex filter condition * complex SQL Note: Calcite's support for the arrow feather format is not complete, so before adding the testing process, we may need to implement new features first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (CALCITE-6297) Support sub-queries in arrow adapter
[ https://issues.apache.org/jira/browse/CALCITE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo closed CALCITE-6297. --- Resolution: Not A Problem It is not a valid issue, so I close it. > Support sub-queries in arrow adapter > > > Key: CALCITE-6297 > URL: https://issues.apache.org/jira/browse/CALCITE-6297 > Project: Calcite > Issue Type: Sub-task >Reporter: hongyu guo >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-6297) Support sub-queries in arrow adapter
[ https://issues.apache.org/jira/browse/CALCITE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo updated CALCITE-6297: Parent: (was: CALCITE-2040) Issue Type: Bug (was: Sub-task) > Support sub-queries in arrow adapter > > > Key: CALCITE-6297 > URL: https://issues.apache.org/jira/browse/CALCITE-6297 > Project: Calcite > Issue Type: Bug >Reporter: hongyu guo >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-6297) Support sub-queries in arrow adapter
[ https://issues.apache.org/jira/browse/CALCITE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo updated CALCITE-6297: Parent: CALCITE-6304 Issue Type: Sub-task (was: Bug) > Support sub-queries in arrow adapter > > > Key: CALCITE-6297 > URL: https://issues.apache.org/jira/browse/CALCITE-6297 > Project: Calcite > Issue Type: Sub-task >Reporter: hongyu guo >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-6299) Support JOIN in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6299:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Bug)
> Support JOIN in arrow adapter
> -
>
> Key: CALCITE-6299
> URL: https://issues.apache.org/jira/browse/CALCITE-6299
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> In unit test org.apache.calcite.adapter.arrow.ArrowAdapterTest#testJoin
> {code:java}
> select t1."intField", t2."intField" from arrowdata t1 join arrowdata t2 on
> t1."intField" = t2."intField" {code}
> should return
> {code:java}
> 1
> 2
> 3
> 4
> 5
> {code}
> but return
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6298) Support UNION in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6298:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Bug)
> Support UNION in arrow adapter
> --
>
> Key: CALCITE-6298
> URL: https://issues.apache.org/jira/browse/CALCITE-6298
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> Query
> {code:sql}
> (select "intField"
> from arrowdata
> where "intField" = 2)
> union
> (select "intField"
> from arrowdata
> where "intField" = 1)
> {code}
> should return
> {code}
> 1
> 2
> {code}
> but return
> {code:java}
> 1
> {code}
> See org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowUnion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6299) Support JOIN in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6299:
Parent: (was: CALCITE-2040)
Issue Type: Bug (was: Sub-task)
> Support JOIN in arrow adapter
> -
>
> Key: CALCITE-6299
> URL: https://issues.apache.org/jira/browse/CALCITE-6299
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Major
>
> In unit test org.apache.calcite.adapter.arrow.ArrowAdapterTest#testJoin
> {code:java}
> select t1."intField", t2."intField" from arrowdata t1 join arrowdata t2 on
> t1."intField" = t2."intField" {code}
> should return
> {code:java}
> 1
> 2
> 3
> 4
> 5
> {code}
> but return
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6298) Support UNION in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6298:
Parent: (was: CALCITE-2040)
Issue Type: Bug (was: Sub-task)
> Support UNION in arrow adapter
> --
>
> Key: CALCITE-6298
> URL: https://issues.apache.org/jira/browse/CALCITE-6298
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Major
>
> Query
> {code:sql}
> (select "intField"
> from arrowdata
> where "intField" = 2)
> union
> (select "intField"
> from arrowdata
> where "intField" = 1)
> {code}
> should return
> {code}
> 1
> 2
> {code}
> but return
> {code:java}
> 1
> {code}
> See org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowUnion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6295) Support IS NOT NULL in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6295:
Parent: (was: CALCITE-2040)
Issue Type: Bug (was: Sub-task)
> Support IS NOT NULL in arrow adapter
>
>
> Key: CALCITE-6295
> URL: https://issues.apache.org/jira/browse/CALCITE-6295
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is not null
> order by "intField"
> limit 1
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NOT NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6295) Support IS NOT NULL in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6295:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Bug)
> Support IS NOT NULL in arrow adapter
>
>
> Key: CALCITE-6295
> URL: https://issues.apache.org/jira/browse/CALCITE-6295
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is not null
> order by "intField"
> limit 1
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NOT NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6296) Support IS NULL in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6296:
Parent: (was: CALCITE-2040)
Issue Type: Bug (was: Sub-task)
> Support IS NULL in arrow adapter
>
>
> Key: CALCITE-6296
> URL: https://issues.apache.org/jira/browse/CALCITE-6296
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Major
>
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is null
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6296) Support IS NULL in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6296:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Bug)
> Support IS NULL in arrow adapter
>
>
> Key: CALCITE-6296
> URL: https://issues.apache.org/jira/browse/CALCITE-6296
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is null
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6292) Support more arrow type
[
https://issues.apache.org/jira/browse/CALCITE-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6292:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Bug)
> Support more arrow type
> ---
>
> Key: CALCITE-6292
> URL: https://issues.apache.org/jira/browse/CALCITE-6292
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> All arrow type:
> {code:java}
> public static enum ArrowTypeID {
> Null(Type.Null),
> Struct(Type.Struct_),
> List(Type.List),
> LargeList(Type.LargeList),
> FixedSizeList(Type.FixedSizeList),
> Union(Type.Union),
> Map(Type.Map),
> Int(Type.Int),
> FloatingPoint(Type.FloatingPoint),
> Utf8(Type.Utf8),
> LargeUtf8(Type.LargeUtf8),
> Binary(Type.Binary),
> LargeBinary(Type.LargeBinary),
> FixedSizeBinary(Type.FixedSizeBinary),
> Bool(Type.Bool),
> Decimal(Type.Decimal),
> Date(Type.Date),
> Time(Type.Time),
> Timestamp(Type.Timestamp),
> Interval(Type.Interval),
> Duration(Type.Duration),
> NONE(Type.NONE);
> }
> {code}
> we support now:
> {code:java}
> public static ArrowFieldType of(ArrowType arrowType) {
> switch (arrowType.getTypeID()) {
> case Int:
> int bitWidth = ((ArrowType.Int) arrowType).getBitWidth();
> switch (bitWidth) {
> case 64:
> return LONG;
> case 32:
> return INT;
> case 16:
> return SHORT;
> case 8:
> return BYTE;
> default:
> throw new IllegalArgumentException("Unsupported Int bit width: " +
> bitWidth);
> }
> case Bool:
> return BOOLEAN;
> case Utf8:
> return STRING;
> case FloatingPoint:
> FloatingPointPrecision precision = ((ArrowType.FloatingPoint)
> arrowType).getPrecision();
> switch (precision) {
> case SINGLE:
> return FLOAT;
> case DOUBLE:
> return DOUBLE;
> default:
> throw new IllegalArgumentException("Unsupported Floating point
> precision: " + precision);
> }
> case Date:
> return DATE;
> case Decimal:
> return DECIMAL;
> default:
> throw new IllegalArgumentException("Unsupported type: " + arrowType);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6294) Support IN filter in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6294:
Parent: (was: CALCITE-2040)
Issue Type: Bug (was: Sub-task)
> Support IN filter in arrow adapter
> --
>
> Key: CALCITE-6294
> URL: https://issues.apache.org/jira/browse/CALCITE-6294
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:java}
> select "intField", "stringField"
> from arrowdata
> where "intField" in (0, 1, 2)
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
> See
> org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6292) Support more arrow type
[
https://issues.apache.org/jira/browse/CALCITE-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6292:
Parent: (was: CALCITE-2040)
Issue Type: Bug (was: Sub-task)
> Support more arrow type
> ---
>
> Key: CALCITE-6292
> URL: https://issues.apache.org/jira/browse/CALCITE-6292
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Major
>
> All arrow type:
> {code:java}
> public static enum ArrowTypeID {
> Null(Type.Null),
> Struct(Type.Struct_),
> List(Type.List),
> LargeList(Type.LargeList),
> FixedSizeList(Type.FixedSizeList),
> Union(Type.Union),
> Map(Type.Map),
> Int(Type.Int),
> FloatingPoint(Type.FloatingPoint),
> Utf8(Type.Utf8),
> LargeUtf8(Type.LargeUtf8),
> Binary(Type.Binary),
> LargeBinary(Type.LargeBinary),
> FixedSizeBinary(Type.FixedSizeBinary),
> Bool(Type.Bool),
> Decimal(Type.Decimal),
> Date(Type.Date),
> Time(Type.Time),
> Timestamp(Type.Timestamp),
> Interval(Type.Interval),
> Duration(Type.Duration),
> NONE(Type.NONE);
> }
> {code}
> we support now:
> {code:java}
> public static ArrowFieldType of(ArrowType arrowType) {
> switch (arrowType.getTypeID()) {
> case Int:
> int bitWidth = ((ArrowType.Int) arrowType).getBitWidth();
> switch (bitWidth) {
> case 64:
> return LONG;
> case 32:
> return INT;
> case 16:
> return SHORT;
> case 8:
> return BYTE;
> default:
> throw new IllegalArgumentException("Unsupported Int bit width: " +
> bitWidth);
> }
> case Bool:
> return BOOLEAN;
> case Utf8:
> return STRING;
> case FloatingPoint:
> FloatingPointPrecision precision = ((ArrowType.FloatingPoint)
> arrowType).getPrecision();
> switch (precision) {
> case SINGLE:
> return FLOAT;
> case DOUBLE:
> return DOUBLE;
> default:
> throw new IllegalArgumentException("Unsupported Floating point
> precision: " + precision);
> }
> case Date:
> return DATE;
> case Decimal:
> return DECIMAL;
> default:
> throw new IllegalArgumentException("Unsupported type: " + arrowType);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6294) Support IN filter in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6294:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Bug)
> Support IN filter in arrow adapter
> --
>
> Key: CALCITE-6294
> URL: https://issues.apache.org/jira/browse/CALCITE-6294
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:java}
> select "intField", "stringField"
> from arrowdata
> where "intField" in (0, 1, 2)
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
> See
> org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6291) Support converting ArrowTable to Queryable
[
https://issues.apache.org/jira/browse/CALCITE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6291:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Bug)
> Support converting ArrowTable to Queryable
> --
>
> Key: CALCITE-6291
> URL: https://issues.apache.org/jira/browse/CALCITE-6291
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Minor
>
> Now we can not convert ArrowTable to Queryable
> {code:java}
> @Override public Queryable asQueryable(QueryProvider queryProvider,
> SchemaPlus schema, String tableName) {
> throw new UnsupportedOperationException();
> } {code}
> See more in org.apache.calcite.adapter.arrow.ArrowTable#asQueryable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6291) Support converting ArrowTable to Queryable
[
https://issues.apache.org/jira/browse/CALCITE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6291:
Parent: (was: CALCITE-2040)
Issue Type: Bug (was: Sub-task)
> Support converting ArrowTable to Queryable
> --
>
> Key: CALCITE-6291
> URL: https://issues.apache.org/jira/browse/CALCITE-6291
> Project: Calcite
> Issue Type: Bug
>Reporter: hongyu guo
>Priority: Minor
>
> Now we can not convert ArrowTable to Queryable
> {code:java}
> @Override public Queryable asQueryable(QueryProvider queryProvider,
> SchemaPlus schema, String tableName) {
> throw new UnsupportedOperationException();
> } {code}
> See more in org.apache.calcite.adapter.arrow.ArrowTable#asQueryable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6293) Support OR condition in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6293:
Parent: CALCITE-6304
Issue Type: Sub-task (was: Improvement)
> Support OR condition in arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
> will throw an exception:
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6293) Support OR condition in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6293:
Parent: (was: CALCITE-2040)
Issue Type: Improvement (was: Sub-task)
> Support OR condition in arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Improvement
>Reporter: hongyu guo
>Priority: Major
>
> For example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
> will throw an exception:
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6304) Improve the Arrow adapter
hongyu guo created CALCITE-6304:
---
Summary: Improve the Arrow adapter
Key: CALCITE-6304
URL: https://issues.apache.org/jira/browse/CALCITE-6304
Project: Calcite
Issue Type: Improvement
Reporter: hongyu guo
After merging [CALCITE-2040], we obtained the initial version of the arrow
adapter. But, we encountered issues with certain queries throwing an exception
or producing incorrect results in the arrow adapter. For example, consider the
following query:
{code:java}
select "intField", "stringField"
from arrowdata
where "intField" in (0, 1, 2)
{code}
This query will throw an exception, So we need to improve it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6299) Support JOIN in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6299:
Description:
In unit test org.apache.calcite.adapter.arrow.ArrowAdapterTest#testJoin
{code:java}
select t1."intField", t2."intField" from arrowdata t1 join arrowdata t2 on
t1."intField" = t2."intField" {code}
should return
{code:java}
1
2
3
4
5
{code}
but return
{code:java}
{code}
> Support JOIN in arrow adapter
> -
>
> Key: CALCITE-6299
> URL: https://issues.apache.org/jira/browse/CALCITE-6299
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> In unit test org.apache.calcite.adapter.arrow.ArrowAdapterTest#testJoin
> {code:java}
> select t1."intField", t2."intField" from arrowdata t1 join arrowdata t2 on
> t1."intField" = t2."intField" {code}
> should return
> {code:java}
> 1
> 2
> 3
> 4
> 5
> {code}
> but return
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6298) Support UNION in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6298:
Description:
Query
{code:sql}
(select "intField"
from arrowdata
where "intField" = 2)
union
(select "intField"
from arrowdata
where "intField" = 1)
{code}
should return
{code}
1
2
{code}
but return
{code:java}
1
{code}
See org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowUnion
> Support UNION in arrow adapter
> --
>
> Key: CALCITE-6298
> URL: https://issues.apache.org/jira/browse/CALCITE-6298
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> Query
> {code:sql}
> (select "intField"
> from arrowdata
> where "intField" = 2)
> union
> (select "intField"
> from arrowdata
> where "intField" = 1)
> {code}
> should return
> {code}
> 1
> 2
> {code}
> but return
> {code:java}
> 1
> {code}
> See org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowUnion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6297) Support sub-queries in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6297:
Description: (was: For example:
{code:java}
select "intField", "stringField"
from arrowdata
where "intField" in (0, 1, 2)
{code}
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43) at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
See
org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter)
> Support sub-queries in arrow adapter
>
>
> Key: CALCITE-6297
> URL: https://issues.apache.org/jira/browse/CALCITE-6297
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6296) Support IS NULL in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6296:
Description:
{code:sql}
select "intField", "stringField"
from arrowdata
where "intField" is null
{code}
will throw an exception
{code:java}
java.lang.AssertionError: cannot translate IS NULL($0)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
> Support IS NULL in arrow adapter
>
>
> Key: CALCITE-6296
> URL: https://issues.apache.org/jira/browse/CALCITE-6296
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is null
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6294) Support IN filter in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6294:
Description:
For example:
{code:java}
select "intField", "stringField"
from arrowdata
where "intField" in (0, 1, 2)
{code}
will throw an exception
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43) at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
See
org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
was:
For example:
{code:java}
select "intField", "stringField"
from arrowdata
where "intField" in (0, 1, 2)
{code}
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43) at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
See
org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
> Support IN filter in arrow adapter
> --
>
> Key: CALCITE-6294
> URL: https://issues.apache.org/jira/browse/CALCITE-6294
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:java}
> select "intField", "stringField"
> from arrowdata
> where "intField" in (0, 1, 2)
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
> See
> org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6295) Support IS NOT NULL in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6295:
Description:
For example:
{code:sql}
select "intField", "stringField"
from arrowdata
where "intField" is not null
order by "intField"
limit 1
{code}
will throw an exception
{code:java}
java.lang.AssertionError: cannot translate IS NOT NULL($0)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
> Support IS NOT NULL in arrow adapter
>
>
> Key: CALCITE-6295
> URL: https://issues.apache.org/jira/browse/CALCITE-6295
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:sql}
> select "intField", "stringField"
> from arrowdata
> where "intField" is not null
> order by "intField"
> limit 1
> {code}
> will throw an exception
> {code:java}
> java.lang.AssertionError: cannot translate IS NOT NULL($0)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch2(ArrowTranslator.java:131)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:111)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6294) Support IN filter in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6294:
Description:
For example:
{code:java}
select "intField", "stringField"
from arrowdata
where "intField" in (0, 1, 2)
{code}
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43) at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
See
org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
> Support IN filter in arrow adapter
> --
>
> Key: CALCITE-6294
> URL: https://issues.apache.org/jira/browse/CALCITE-6294
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:java}
> select "intField", "stringField"
> from arrowdata
> where "intField" in (0, 1, 2)
> {code}
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
> See
> org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6297) Support sub-queries in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6297:
Description:
For example:
{code:java}
select "intField", "stringField"
from arrowdata
where "intField" in (0, 1, 2)
{code}
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43) at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
See
org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
> Support sub-queries in arrow adapter
>
>
> Key: CALCITE-6297
> URL: https://issues.apache.org/jira/browse/CALCITE-6297
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example:
> {code:java}
> select "intField", "stringField"
> from arrowdata
> where "intField" in (0, 1, 2)
> {code}
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 0), =($0, 1), =($0, 2))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateAnd(ArrowTranslator.java:109)
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:68)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
> See
> org.apache.calcite.adapter.arrow.ArrowAdapterTest#testArrowProjectFieldsWithInFilter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6291) Support converting ArrowTable to Queryable
[
https://issues.apache.org/jira/browse/CALCITE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6291:
Description:
Now we can not convert ArrowTable to Queryable
{code:java}
@Override public Queryable asQueryable(QueryProvider queryProvider,
SchemaPlus schema, String tableName) {
throw new UnsupportedOperationException();
} {code}
See more in org.apache.calcite.adapter.arrow.ArrowTable#asQueryable
> Support converting ArrowTable to Queryable
> --
>
> Key: CALCITE-6291
> URL: https://issues.apache.org/jira/browse/CALCITE-6291
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Minor
>
> Now we can not convert ArrowTable to Queryable
> {code:java}
> @Override public Queryable asQueryable(QueryProvider queryProvider,
> SchemaPlus schema, String tableName) {
> throw new UnsupportedOperationException();
> } {code}
> See more in org.apache.calcite.adapter.arrow.ArrowTable#asQueryable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6293) Support OR condition in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823987#comment-17823987
]
hongyu guo commented on CALCITE-6293:
-
Thanks for the explanation, [~asolimando]. I will now add some descriptions for
all the tickets I have created.
> Support OR condition in arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
> will throw an exception:
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6293) Support OR condition in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6293:
Description:
For example
{code:java}
String sql = "select \"intField\", \"stringField\"\n"
+ "from arrowdata\n"
+ "where \"intField\"=12 or \"stringField\"='12'"; {code}
will throw an exception:
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
was:
for example
{code:java}
String sql = "select \"intField\", \"stringField\"\n"
+ "from arrowdata\n"
+ "where \"intField\"=12 or \"stringField\"='12'"; {code}
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
> Support OR condition in arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> For example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
> will throw an exception:
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6293) Support OR condition in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6293:
Description:
for example
{code:java}
String sql = "select \"intField\", \"stringField\"\n"
+ "from arrowdata\n"
+ "where \"intField\"=12 or \"stringField\"='12'"; {code}
{code:java}
java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
at
org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
at
org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
at
org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
{code}
was:
for example
{code:java}
String sql = "select \"intField\", \"stringField\"\n"
+ "from arrowdata\n"
+ "where \"intField\"=12 or \"stringField\"='12'"; {code}
> Support OR condition in arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> for example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
> {code:java}
> java.lang.AssertionError: cannot translate OR(=($0, 12), =($1, '12'))
> at
> org.apache.calcite.adapter.arrow.ArrowTranslator.translateMatch(ArrowTranslator.java:70)
> at
> org.apache.calcite.adapter.arrow.ArrowFilter.(ArrowFilter.java:43)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.convert(ArrowRules.java:97)
> at
> org.apache.calcite.adapter.arrow.ArrowRules$ArrowFilterRule.onMatch(ArrowRules.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6293) Support OR condition in arrow adapter
[
https://issues.apache.org/jira/browse/CALCITE-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6293:
Summary: Support OR condition in arrow adapter (was: Support OR condition
for arrow adapter)
> Support OR condition in arrow adapter
> -
>
> Key: CALCITE-6293
> URL: https://issues.apache.org/jira/browse/CALCITE-6293
> Project: Calcite
> Issue Type: Sub-task
>Reporter: hongyu guo
>Priority: Major
>
> for example
> {code:java}
> String sql = "select \"intField\", \"stringField\"\n"
> + "from arrowdata\n"
> + "where \"intField\"=12 or \"stringField\"='12'"; {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6299) Support JOIN in arrow adapter
hongyu guo created CALCITE-6299: --- Summary: Support JOIN in arrow adapter Key: CALCITE-6299 URL: https://issues.apache.org/jira/browse/CALCITE-6299 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6297) Support sub-queries in arrow adapter
hongyu guo created CALCITE-6297: --- Summary: Support sub-queries in arrow adapter Key: CALCITE-6297 URL: https://issues.apache.org/jira/browse/CALCITE-6297 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6298) Support UNION in arrow adapter
hongyu guo created CALCITE-6298: --- Summary: Support UNION in arrow adapter Key: CALCITE-6298 URL: https://issues.apache.org/jira/browse/CALCITE-6298 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6296) Support IS NULL in arrow adapter
hongyu guo created CALCITE-6296: --- Summary: Support IS NULL in arrow adapter Key: CALCITE-6296 URL: https://issues.apache.org/jira/browse/CALCITE-6296 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6295) Support IS NOT NULL in arrow adapter
hongyu guo created CALCITE-6295: --- Summary: Support IS NOT NULL in arrow adapter Key: CALCITE-6295 URL: https://issues.apache.org/jira/browse/CALCITE-6295 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6294) Support IN filter in arrow adapter
hongyu guo created CALCITE-6294: --- Summary: Support IN filter in arrow adapter Key: CALCITE-6294 URL: https://issues.apache.org/jira/browse/CALCITE-6294 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6293) Support OR condition for arrow adapter
hongyu guo created CALCITE-6293:
---
Summary: Support OR condition for arrow adapter
Key: CALCITE-6293
URL: https://issues.apache.org/jira/browse/CALCITE-6293
Project: Calcite
Issue Type: Sub-task
Reporter: hongyu guo
for example
{code:java}
String sql = "select \"intField\", \"stringField\"\n"
+ "from arrowdata\n"
+ "where \"intField\"=12 or \"stringField\"='12'"; {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6292) Support more arrow type
hongyu guo created CALCITE-6292:
---
Summary: Support more arrow type
Key: CALCITE-6292
URL: https://issues.apache.org/jira/browse/CALCITE-6292
Project: Calcite
Issue Type: Sub-task
Reporter: hongyu guo
All arrow type:
{code:java}
public static enum ArrowTypeID {
Null(Type.Null),
Struct(Type.Struct_),
List(Type.List),
LargeList(Type.LargeList),
FixedSizeList(Type.FixedSizeList),
Union(Type.Union),
Map(Type.Map),
Int(Type.Int),
FloatingPoint(Type.FloatingPoint),
Utf8(Type.Utf8),
LargeUtf8(Type.LargeUtf8),
Binary(Type.Binary),
LargeBinary(Type.LargeBinary),
FixedSizeBinary(Type.FixedSizeBinary),
Bool(Type.Bool),
Decimal(Type.Decimal),
Date(Type.Date),
Time(Type.Time),
Timestamp(Type.Timestamp),
Interval(Type.Interval),
Duration(Type.Duration),
NONE(Type.NONE);
}
{code}
we support now:
{code:java}
public static ArrowFieldType of(ArrowType arrowType) {
switch (arrowType.getTypeID()) {
case Int:
int bitWidth = ((ArrowType.Int) arrowType).getBitWidth();
switch (bitWidth) {
case 64:
return LONG;
case 32:
return INT;
case 16:
return SHORT;
case 8:
return BYTE;
default:
throw new IllegalArgumentException("Unsupported Int bit width: " +
bitWidth);
}
case Bool:
return BOOLEAN;
case Utf8:
return STRING;
case FloatingPoint:
FloatingPointPrecision precision = ((ArrowType.FloatingPoint)
arrowType).getPrecision();
switch (precision) {
case SINGLE:
return FLOAT;
case DOUBLE:
return DOUBLE;
default:
throw new IllegalArgumentException("Unsupported Floating point
precision: " + precision);
}
case Date:
return DATE;
case Decimal:
return DECIMAL;
default:
throw new IllegalArgumentException("Unsupported type: " + arrowType);
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6291) Support converting ArrowTable to Queryable
hongyu guo created CALCITE-6291: --- Summary: Support converting ArrowTable to Queryable Key: CALCITE-6291 URL: https://issues.apache.org/jira/browse/CALCITE-6291 Project: Calcite Issue Type: Sub-task Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo reassigned CALCITE-2040: --- Assignee: hongyu guo > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo reassigned CALCITE-2040: --- Assignee: hongyu guo (was: Julian Hyde) > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: hongyu guo >Priority: Major > Labels: pull-request-available > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6257) StarRocks dialect implementation
[ https://issues.apache.org/jira/browse/CALCITE-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo resolved CALCITE-6257. - Fix Version/s: 1.37.0 Resolution: Fixed Resloved in [6ba31302|https://github.com/apache/calcite/commit/6ba31302be35f19e0ccf289a1c6705a79bf87587]. Thanks [~eveywu]! > StarRocks dialect implementation > > > Key: CALCITE-6257 > URL: https://issues.apache.org/jira/browse/CALCITE-6257 > Project: Calcite > Issue Type: New Feature >Reporter: EveyWu >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > > StarRocks has an MPP architecture and is equipped with a fully vectorized > execution engine, a columnar storage engine that supports real-time updates. > [https://docs.starrocks.io/docs/introduction/StarRocks_intro/] > At present, StarRocks has a wide range of applications. Implementing dialect > in Calcite will be valuable for many bi-services. > It closely follows MySQL but has enough differences to warrant a separate > dialect in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6254) Support table function calls in FROM clause without TABLE() wrapper
[
https://issues.apache.org/jira/browse/CALCITE-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6254.
-
Fix Version/s: 1.37.0
Resolution: Fixed
Resloved in
[331f985|https://github.com/apache/calcite/commit/331f9859445b112e9ffc03e3c6aeff808e023034].
Thanks for your contribution, [~barrkel]!
> Support table function calls in FROM clause without TABLE() wrapper
> ---
>
> Key: CALCITE-6254
> URL: https://issues.apache.org/jira/browse/CALCITE-6254
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.36.0
>Reporter: Barry Kelly
>Priority: Minor
> Labels: parser, pull-request-available
> Fix For: 1.37.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently, when selecting from a table function, the function call needs to
> be wrapped in TABLE() like this:
> {{ SELECT * FROM TABLE(table_func('args'));}}
> Many dialects (SQL Server, PostgreSQL, BigQuery) do not require this:
> {{ SELECT * FROM table_func('args');}}
> The current Calcite grammar can be extended to permit this syntax without
> conflicting with other productions.
> There is a close call with the dynamic columns feature used by
> [Phoenix|https://phoenix.apache.org/dynamic_columns.html]
> {{ SELECT * FROM EventLog(lastGCTime TIME)}}
> It can be disambiguated with 3 tokens of lookahead, seeing past the '(' and
> identifier to the presence of a type production. The table extension clause
> for dynamic columns requires at least one field, so the empty case of '()' is
> not ambiguous either.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-2040) Create adapter for Apache Arrow
[ https://issues.apache.org/jira/browse/CALCITE-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819076#comment-17819076 ] hongyu guo commented on CALCITE-2040: - Does anyone has any suggestion for [https://github.com/apache/calcite/pull/3666] ? Some discussion can be found in mail thread [https://lists.apache.org/thread/z4qzgnzov7sdjorjvkx8w35m376dwm3y] > Create adapter for Apache Arrow > --- > > Key: CALCITE-2040 > URL: https://issues.apache.org/jira/browse/CALCITE-2040 > Project: Calcite > Issue Type: Bug >Reporter: Julian Hyde >Assignee: Julian Hyde >Priority: Major > Labels: pull-request-available > Attachments: arrow_data.py > > Time Spent: 1.5h > Remaining Estimate: 0h > > Create an adapter for [Apache Arrow|http://arrow.apache.org/]. This would > allow people to execute SQL statements, via JDBC or ODBC, on data stored in > Arrow in-memory format. > Since Arrow is an in-memory format, it is not as straightforward as reading, > say, CSV files using the file adapter: an Arrow data set does not have a URL. > (Unless we use Arrow's > [Feather|https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/] > format, or use an in-memory file system such as Alluxio.) So we would need > to devise a way of addressing Arrow data sets. > Also, since Arrow is an extremely efficient format for processing data, it > would also be good to have Arrow as a calling convention. That is, > implementations of relational operators such as Filter, Project, Aggregate in > addition to just TableScan. > Lastly, when we have an Arrow convention, if we build adapters for file > formats (for instance the bioinformatics formats SAM, VCF, FASTQ discussed in > CALCITE-2025) it would make a lot of sense to translate those formats > directly into Arrow (applying simple projects and filters first if > applicable). Those adapters would belong as a "contrib" module in the Arrow > project better than in Calcite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6258) Map value constructor is unparsed incorrectly for PrestoSqlDialect
[
https://issues.apache.org/jira/browse/CALCITE-6258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6258.
-
Fix Version/s: 1.37.0
Resolution: Fixed
Resolved in
[4369e8|https://github.com/apache/calcite/commit/4369e88b15a59333c05abeda38cf8799d61a8842].
Thanks [~eveywu].
> Map value constructor is unparsed incorrectly for PrestoSqlDialect
> --
>
> Key: CALCITE-6258
> URL: https://issues.apache.org/jira/browse/CALCITE-6258
> Project: Calcite
> Issue Type: Bug
>Reporter: EveyWu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> Presto Map function：{{{}map{}}}({_}array{_}, {_}array{_})
> https://teradata.github.io/presto/docs/current/functions/map.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-6241) Enable a few existing functions to Spark library
[ https://issues.apache.org/jira/browse/CALCITE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo resolved CALCITE-6241. - Fix Version/s: 1.37.0 Resolution: Fixed Resolved in [b5091fe94|https://github.com/apache/calcite/commit/b5091fe94d38ee952bde116ba6e04f00deb7308a]. Thanks for your contribution! [~eveywu] > Enable a few existing functions to Spark library > > > Key: CALCITE-6241 > URL: https://issues.apache.org/jira/browse/CALCITE-6241 > Project: Calcite > Issue Type: Improvement >Reporter: EveyWu >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > Attachments: image-2024-02-07-07-08-59-868.png > > > Add Spark as a supported library for functions that have already been > implemented for other libraries. > Spark Functions > Link：[https://spark.apache.org/docs/latest/api/sql/index.html|https://spark.apache.org/docs/latest/api/sql/index.html#rtrim] > Enable function List： > * DECODE > * NVL > * IFNULL > * LEN > * LENGTH > * LPAD > * RPAD > * LTRIM > * RTRIM > * ENDSWITH > * STARTSWITH > * GREATEST > * LEAST > * TRANSLATE > * BOOL_AND > * BOOL_OR > * DATE_FROM_UNIX_DATE > * UNIX_DATE > * LEFT > * REPEAT > * RIGHT > * SPACE > * TIMESTAMP_SECONDS > * TIMESTAMP_MILLIS > * TIMESTAMP_MICROS > * UNIX_SECONDS > * UNIX_MILLIS > * UNIX_MICROS > * MD5 > * SHA1 > * POW > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6223) Add MAP_CONTAINS_KEY function (enabled in Spark library)
[
https://issues.apache.org/jira/browse/CALCITE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6223.
-
Resolution: Fixed
Resolved in
[cf91b789d|https://github.com/apache/calcite/commit/cf91b789d174dd72119cfbe46a646deeab753c7f].
Thanks for your contribution! [~caicancai]
> Add MAP_CONTAINS_KEY function (enabled in Spark library)
>
>
> Key: CALCITE-6223
> URL: https://issues.apache.org/jira/browse/CALCITE-6223
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.36.0
>Reporter: Caican Cai
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> Add MAP_CONTAINS_KEY function (enabled in Spark library)
> [https://spark.apache.org/docs/3.4.0/api/sql/index.html#map_contains_key]
>
>
> {code:java}
> > SELECT map_contains_key(map(1, 'a', 2, 'b'), 1);
> true
> > SELECT map_contains_key(map(1, 'a', 2, 'b'), 3);
> false {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6242) The "exists" library function throws a "param not found" error when a column is used in lambda evaluation logic.
[
https://issues.apache.org/jira/browse/CALCITE-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17814021#comment-17814021
]
hongyu guo edited comment on CALCITE-6242 at 2/4/24 3:26 AM:
-
Thanks for reporting this bug! In CALCITE-3679, referencing columns is not
supported in lambda functions, and we need to add this feature.
was (Author: JIRAUSER300840):
Thanks for report this! In CALCITE-3679, referencing columns is not supported
in lambda functions, and we need to add this feature.
> The "exists" library function throws a "param not found" error when a column
> is used in lambda evaluation logic.
>
>
> Key: CALCITE-6242
> URL: https://issues.apache.org/jira/browse/CALCITE-6242
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Hanumath Rao Maduri
>Priority: Major
>
> The following query is throwing a "param not found" in calcite whereas the
> same query works fine in apache spark.
> {code:java}
> select *
> from (select array(1, 2, 3) as arr) as t1 inner join
> (select 1 as v) as t2 on "EXISTS"(arr, x -> x = t2.v); {code}
> The following error reported for the above query
> {code:java}
> > java.sql.SQLException: Error while executing SQL "select *
> > from (select array(1, 2, 3) as arr) as t1 inner join
> > (select 1 as v) as t2 on "EXISTS"(arr, x -> x = t2.v)": From line 3,
> > column 54 to line 3, column 57: Param 'T2.V' not found in lambda expression
> > '`X` -> `X` = `T2`.`V`' {code}
> Steps to repro:
> 1. Place the query in lambda.iq
> 2. Run CoreQuidemTest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6242) The "exists" library function throws a "param not found" error when a column is used in lambda evaluation logic.
[
https://issues.apache.org/jira/browse/CALCITE-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17814021#comment-17814021
]
hongyu guo commented on CALCITE-6242:
-
Thanks for report this! In CALCITE-3679, referencing columns is not supported
in lambda functions, and we need to add this feature.
> The "exists" library function throws a "param not found" error when a column
> is used in lambda evaluation logic.
>
>
> Key: CALCITE-6242
> URL: https://issues.apache.org/jira/browse/CALCITE-6242
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Hanumath Rao Maduri
>Priority: Major
>
> The following query is throwing a "param not found" in calcite whereas the
> same query works fine in apache spark.
> {code:java}
> select *
> from (select array(1, 2, 3) as arr) as t1 inner join
> (select 1 as v) as t2 on "EXISTS"(arr, x -> x = t2.v); {code}
> The following error reported for the above query is
> {code:java}
> > java.sql.SQLException: Error while executing SQL "select *
> > from (select array(1, 2, 3) as arr) as t1 inner join
> > (select 1 as v) as t2 on "EXISTS"(arr, x -> x = t2.v)": From line 3,
> > column 54 to line 3, column 57: Param 'T2.V' not found in lambda expression
> > '`X` -> `X` = `T2`.`V`' {code}
> Steps to repro:
> 1. Place the query in lambda.iq
> 2. Run CoreQuidemTest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6242) The "exists" library function throws a "param not found" error when a column is used in lambda evaluation logic.
[
https://issues.apache.org/jira/browse/CALCITE-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo updated CALCITE-6242:
Component/s: core
(was: spark)
> The "exists" library function throws a "param not found" error when a column
> is used in lambda evaluation logic.
>
>
> Key: CALCITE-6242
> URL: https://issues.apache.org/jira/browse/CALCITE-6242
> Project: Calcite
> Issue Type: Bug
> Components: core
>Reporter: Hanumath Rao Maduri
>Priority: Major
>
> The following query is throwing a "param not found" in calcite whereas the
> same query works fine in apache spark.
> {code:java}
> select *
> from (select array(1, 2, 3) as arr) as t1 inner join
> (select 1 as v) as t2 on "EXISTS"(arr, x -> x = t2.v); {code}
> The following error reported for the above query is
> {code:java}
> > java.sql.SQLException: Error while executing SQL "select *
> > from (select array(1, 2, 3) as arr) as t1 inner join
> > (select 1 as v) as t2 on "EXISTS"(arr, x -> x = t2.v)": From line 3,
> > column 54 to line 3, column 57: Param 'T2.V' not found in lambda expression
> > '`X` -> `X` = `T2`.`V`' {code}
> Steps to repro:
> 1. Place the query in lambda.iq
> 2. Run CoreQuidemTest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-6231) JDBC adapter generates "UNNEST" when it should generate "UNNEST ... WITH ORDINALITY"
[
https://issues.apache.org/jira/browse/CALCITE-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6231.
-
Fix Version/s: 1.37.0
Resolution: Fixed
> JDBC adapter generates "UNNEST" when it should generate "UNNEST ... WITH
> ORDINALITY"
>
>
> Key: CALCITE-6231
> URL: https://issues.apache.org/jira/browse/CALCITE-6231
> Project: Calcite
> Issue Type: Bug
>Reporter: EveyWu
>Assignee: EveyWu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> The syntax UNNEST() WITH ORDINALITY is missing the ORDINALITY keyword.
> For example sql：
> {code:java}
> select did + 1 from
> unnest(select collect("department_id") as deptid from "department")
> with ordinality as t(did, pos){code}
>
> current planned sql：
> {code:java}
> SELECT DEPTID + 1 FROM UNNEST (
> SELECT COLLECT("department_id") AS "DEPTID" FROM "foodmart"."department")
> AS "t0" ("DEPTID", "ORDINALITY") {code}
>
> fixed planned sql：
> {code:java}
> SELECT "DEPTID" + 1 FROM UNNEST (
> SELECT COLLECT("department_id") AS "DEPTID" FROM "foodmart"."department")
> WITH ORDINALITY AS "t0" ("DEPTID", "ORDINALITY") {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6231) JDBC adapter generates "UNNEST" when it should generate "UNNEST ... WITH ORDINALITY"
[
https://issues.apache.org/jira/browse/CALCITE-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812562#comment-17812562
]
hongyu guo edited comment on CALCITE-6231 at 1/31/24 7:15 AM:
--
Fixed in
[351ddeb4|https://github.com/apache/calcite/commit/351ddeb47b8dfb5c196c563920290a79575e9864].
[~eveywu] Thanks for your contribution!
was (Author: JIRAUSER300840):
Fix in
[351ddeb4|https://github.com/apache/calcite/commit/351ddeb47b8dfb5c196c563920290a79575e9864].
[~eveywu] Thanks for your contribution!
> JDBC adapter generates "UNNEST" when it should generate "UNNEST ... WITH
> ORDINALITY"
>
>
> Key: CALCITE-6231
> URL: https://issues.apache.org/jira/browse/CALCITE-6231
> Project: Calcite
> Issue Type: Bug
>Reporter: EveyWu
>Assignee: EveyWu
>Priority: Major
> Labels: pull-request-available
>
> The syntax UNNEST() WITH ORDINALITY is missing the ORDINALITY keyword.
> For example sql：
> {code:java}
> select did + 1 from
> unnest(select collect("department_id") as deptid from "department")
> with ordinality as t(did, pos){code}
>
> current planned sql：
> {code:java}
> SELECT DEPTID + 1 FROM UNNEST (
> SELECT COLLECT("department_id") AS "DEPTID" FROM "foodmart"."department")
> AS "t0" ("DEPTID", "ORDINALITY") {code}
>
> fixed planned sql：
> {code:java}
> SELECT "DEPTID" + 1 FROM UNNEST (
> SELECT COLLECT("department_id") AS "DEPTID" FROM "foodmart"."department")
> WITH ORDINALITY AS "t0" ("DEPTID", "ORDINALITY") {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6231) JDBC adapter generates "UNNEST" when it should generate "UNNEST ... WITH ORDINALITY"
[
https://issues.apache.org/jira/browse/CALCITE-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812562#comment-17812562
]
hongyu guo commented on CALCITE-6231:
-
Fix in
[351ddeb4|https://github.com/apache/calcite/commit/351ddeb47b8dfb5c196c563920290a79575e9864].
[~eveywu] Thanks for your contribution!
> JDBC adapter generates "UNNEST" when it should generate "UNNEST ... WITH
> ORDINALITY"
>
>
> Key: CALCITE-6231
> URL: https://issues.apache.org/jira/browse/CALCITE-6231
> Project: Calcite
> Issue Type: Bug
>Reporter: EveyWu
>Assignee: EveyWu
>Priority: Major
> Labels: pull-request-available
>
> The syntax UNNEST() WITH ORDINALITY is missing the ORDINALITY keyword.
> For example sql：
> {code:java}
> select did + 1 from
> unnest(select collect("department_id") as deptid from "department")
> with ordinality as t(did, pos){code}
>
> current planned sql：
> {code:java}
> SELECT DEPTID + 1 FROM UNNEST (
> SELECT COLLECT("department_id") AS "DEPTID" FROM "foodmart"."department")
> AS "t0" ("DEPTID", "ORDINALITY") {code}
>
> fixed planned sql：
> {code:java}
> SELECT "DEPTID" + 1 FROM UNNEST (
> SELECT COLLECT("department_id") AS "DEPTID" FROM "foodmart"."department")
> WITH ORDINALITY AS "t0" ("DEPTID", "ORDINALITY") {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (CALCITE-6230) Jira case collections
[ https://issues.apache.org/jira/browse/CALCITE-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo updated CALCITE-6230: Description: Related thread: [Collection of some jira cases|https://lists.apache.org/thread/tmtql0yt3nyn6obdw677pm6lyxljpg0w] syntax: CALCITE-5852: MERGE INTO CALCITE-5301: AT TIME ZONE CALCITE-5386: LIKE (ANY | SOME | ALL) CALCITE-5216: Cannot parse parenthesized nested WITH clause CALCITE-5205: Supports hint option as string and numeric literal CALCITE-5168: Allow AS after parenthesized JOIN CALCITE-5084: Support ROWS syntax with TABLESAMPLE CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY statement CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands data type: CALCITE-5346: type aliases CALCITE-4918: Add a VARIANT data type rule and optimization: CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= CALCITE-4052: Enable Top-down Optimization dialect: CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) function: CALCITE-5809: enable already available functions in Apache Spark Library CALCITE-5087: Support bitwise operators CALCITE-4521: Support User Defined Table-valued Function CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not unique CALCITE-3683: Enhanced MATH Function CALCITE-3646: MySQL compression functions CALCITE-2031: Implement more OpenGIS functions CALCITE-2871: Implement JSON_TABLE table function was: Relate thread: [Collection of some jira cases|https://lists.apache.org/thread/tmtql0yt3nyn6obdw677pm6lyxljpg0w] syntax: CALCITE-5852: MERGE INTO CALCITE-5301: AT TIME ZONE CALCITE-5386: LIKE (ANY | SOME | ALL) CALCITE-5216: Cannot parse parenthesized nested WITH clause CALCITE-5205: Supports hint option as string and numeric literal CALCITE-5168: Allow AS after parenthesized JOIN CALCITE-5084: Support ROWS syntax with TABLESAMPLE CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY statement CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands data type: CALCITE-5346: type aliases CALCITE-4918: Add a VARIANT data type rule and optimization: CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= CALCITE-4052: Enable Top-down Optimization dialect: CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) function: CALCITE-5809: enable already available functions in Apache Spark Library CALCITE-5087: Support bitwise operators CALCITE-4521: Support User Defined Table-valued Function CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not unique CALCITE-3683: Enhanced MATH Function CALCITE-3646: MySQL compression functions CALCITE-2031: Implement more OpenGIS functions CALCITE-2871: Implement JSON_TABLE table function > Jira case collections > - > > Key: CALCITE-6230 > URL: https://issues.apache.org/jira/browse/CALCITE-6230 > Project: Calcite > Issue Type: Task >Reporter: hongyu guo >Priority: Minor > > Related thread: [Collection of some jira > cases|https://lists.apache.org/thread/tmtql0yt3nyn6obdw677pm6lyxljpg0w] > syntax: > CALCITE-5852: MERGE INTO > CALCITE-5301: AT TIME ZONE > CALCITE-5386: LIKE (ANY | SOME | ALL) > CALCITE-5216: Cannot parse parenthesized nested WITH clause > CALCITE-5205: Supports hint option as string and numeric literal > CALCITE-5168: Allow AS after parenthesized JOIN > CALCITE-5084: Support ROWS syntax with TABLESAMPLE > CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) > CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ > CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY > statement > CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax > CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands > data type: > CALCITE-5346: type aliases > CALCITE-4918: Add a VARIANT data type > rule and optimization: > CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= > CALCITE-4052: Enable Top-down Optimization > dialect: > CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) > function: > CALCITE-5809: enable already available functions in Apache Spark Library > CALCITE-5087: Support bitwise operators > CALCITE-4521: Support User Defined Table-valued Function > CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not
[jira] [Updated] (CALCITE-6230) Jira case collections
[ https://issues.apache.org/jira/browse/CALCITE-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo updated CALCITE-6230: Description: Relate thread: [Collection of some jira cases|https://lists.apache.org/thread/tmtql0yt3nyn6obdw677pm6lyxljpg0w] syntax: CALCITE-5852: MERGE INTO CALCITE-5301: AT TIME ZONE CALCITE-5386: LIKE (ANY | SOME | ALL) CALCITE-5216: Cannot parse parenthesized nested WITH clause CALCITE-5205: Supports hint option as string and numeric literal CALCITE-5168: Allow AS after parenthesized JOIN CALCITE-5084: Support ROWS syntax with TABLESAMPLE CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY statement CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands data type: CALCITE-5346: type aliases CALCITE-4918: Add a VARIANT data type rule and optimization: CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= CALCITE-4052: Enable Top-down Optimization dialect: CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) function: CALCITE-5809: enable already available functions in Apache Spark Library CALCITE-5087: Support bitwise operators CALCITE-4521: Support User Defined Table-valued Function CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not unique CALCITE-3683: Enhanced MATH Function CALCITE-3646: MySQL compression functions CALCITE-2031: Implement more OpenGIS functions CALCITE-2871: Implement JSON_TABLE table function was: syntax: CALCITE-5852: MERGE INTO CALCITE-5301: AT TIME ZONE CALCITE-5386: LIKE (ANY | SOME | ALL) CALCITE-5216: Cannot parse parenthesized nested WITH clause CALCITE-5205: Supports hint option as string and numeric literal CALCITE-5168: Allow AS after parenthesized JOIN CALCITE-5084: Support ROWS syntax with TABLESAMPLE CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY statement CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands data type: CALCITE-5346: type aliases CALCITE-4918: Add a VARIANT data type rule and optimization: CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= CALCITE-4052: Enable Top-down Optimization dialect: CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) function: CALCITE-5809: enable already available functions in Apache Spark Library CALCITE-5087: Support bitwise operators CALCITE-4521: Support User Defined Table-valued Function CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not unique CALCITE-3683: Enhanced MATH Function CALCITE-3646: MySQL compression functions CALCITE-2031: Implement more OpenGIS functions CALCITE-2871: Implement JSON_TABLE table function > Jira case collections > - > > Key: CALCITE-6230 > URL: https://issues.apache.org/jira/browse/CALCITE-6230 > Project: Calcite > Issue Type: Task >Reporter: hongyu guo >Priority: Minor > > Relate thread: [Collection of some jira > cases|https://lists.apache.org/thread/tmtql0yt3nyn6obdw677pm6lyxljpg0w] > syntax: > CALCITE-5852: MERGE INTO > CALCITE-5301: AT TIME ZONE > CALCITE-5386: LIKE (ANY | SOME | ALL) > CALCITE-5216: Cannot parse parenthesized nested WITH clause > CALCITE-5205: Supports hint option as string and numeric literal > CALCITE-5168: Allow AS after parenthesized JOIN > CALCITE-5084: Support ROWS syntax with TABLESAMPLE > CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) > CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ > CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY > statement > CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax > CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands > data type: > CALCITE-5346: type aliases > CALCITE-4918: Add a VARIANT data type > rule and optimization: > CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= > CALCITE-4052: Enable Top-down Optimization > dialect: > CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) > function: > CALCITE-5809: enable already available functions in Apache Spark Library > CALCITE-5087: Support bitwise operators > CALCITE-4521: Support User Defined Table-valued Function > CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not > unique > CALCITE-3683: Enhanced MATH Function > CALCITE-3646: MySQL compression functions > CALCITE-2031:
[jira] [Created] (CALCITE-6230) Jira case collections
hongyu guo created CALCITE-6230: --- Summary: Jira case collections Key: CALCITE-6230 URL: https://issues.apache.org/jira/browse/CALCITE-6230 Project: Calcite Issue Type: Task Reporter: hongyu guo syntax: CALCITE-5852: MERGE INTO CALCITE-5301: AT TIME ZONE CALCITE-5386: LIKE (ANY | SOME | ALL) CALCITE-5216: Cannot parse parenthesized nested WITH clause CALCITE-5205: Supports hint option as string and numeric literal CALCITE-5168: Allow AS after parenthesized JOIN CALCITE-5084: Support ROWS syntax with TABLESAMPLE CALCITE-5066: Support variables with “@" and “@@" prefixes (like MySQL) CALCITE-4705: Support hints like /*+ skewjoin(a(c0, c1)) */ CALCITE-4455: Babel parser support Spark INSERT OVERWRITE TABLE/DIRECTORY statement CALCITE-3970: Table-valued function TUMBLE uses non-standard syntax CALCITE-5681: Support authorization via GRANT and REVOKE DDL commands data type: CALCITE-5346: type aliases CALCITE-4918: Add a VARIANT data type rule and optimization: CALCITE-4843: optimize ALL,SOME sub-query list when op is the <, <=,> or >= CALCITE-4052: Enable Top-down Optimization dialect: CALCITE-4782: Allow 'CAST(numeric AS BOOLEAN)' (if enabled by conformance) function: CALCITE-5809: enable already available functions in Apache Spark Library CALCITE-5087: Support bitwise operators CALCITE-4521: Support User Defined Table-valued Function CALCITE-4484: Add UNIQUE_VALUE(x) aggregate function, that throws if x is not unique CALCITE-3683: Enhanced MATH Function CALCITE-3646: MySQL compression functions CALCITE-2031: Implement more OpenGIS functions CALCITE-2871: Implement JSON_TABLE table function -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6229) Upgrade json-path version to 2.9.0
[ https://issues.apache.org/jira/browse/CALCITE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo resolved CALCITE-6229. - Fix Version/s: 1.37.0 Resolution: Fixed Fixed in [e17098d47|https://github.com/apache/calcite/commit/e17098d47f3c31e4d90cc17e6e1da1175bf49ae4]. [~wnoble] Thanks for your contribution! > Upgrade json-path version to 2.9.0 > -- > > Key: CALCITE-6229 > URL: https://issues.apache.org/jira/browse/CALCITE-6229 > Project: Calcite > Issue Type: Bug >Reporter: Will Noble >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > > There is a CVE: https://osv.dev/vulnerability/GHSA-pfh2-hfmq-phg5 > Probably not critical for most users since it would be strange for > unsanitized user input to make it into the JSON parser through Calcite, but > should be easy to patch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6213) The default behavior of NullCollation in Presto is LAST
[
https://issues.apache.org/jira/browse/CALCITE-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6213.
-
Fix Version/s: 1.37.0
Assignee: EveyWu
Resolution: Fixed
> The default behavior of NullCollation in Presto is LAST
> ---
>
> Key: CALCITE-6213
> URL: https://issues.apache.org/jira/browse/CALCITE-6213
> Project: Calcite
> Issue Type: Bug
>Reporter: EveyWu
>Assignee: EveyWu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
> Attachments: image-2024-01-19-23-54-26-383.png,
> image-2024-01-19-23-57-00-252.png
>
>
> [https://prestodb.io/docs/0.285/sql/select.html]
> Presto default null ordering is NULLS LAST
> !image-2024-01-19-23-54-26-383.png|width=511,height=168!
>
> Demo：select by presto with nulls last
> {code:java}
> select * from product order by brand_name; {code}
> !image-2024-01-19-23-57-00-252.png|width=517,height=83!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6213) The default behavior of NullCollation in Presto is LAST
[
https://issues.apache.org/jira/browse/CALCITE-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810301#comment-17810301
]
hongyu guo commented on CALCITE-6213:
-
Fixed in
[aca7f02d|https://github.com/apache/calcite/commit/aca7f02dfd510297bd56b07ab94d41033a5146b4].
[~eveywu] Very nice work, thanks for your contribution!
> The default behavior of NullCollation in Presto is LAST
> ---
>
> Key: CALCITE-6213
> URL: https://issues.apache.org/jira/browse/CALCITE-6213
> Project: Calcite
> Issue Type: Bug
>Reporter: EveyWu
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-01-19-23-54-26-383.png,
> image-2024-01-19-23-57-00-252.png
>
>
> [https://prestodb.io/docs/0.285/sql/select.html]
> Presto default null ordering is NULLS LAST
> !image-2024-01-19-23-54-26-383.png|width=511,height=168!
>
> Demo：select by presto with nulls last
> {code:java}
> select * from product order by brand_name; {code}
> !image-2024-01-19-23-57-00-252.png|width=517,height=83!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6184) Add checkNullTest on SqlOperatorTest
[ https://issues.apache.org/jira/browse/CALCITE-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804613#comment-17804613 ] hongyu guo commented on CALCITE-6184: - Fixed in [https://github.com/apache/calcite/commit/294e42f0fb30963dcb452f4f3b5ed1627a47a296] [~caicancai] , thanks for your contribution! > Add checkNullTest on SqlOperatorTest > > > Key: CALCITE-6184 > URL: https://issues.apache.org/jira/browse/CALCITE-6184 > Project: Calcite > Issue Type: Test > Components: tests >Affects Versions: 1.36.0 >Reporter: Caican Cai >Assignee: Caican Cai >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > > Some tests in calcite's SqlOperatorTest lack checknull. I think the checknull > test should be added. > 1. Conducive to the learning of new developers > 2. Based on the checknull test, we may be able to find some bugs in calcite’s > sqoperator. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
[ https://issues.apache.org/jira/browse/CALCITE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801085#comment-17801085 ] hongyu guo edited comment on CALCITE-6183 at 1/2/24 7:07 AM: - Fixed in [d7e171e|https://github.com/apache/calcite/commit/d7e171e3b0296cdbec2f9569faf0c94351e10700]. was (Author: JIRAUSER300840): Resolved in [d7e171e|https://github.com/apache/calcite/commit/d7e171e3b0296cdbec2f9569faf0c94351e10700]. > The second parameter of RexProgramBuilder#registerInternal is always false > -- > > Key: CALCITE-6183 > URL: https://issues.apache.org/jira/browse/CALCITE-6183 > Project: Calcite > Issue Type: Improvement >Reporter: hongyu guo >Assignee: hongyu guo >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
[ https://issues.apache.org/jira/browse/CALCITE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo updated CALCITE-6183: Fix Version/s: 1.37.0 > The second parameter of RexProgramBuilder#registerInternal is always false > -- > > Key: CALCITE-6183 > URL: https://issues.apache.org/jira/browse/CALCITE-6183 > Project: Calcite > Issue Type: Improvement >Reporter: hongyu guo >Assignee: hongyu guo >Priority: Minor > Labels: pull-request-available > Fix For: 1.37.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
[ https://issues.apache.org/jira/browse/CALCITE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801085#comment-17801085 ] hongyu guo edited comment on CALCITE-6183 at 12/29/23 2:35 AM: --- Resolved in [d7e171e|https://github.com/apache/calcite/commit/d7e171e3b0296cdbec2f9569faf0c94351e10700]. was (Author: JIRAUSER300840): Fixed in [d7e171e|https://github.com/apache/calcite/commit/d7e171e3b0296cdbec2f9569faf0c94351e10700]. > The second parameter of RexProgramBuilder#registerInternal is always false > -- > > Key: CALCITE-6183 > URL: https://issues.apache.org/jira/browse/CALCITE-6183 > Project: Calcite > Issue Type: Improvement >Reporter: hongyu guo >Assignee: hongyu guo >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
[ https://issues.apache.org/jira/browse/CALCITE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo resolved CALCITE-6183. - Resolution: Resolved > The second parameter of RexProgramBuilder#registerInternal is always false > -- > > Key: CALCITE-6183 > URL: https://issues.apache.org/jira/browse/CALCITE-6183 > Project: Calcite > Issue Type: Improvement >Reporter: hongyu guo >Assignee: hongyu guo >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
[ https://issues.apache.org/jira/browse/CALCITE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801085#comment-17801085 ] hongyu guo edited comment on CALCITE-6183 at 12/29/23 2:35 AM: --- Fixed in [d7e171e|https://github.com/apache/calcite/commit/d7e171e3b0296cdbec2f9569faf0c94351e10700]. was (Author: JIRAUSER300840): Fixed in [d7e171e|d7e171e3b0296cdbec2f9569faf0c94351e10700]. > The second parameter of RexProgramBuilder#registerInternal is always false > -- > > Key: CALCITE-6183 > URL: https://issues.apache.org/jira/browse/CALCITE-6183 > Project: Calcite > Issue Type: Improvement >Reporter: hongyu guo >Assignee: hongyu guo >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
[ https://issues.apache.org/jira/browse/CALCITE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801085#comment-17801085 ] hongyu guo commented on CALCITE-6183: - Fixed in [d7e171e|d7e171e3b0296cdbec2f9569faf0c94351e10700]. > The second parameter of RexProgramBuilder#registerInternal is always false > -- > > Key: CALCITE-6183 > URL: https://issues.apache.org/jira/browse/CALCITE-6183 > Project: Calcite > Issue Type: Improvement >Reporter: hongyu guo >Assignee: hongyu guo >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
[ https://issues.apache.org/jira/browse/CALCITE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongyu guo reassigned CALCITE-6183: --- Assignee: hongyu guo > The second parameter of RexProgramBuilder#registerInternal is always false > -- > > Key: CALCITE-6183 > URL: https://issues.apache.org/jira/browse/CALCITE-6183 > Project: Calcite > Issue Type: Improvement >Reporter: hongyu guo >Assignee: hongyu guo >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6183) The second parameter of RexProgramBuilder#registerInternal is always false
hongyu guo created CALCITE-6183: --- Summary: The second parameter of RexProgramBuilder#registerInternal is always false Key: CALCITE-6183 URL: https://issues.apache.org/jira/browse/CALCITE-6183 Project: Calcite Issue Type: Improvement Reporter: hongyu guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (CALCITE-5956) Support parsing non-standard collection types
[
https://issues.apache.org/jira/browse/CALCITE-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo closed CALCITE-5956.
---
Resolution: Duplicate
Duplicates with CALCITE-5570, so I have closed this case.
> Support parsing non-standard collection types
> -
>
> Key: CALCITE-5956
> URL: https://issues.apache.org/jira/browse/CALCITE-5956
> Project: Calcite
> Issue Type: New Feature
> Components: core
>Affects Versions: 1.35.0
>Reporter: hongyu guo
>Assignee: hongyu guo
>Priority: Major
>
> Calcite support SQL standard collection types. For example:
> {code:sql}
> INTEGER ARRAY;
> INTEGER ARRAY ARRAY;
> VARCHAR(5) MULTISET;
> INTEGER MULTISET ARRAY;{code}
> Many DBMS support a dialect of {{ARRAY}} and {{MULITSET}},
> I think Calcite can also support this dialect in core Parser.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CALCITE-6116) Add EXISTS function (enabled in Spark library)
[
https://issues.apache.org/jira/browse/CALCITE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800926#comment-17800926
]
hongyu guo commented on CALCITE-6116:
-
Fixed in
[e877885|https://github.com/apache/calcite/commit/e877885ed90127a4cadb25f1b718f91375fe6164].
[~julianhyde] and [~mbudiu], thanks for your review!
> Add EXISTS function (enabled in Spark library)
> --
>
> Key: CALCITE-6116
> URL: https://issues.apache.org/jira/browse/CALCITE-6116
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.36.0
>Reporter: hongyu guo
>Assignee: hongyu guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> exists(expr, pred) - Tests whether a predicate holds for one or more elements
> in the array.
> {code:sql}
> > SELECT `EXISTS`(array(1, 2, 3), x -> x % 2 == 0);
> true
> > SELECT `EXISTS`(array(1, 2, 3), x -> x % 2 == 10);
> false
> > SELECT `EXISTS`(array(1, null, 3), x -> x % 2 == 0);
> NULL
> > SELECT `EXISTS`(array(0, null, 2, 3, null), x -> x IS NULL);
> true
> > SELECT `EXISTS`(array(1, 2, 3), x -> x IS NULL);
> false
> {code}
> In Calcite, EXISTS is a keyword, so we need to specify the function with back
> quotes.
> Moreover, `EXISTS` is a higher-order function, and if we want to support
> higher-order functions in Calcite, we must first support lambda expressions
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (CALCITE-6116) Add EXISTS function (enabled in Spark library)
[
https://issues.apache.org/jira/browse/CALCITE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hongyu guo resolved CALCITE-6116.
-
Resolution: Fixed
> Add EXISTS function (enabled in Spark library)
> --
>
> Key: CALCITE-6116
> URL: https://issues.apache.org/jira/browse/CALCITE-6116
> Project: Calcite
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.36.0
>Reporter: hongyu guo
>Assignee: hongyu guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.37.0
>
>
> exists(expr, pred) - Tests whether a predicate holds for one or more elements
> in the array.
> {code:sql}
> > SELECT `EXISTS`(array(1, 2, 3), x -> x % 2 == 0);
> true
> > SELECT `EXISTS`(array(1, 2, 3), x -> x % 2 == 10);
> false
> > SELECT `EXISTS`(array(1, null, 3), x -> x % 2 == 0);
> NULL
> > SELECT `EXISTS`(array(0, null, 2, 3, null), x -> x IS NULL);
> true
> > SELECT `EXISTS`(array(1, 2, 3), x -> x IS NULL);
> false
> {code}
> In Calcite, EXISTS is a keyword, so we need to specify the function with back
> quotes.
> Moreover, `EXISTS` is a higher-order function, and if we want to support
> higher-order functions in Calcite, we must first support lambda expressions
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
