[jira] [Created] (CALCITE-6410) dateadd(MONTH, 3, date '2016-02-24') parsing failed
Caican Cai created CALCITE-6410:
---
Summary: dateadd(MONTH, 3, date '2016-02-24') parsing failed
Key: CALCITE-6410
URL: https://issues.apache.org/jira/browse/CALCITE-6410
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.37.0
Reporter: Caican Cai
Fix For: 1.38.0
The dateadd and datediff tests are missing in SqlOperatorTest. I tried to add
them to improve it. The test code is as follows
{code:java}
@Test void testDateAdd2() {
final SqlOperatorFixture f = Fixtures.forOperators(true)
.setFor(SqlLibraryOperators.DATEADD);
MONTH_VARIANTS.forEach(s ->
f.checkScalar("dateadd(" + s
+ ", 3, date '2016-02-24')",
"2016-05-24 12:42:25", "TIMESTAMP(0) NOT NULL"));
} {code}
fail message：
java.lang.RuntimeException: Error while parsing query: values (dateadd(MONTH,
3, date '2016-02-24'))
at
org.apache.calcite.sql.test.AbstractSqlTester.parseAndValidate(AbstractSqlTester.java:159)
at
org.apache.calcite.sql.test.AbstractSqlTester.validateAndThen(AbstractSqlTester.java:250)
at
org.apache.calcite.test.SqlOperatorFixtureImpl.lambda$forEachQueryValidateAndThen$1(SqlOperatorFixtureImpl.java:154)
at
org.apache.calcite.sql.test.AbstractSqlTester.forEachQuery(AbstractSqlTester.java:450)
at
org.apache.calcite.test.SqlOperatorFixtureImpl.forEachQueryValidateAndThen(SqlOperatorFixtureImpl.java:153)
at
org.apache.calcite.test.SqlOperatorFixtureImpl.checkType(SqlOperatorFixtureImpl.java:130)
at
org.apache.calcite.sql.test.SqlOperatorFixture.checkScalar(SqlOperatorFixture.java:220)
at
org.apache.calcite.test.SqlOperatorTest.lambda$testDateAdd2$150(SqlOperatorTest.java:12789)
at java.util.Arrays$ArrayList.forEach(Arrays.java:3880)
at
org.apache.calcite.test.SqlOperatorTest.testDateAdd2(SqlOperatorTest.java:12788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.sup
[jira] [Created] (CALCITE-6409) Character types and Boolean types are comparable
Caican Cai created CALCITE-6409: --- Summary: Character types and Boolean types are comparable Key: CALCITE-6409 URL: https://issues.apache.org/jira/browse/CALCITE-6409 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.37.0 Reporter: Caican Cai Fix For: 1.38.0 In the SameOperandTypeChecker method, the char type and the boolean type are comparable because of the call to the isComparable method. Comparability of char types and boolean types returns true in the isComparable method. However, char types and Boolean types in Spark are incomparable. Does Calcite have fixed standards here? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6408) Not-null ThreadLocal
Julian Hyde created CALCITE-6408:
Summary: Not-null ThreadLocal
Key: CALCITE-6408
URL: https://issues.apache.org/jira/browse/CALCITE-6408
Project: Calcite
Issue Type: Improvement
Reporter: Julian Hyde
Many points in the code need to wrap {{ThreadLocal.get()}} in
{{requireNonNull}} or {{castNonNull}} to keep checkerFramework happy. This
change removes those, by providing a sub-class of ThreadLocal whose values are
known to be never null.
In CALCITE-915 we added {{class TryThreadLocal}} to make it easier to remember
to unset a thread-local value, and in commit
[f82028f4|https://github.com/apache/calcite/commit/f82028f4b983707bd9af3d9ba73820ef2431e971]
we made its {{get}} method return a not-null value. But the only constructor
was {{{}TryThreadLocal.of(T initialValue){}}}. In this change, we add
{{{}TryThreadLocal.withSupplier(Supplier){}}}, so that each thread can have
its own value, and convert all uses of {{ThreadLocal}} where to use it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6407) DECIMAL types with scale > precision should be tested
Mihai Budiu created CALCITE-6407: Summary: DECIMAL types with scale > precision should be tested Key: CALCITE-6407 URL: https://issues.apache.org/jira/browse/CALCITE-6407 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.37.0 Reporter: Mihai Budiu Calcite allows DECIMAL types where the scale > precision. However, whether these are legal should be probably defined by the type system. This case should be also more thoroughly tested. See also https://issues.apache.org/jira/browse/CALCITE-5651 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6406) Negative scales for DECIMAL types are not tested
Mihai Budiu created CALCITE-6406: Summary: Negative scales for DECIMAL types are not tested Key: CALCITE-6406 URL: https://issues.apache.org/jira/browse/CALCITE-6406 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.37.0 Reporter: Mihai Budiu Calcite allows DECIMAL types to have negative scales. This is in line with other SQL dialects. I suspect that the type system should decide whether negative scales are allowed or not. Moreover, if they are allowed, they should be tested properly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6405) TO_TIMESTAMP doesn't work
Oleg Alekseev created CALCITE-6405:
--
Summary: TO_TIMESTAMP doesn't work
Key: CALCITE-6405
URL: https://issues.apache.org/jira/browse/CALCITE-6405
Project: Calcite
Issue Type: Bug
Reporter: Oleg Alekseev
TO_TIMESTAMP doesn't work... it leads to java.lang.RuntimeException: cannot
translate call TO_TIMESTAMP($t1, $t2)
PARSE_TIMESTAMP works good
```
import org.apache.calcite.adapter.jdbc.JdbcSchema;
import org.apache.calcite.avatica.util.Casing;
import org.apache.calcite.config.CalciteConnectionProperty;
import org.apache.calcite.jdbc.CalciteConnection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Properties;
import org.apache.calcite.jdbc.Driver;
import org.apache.calcite.schema.SchemaPlus;
import javax.sql.DataSource;
public class Test {
public static void main(String[] args) throws SQLException {
CalciteConnection connection = initCalciteConnection();
DataSource oracleDataSource1 = JdbcSchema.dataSource(
"jdbc:oracle:thin:@//:/",
"oracle.jdbc.OracleDriver",
"",
""
);
String schemaName = "oracle_1";
SchemaPlus rootSchema = connection.getRootSchema();
JdbcSchema jdbcSchema = JdbcSchema.create(rootSchema, schemaName,
oracleDataSource1, null, "");
rootSchema.add(schemaName, jdbcSchema);
// String sql = """
// SELECT PARSE_TIMESTAMP('%d.%m.%Y %H:%M:%S', '01.01.2024 00:00:00')
// """;
String sql = """
SELECT TO_TIMESTAMP('2024-01-01 00:00:00', '-MM-DD HH24:MI:SS')
""";
ResultSet resultSet = connection.createStatement().executeQuery(sql);
while (resultSet.next()) {
for (int i=1; i <= resultSet.getMetaData().getColumnCount(); i++) {
System.out.println(resultSet.getString(i));
System.out.println(" ");
}
System.out.println(" ");
}
}
private static CalciteConnection initCalciteConnection() throws
SQLException {
DriverManager.registerDriver(new Driver());
Properties properties = new Properties();
// properties.putIfAbsent(CalciteConnectionProperty.FUN.camelName(),
"oracle,postgresql,bigquery");
properties.putIfAbsent(CalciteConnectionProperty.FUN.camelName(),
"all");
properties.putIfAbsent(CalciteConnectionProperty.CASE_SENSITIVE.camelName(),
"false");
properties.putIfAbsent(CalciteConnectionProperty.QUOTED_CASING.camelName(),
Casing.UNCHANGED.name());
properties.putIfAbsent(CalciteConnectionProperty.UNQUOTED_CASING.camelName(),
Casing.UNCHANGED.name());
return DriverManager.getConnection("jdbc:calcite:",
properties).unwrap(CalciteConnection.class);
}
}
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6404) update support subquery
colagy wang created CALCITE-6404:
Summary: update support subquery
Key: CALCITE-6404
URL: https://issues.apache.org/jira/browse/CALCITE-6404
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.37.0
Reporter: colagy wang
calcite have not support this sql yet, can support this?
{code:java}
UPDATE "cubeappdata"."UD_1_QNBB6_ptfwfyybmfyb"
SET sys_sort = subquery.rownum * 10 FROM (SELECT id, ROW_NUMBER() OVER
(ORDER BY create_time) AS rownum FROM "cubeappdata"."UD_1_QNBB6_ptfwfyybmfyb")
AS subquery
WHERE "cubeappdata"."UD_1_QNBB6_ptfwfyybmfyb".id = subquery.id;
{code}
I had using subquery in update sql, but it did not working.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6403) Rule AGGREGATE_EXPAND_DISTINCT_AGGREGATES is unsound
Mihai Budiu created CALCITE-6403:
Summary: Rule AGGREGATE_EXPAND_DISTINCT_AGGREGATES is unsound
Key: CALCITE-6403
URL: https://issues.apache.org/jira/browse/CALCITE-6403
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.37.0
Reporter: Mihai Budiu
Here is a simple SQL program that operates on a collection with no input rows:
{code:sql}
SELECT COUNT(*), COUNT(DISTINCT COL1) FROM T WHERE false"
{code}
The rewrite rule expands this into:
{code}
LogicalProject(EXPR$0=[CAST($0):BIGINT NOT NULL], EXPR$1=[$1]), id = 39
LogicalAggregate(group=[{}], EXPR$0=[MIN($1) FILTER $3],
EXPR$1=[COUNT($0) FILTER $2]), id = 37
LogicalProject(COL1=[$0], EXPR$0=[$1], $g_0=[=($2, 0)], $g_1=[=($2,
1)]), id = 35
LogicalAggregate(group=[{0}], groups=[[{0}, {}]], EXPR$0=[COUNT()],
$g=[GROUPING($0)]), id = 30
LogicalValues(tuples=[[]]), id = 22
{code}
Notice that there is an inner group-by aggregate that produces an empty set,
and an outer aggregation that uses MIN. MIN for an empty collection is NULL,
whereas the original query should have produced 0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6402) Aggregates implied in grouping sets have a wrong nullability at validation stage
Claude Brisson created CALCITE-6402:
---
Summary: Aggregates implied in grouping sets have a wrong
nullability at validation stage
Key: CALCITE-6402
URL: https://issues.apache.org/jira/browse/CALCITE-6402
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Claude Brisson
As long as the empty set is present among the grouping sets, which is always
the case for CUBE and ROLLAP, the (unfiltered) result will contain a row with
the global aggregate. And on such a row, most standard aggregate functions are
nullable (even on a non-null column, for the empty rowset).
But the SUM function, for instance, has the following return type inference:
{code}
public static final SqlReturnTypeInference AGG_SUM = opBinding -> {
final RelDataTypeFactory typeFactory = opBinding.getTypeFactory();
final RelDataType type = typeFactory.getTypeSystem()
.deriveSumType(typeFactory, opBinding.getOperandType(0));
if (opBinding.getGroupCount() == 0 || opBinding.hasFilter()) {
return typeFactory.createTypeWithNullability(type, true);
} else {
return type;
}
};
{code}
If the operand is not nullable, since the group count will be non-zero for a
rollup, a cube or a grouping sets containing the empty set.
It seems to me that the group count itself is not a sufficient information to
determine the nullability, we may be lacking a boolean stating whether the
empty group is implied, or the complete list of groups instead of the groups
count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6401) JDBC adapter cannot push down joins with complex JOIN condition
Ulrich Kramer created CALCITE-6401:
--
Summary: JDBC adapter cannot push down joins with complex JOIN
condition
Key: CALCITE-6401
URL: https://issues.apache.org/jira/browse/CALCITE-6401
Project: Calcite
Issue Type: Improvement
Affects Versions: 1.36.0
Reporter: Ulrich Kramer
JDBC adapter (in JdbcJoinRule) cannot push down joins with conditions which
include operations not listed in {{JdbcJoinRule::canJoinOnCondition}}.
See also CALCITE-4907
For example the following statement is executed using an
{{EnumerableNestedLoopJoin}}
{code:SQL}
SELECT
*
FROM
A
JOIN (
SELECT
D."userId",
MAX(D."id") as "id"
FROM
D
GROUP BY
D."userId"
) B ON (
A."id" = B."id" AND A."userId" IS NOT NULL
)
OR (
A."userId" = B."userId" AND A."id" IS NOT NULL
)
{code}
Adding the cases {{IS_NULL}} and {{IS_NOT_NULL}} to
{{JdbcJoinRule::canJoinOnCondition}} fixes the problem for this statement. But
I was not able to find out which cases are also missing here.
Where could I find the associated code in {{JdbcJoin::implement}} that makes it
impossible to create an appropriate SQL statement if all operations were
allowed?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6400) MAP_ENTRIES is not allowed to be empty
Caican Cai created CALCITE-6400:
---
Summary: MAP_ENTRIES is not allowed to be empty
Key: CALCITE-6400
URL: https://issues.apache.org/jira/browse/CALCITE-6400
Project: Calcite
Issue Type: Bug
Components: avatica
Affects Versions: 1.36.0
Reporter: Caican Cai
Fix For: 1.37.0
{code:java}
scala> val df = spark.sql("select map_entries(map('foo', 1, null, 2.0))")
df: org.apache.spark.sql.DataFrame = [map_entries(map(foo, 1, NULL, 2.0)):
array>]scala> df.show()
org.apache.spark.SparkRuntimeException: [NULL_MAP_KEY] Cannot use null as map
key.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.nullAsMapKeyNotAllowedError(QueryExecutionErrors.scala:445)
at
org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder.put(ArrayBasedMapBuilder.scala:56)
at
org.apache.spark.sql.catalyst.expressions.CreateMap.eval(complexTypeCreator.scala:248)
at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:542)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:80)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.$anonfun$constantFolding$4(expressions.scala:90)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1249)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1248)
at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:532)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:90)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.$anonfun$applyOrElse$1(expressions.scala:94)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:207)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:207)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:218)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$3(QueryPlan.scala:223)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.immutable.List.map(List.scala:305)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:223)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:228)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:355)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:228)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:94)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$1.applyOrElse(expressions.scala:93)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformWithPruning(TreeNode.scala:478)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.apply(expressions.scala:93)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.apply(expressions.scala:46)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor
[jira] [Created] (CALCITE-6399) The predicate IN is transformed by the sqltorelConverter into join
kate created CALCITE-6399:
-
Summary: The predicate IN is transformed by the sqltorelConverter
into join
Key: CALCITE-6399
URL: https://issues.apache.org/jira/browse/CALCITE-6399
Project: Calcite
Issue Type: Wish
Components: core
Affects Versions: 1.36.0
Reporter: kate
*Simple Example：*
My SqlToRelConverter：
{code:java}
SqlToRelConverter.Config config = new SqlToRelNodeConfig()
.withTrimUnusedFields(true)
.withRelBuilderFactory(RelFactories.LOGICAL_BUILDER)
.withRelBuilderConfigTransform(relBuilderConfig -> relBuilderConfig
.withSimplify(false))
.withHintStrategyTable(HintStrategyTable.EMPTY);
RelRoot root = sqlToRelConverter.convertQuery(sqlNode, false, false);
{code}
SQL
{code:java}
SELECT cd_gender,
cd_marital_status,
cd_education_status, cd_purchase_estimate,
cd_credit_rating, cd_dep_count,
cd_dep_employed_count, cd_dep_college_countFROM
pg.tpcds.customer c,
pg.tpcds. customer_address ca,
pg.tpcds.customer_demographics
WHERE c.c_current_addr_sk = ca.ca_address_sk
AND cd_demo_sk = c.c_current_cdemo_sk
AND ca_county IN ( 'Lycoming County', 'Sheridan County', 'Kandiyohi
County',Pike County','Greene County' )
{code}
RelNode tree after converting
{code:java}
LogicalProject(cd_gender=[$32], cd_marital_status=[$33],
cd_education_status=[$34], cd_purchase_estimate=[$35], cd_credit_rating=[$36],
cd_dep_count=[$37], cd_dep_employed_count=[$38], cd_dep_college_count=[$39])
LogicalFilter(condition=[AND(=($4, $18), true, =($31, $2))])
LogicalJoin(condition=[=($25, $40)], joinType=[inner])
LogicalJoin(condition=[true], joinType=[inner])
LogicalJoin(condition=[true], joinType=[inner])
LogicalTableScan(table=[[tpcds, customer]])
LogicalTableScan(table=[[tpcds, customer_address]])
LogicalTableScan(table=[[tpcds, customer_demographics]])
LogicalAggregate(group=[{0}])
LogicalUnion(all=[true])
LogicalValues(tuples=[[{ 'Lycoming County' }]])
LogicalValues(tuples=[[{ 'Sheridan County' }]])
LogicalValues(tuples=[[{ 'Kandiyohi County' }]])
LogicalValues(tuples=[[{ 'Pike County' }]])
LogicalValues(tuples=[[{ 'Greene County' }]]) {code}
I'm trying to figure out how to keep in as a filter instead of converting it to
a join value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6398) Spark's ifnull function has only one parameter and the return value is Boolean
Caican Cai created CALCITE-6398:
---
Summary: Spark's ifnull function has only one parameter and the
return value is Boolean
Key: CALCITE-6398
URL: https://issues.apache.org/jira/browse/CALCITE-6398
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Caican Cai
Fix For: 1.37.0
Spark's ifnull function has only one parameter and the return value is Boolean
[https://spark.apache.org/docs/2.3.0/api/sql/index.html#isnull]
But calcite's ifnull has two parameters and returns expr
{code:java}
| b s | IFNULL(value1, value2) | Equivalent to
`NVL(value1, value2)`{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6397) Add NVL2 function (enabled in Spark library)
Caican Cai created CALCITE-6397: --- Summary: Add NVL2 function (enabled in Spark library) Key: CALCITE-6397 URL: https://issues.apache.org/jira/browse/CALCITE-6397 Project: Calcite Issue Type: New Feature Reporter: Caican Cai Add NVL2 function (enabled in Spark library) https://spark.apache.org/docs/2.3.0/api/sql/index.html#nvl2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6396) Add ADD_MONTHS function (enabled in Spark library)
Caican Cai created CALCITE-6396: --- Summary: Add ADD_MONTHS function (enabled in Spark library) Key: CALCITE-6396 URL: https://issues.apache.org/jira/browse/CALCITE-6396 Project: Calcite Issue Type: New Feature Components: core Affects Versions: 1.36.0 Reporter: Caican Cai Fix For: 1.37.0 Add ADD_MONTHS function (enabled in Spark library) https://spark.apache.org/docs/2.3.0/api/sql/index.html#add_months -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6395) Significant precision loss when representing REAL literals
Mihai Budiu created CALCITE-6395:
Summary: Significant precision loss when representing REAL literals
Key: CALCITE-6395
URL: https://issues.apache.org/jira/browse/CALCITE-6395
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
Consider this test that could be a SqlOperatorTest:
{code:java}
f.checkScalar("CAST(CAST('36854775807.0' AS REAL) AS BIGINT)",
"36854775808", "BIGINT NOT NULL");
{code}
The produced result is actually very far:
Expected: is "36854775808"
but: was "36854779904"
This big error comes from two reasons:
- Calcite uses BigDecimal values to represent floating point values, see
[CALCITE-2067]
- When converting a Float value to a BigDecimal in RexBuilder.clean(), the
following sequence is used:
{code:java}
new BigDecimal(((Number) o).doubleValue(), MathContext.DECIMAL32)
{code}
Using a DECIMAL32 math context leads to the precision loss. Just because a
Float uses 32 bits does not mean that the decimal should also use 32 bits.
The real fix is in the PR for [CALCITE-2067], but that hasn't been reviewed for
a long time, so I will submit a fix for the clean() method..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6394) Be able to use input names instead of $i in explain
Gonzalo Ortiz created CALCITE-6394:
--
Summary: Be able to use input names instead of $i in explain
Key: CALCITE-6394
URL: https://issues.apache.org/jira/browse/CALCITE-6394
Project: Calcite
Issue Type: Improvement
Reporter: Gonzalo Ortiz
As Apache Pinot committer, one of the request I've found from our users is to
make explains easier to read. The main complain here is that it requires some
training to be able to read them, specially because in explain names of the
inputs is index based ($0, $1, etc) but the rows itself are indirectly defined
by the each input. A person with enough experience may understand that the rows
generated by a join is formed by appending the input of the left hand side with
the input of the right hand side, but in complex queries it takes time to get
use to that and to picture these rows in your mind.
These complains are also based by the fact that other databases have solved
this issue long ago. For example, in Postgres, given a table `a` and `two_cols`
such as:
{{postgres=# \d a}}
{{Table "public.a"}}
{{Column | Type | Collation | Nullable | Default }}
{{+-+---+--+-}}
{{a | integer | | | }}
{{postgres=# \d two_cols}}
{{ Table "public.two_cols"}}
{{Column | Type | Collation | Nullable | Default }}
{{+-+---+--+-}}
{{a | integer | | | }}
{{b | integer | | |}}
A join explain shows:
{{postgres=# explain select * from a join two_cols on a.a = two_cols.a;}}
{{ QUERY PLAN }}
{{}}
{{Merge Join (cost=338.29..781.81 rows=28815 width=12)}}
{{ Merge Cond: ({*}two_cols.a = a.a{*})}}
{{ -> Sort (cost=158.51..164.16 rows=2260 width=8)}}
{{Sort Key: *two_cols.a*}}
{{-> Seq Scan on two_cols (cost=0.00..32.60 rows=2260 width=8)}}
{{ -> Sort (cost=179.78..186.16 rows=2550 width=4)}}
{{Sort Key: *a.a*}}
{{-> Seq Scan on a (cost=0.00..35.50 rows=2550 width=4)}}
{{(8 rows)}}
In Postgres the explain even conserves the name of the aliases if used.
To be clear, I don't know if it is possible to do this right now. I didn't find
a way myself and after asking in the dev mail least I was invited to write a
Jira issue to discuss about the specific requirements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6393) Byte code of SqlFunctions is invalid
Sergey Nuyanzin created CALCITE-6393:
Summary: Byte code of SqlFunctions is invalid
Key: CALCITE-6393
URL: https://issues.apache.org/jira/browse/CALCITE-6393
Project: Calcite
Issue Type: Bug
Reporter: Sergey Nuyanzin
The issue is a result of testing of Apache Calcite 1.37.0 rc 4 in this thread
[1]
There is test project andprocedure provided by [~MasseGuillaume]
it shows that since Calcite 1.36.0 it starts failing as
{noformat}
java.lang.ArrayIndexOutOfBoundsException: Index 65536 out of bounds for
length 297
at org.objectweb.asm.ClassReader.readLabel(ClassReader.java:2695)
at org.objectweb.asm.ClassReader.createLabel(ClassReader.java:2711)
at
org.objectweb.asm.ClassReader.readTypeAnnotations(ClassReader.java:2777)
at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1929)
at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:745)
{noformat}
Also since Calcite 1.27.0 it starts failing as
{noformat}
java.lang.IllegalArgumentException: Invalid end label (must be visited
first)
at
org.objectweb.asm.util.CheckMethodAdapter.checkLabel(CheckMethodAdapter.java:1453)
at
org.objectweb.asm.util.CheckMethodAdapter.visitLocalVariableAnnotation(CheckMethodAdapter.java:996)
at
org.objectweb.asm.MethodVisitor.visitLocalVariableAnnotation(MethodVisitor.java:757)
at
org.objectweb.asm.commons.MethodRemapper.visitLocalVariableAnnotation(MethodRemapper.java:257)
at org.objectweb.asm.ClassReader.readCode(ClassReader.java:2614)
at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515)
{noformat}
[1] https://lists.apache.org/thread/n6cs1l86mt6fc5q8pcxr97czs3p6w65f
[2] https://github.com/MasseGuillaume/asm-remapper-bug
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6392) Support all PostgreSQL 14 date/time patterns for to_date/to_timestamp
Norman Jordan created CALCITE-6392: -- Summary: Support all PostgreSQL 14 date/time patterns for to_date/to_timestamp Key: CALCITE-6392 URL: https://issues.apache.org/jira/browse/CALCITE-6392 Project: Calcite Issue Type: Sub-task Reporter: Norman Jordan Assignee: Norman Jordan Many of the date/time format patterns supported by PostgreSQL 14 are not supported in Calcite. * HH * US * * S * AM * A.M. * am * a.m. * PM * P.M. * pm * p.m. * Y,YYY * YYY * Y * IYYY * IYY * IY * I * BC * B.C. * bc * b.c. * AD * A.D. * ad * a.d. * MONTH * month * MON * mon * DAY * day * Dy * dy * IDDD * ID * TZH * TZM * OF There are also template pattern modifiers that need to be supported. * FM (prefix) * TH (suffix) * th (suffix) * FX (prefix) * TM (prefix) Some format patterns in Calcite behave differently from PostgreSQL 14. * FF1 * FF2 * FF4 * FF5 * FF6 Also verify that the other existing format strings produce results that match PostgreSQL 14. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6391) Apply mapping to RelCompositeTrait does not apply it to wrapped traits
James Duong created CALCITE-6391: Summary: Apply mapping to RelCompositeTrait does not apply it to wrapped traits Key: CALCITE-6391 URL: https://issues.apache.org/jira/browse/CALCITE-6391 Project: Calcite Issue Type: Bug Reporter: James Duong RelCompositeTrait wraps multiple traits. However it does not override RelTrait#apply(Mapping.TargetMapping). It uses the default implementation which makes the mapping have no effect. The correct behavior would be to propagate the apply() call to wrapped traits. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6390) ArrowAdapterTest fails on Windows 11
Sergey Nuyanzin created CALCITE-6390:
Summary: ArrowAdapterTest fails on Windows 11
Key: CALCITE-6390
URL: https://issues.apache.org/jira/browse/CALCITE-6390
Project: Calcite
Issue Type: Bug
Reporter: Sergey Nuyanzin
That's seems somehow highlights the difference between Windows Server and non
Server
we have tests against Windows Server on gha (windows-latest) and they are green
At the same time local tests on Windows 11 show that {{ArrowAdapterTest}} fails
like
{noformat}
FAILURE 0.0sec, org.apache.calcite.adapter.arrow.ArrowAdapterTest >
executionError
java.io.IOException: Failed to delete temp directory
D:\MyConfiguration\cancai.cai\AppData\Local\Temp\junit5105379620525559011. The
following paths could not be deleted (see suppressed exceptions for details): ,
arrow
at
org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.createIOExceptionWithAttachedFailures(TempDirectory.java:350)
at
org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:251)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$closeAllStoredCloseableValues$3(ExtensionValuesStore.java:68)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:390)
at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore.closeAllStoredCloseableValues(ExtensionValuesStore.java:68)
at
org.junit.jupiter.engine.descriptor.AbstractExtensionContext.close(AbstractExtensionContext.java:80)
at
org.junit.jupiter.engine.execution.JupiterEngineExecutionContext.close(JupiterEngineExecutionContext.java:53)
at
org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:222)
at
org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:57)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$cleanUp$10(NodeTestTask.java:167)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.cleanUp(NodeTestTask.java:167)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:98)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
Suppressed: java.nio.file.DirectoryNotEmptyException:
D:\MyConfiguration\cancai.cai\AppData\Local\Temp\junit5105379620525559011
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6389) RexBuilder.removeCastFromLiteral does not preserve semantics for decimal literals
Mihai Budiu created CALCITE-6389:
Summary: RexBuilder.removeCastFromLiteral does not preserve
semantics for decimal literals
Key: CALCITE-6389
URL: https://issues.apache.org/jira/browse/CALCITE-6389
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
This is a bug which I keep fixing as part of several pull requests (not yet
merged), so I have decided to make it into a separate issue.
The code in removeCastFromLiteral is supposed to remove casts which are useless.
However, the code is too aggressive for decimal literals. In particular, this
fragment:
{code:java}
if (toType.getSqlTypeName() == SqlTypeName.DECIMAL) {
final BigDecimal decimalValue = (BigDecimal) value;
return SqlTypeUtil.isValidDecimalValue(decimalValue, toType);
}
{code}
There are at least two bugs here:
- this code removes casts from literals even if they represent types such as
interval
- this code does not check properly that the literal can be represented by the
type specified by the cast. In particular, the function
SqlTypeUtil.isValidDecimalValue does not correctly consider the scale of the
resulting type, and may return 'true' even when a number requires rescaling.
Removing casts in such a case changes the literal's value. I have submitted a
fix for this bug as part of [CALCITE-6322], but that PR hasn't been merged yet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6388) PsTableFunction throws NumberFormatException when the 'user' column has spaces
Alessandro Solimando created CALCITE-6388:
-
Summary: PsTableFunction throws NumberFormatException when the
'user' column has spaces
Key: CALCITE-6388
URL: https://issues.apache.org/jira/browse/CALCITE-6388
Project: Calcite
Issue Type: Bug
Components: os-adapter
Affects Versions: 1.36.0
Reporter: Alessandro Solimando
Assignee: Alessandro Solimando
Fix For: 1.37.0
Line parsing splits on spaces
([PsTableFunction.java#L77|https://github.com/apache/calcite/blob/aa8d81bf1ff39e3632aeb856fc4cc247ce9727e5/plus/src/main/java/org/apache/calcite/adapter/os/PsTableFunction.java#L77]),
which breaks whenever the "user" contains at least a space.
An example output on my laptop is as follows:
{noformat}
$ ps ax -o
ppid=,pid=,pgid=,tpgid=,stat=,user=,pcpu=,pmem=,vsz=,rss=,tty=,start=,time=,uid=,ruid=,sess=,comm=
| grep startup
1 6728 6728 0 S startup user 0.0 0.0 410266096 2528 ??
11Apr24 0:16.97 501 501 0 /usr/sbin/distnoted
1 6729 6729 0 SN startup user 0.0 0.1 410332256 17616 ??
11Apr24 0:42.41 501 501 0
/System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdbulkimport
1 6750 6750 0 S startup user 0.0 0.0 410376144 13344 ??
11Apr24 0:40.39 501 501 0 /usr/libexec/lsd
1 10148 10148 0 S startup user 0.0 0.0 410354816 5488 ??
8:26PM 0:00.31 501 501 0 /usr/libexec/containermanagerd
1 95313 95313 0 S startup user 0.0 0.0 410357344 6576 ??
Fri05PM 0:00.32 501 501 0 /usr/libexec/trustd{noformat}
When running the test it fails with the following stack trace:
{code:java}
Error while executing SQL "select distinct `user` from ps": while parsing value
[user] of field [pcpu] in line [ 1 6728 6728 0 S startup user
0.0 0.0 410266096 2528 ?? 11Apr24 0:16.95 501 501 0
/usr/sbin/distnoted]
java.sql.SQLException: Error while executing SQL "select distinct `user` from
ps": while parsing value [user] of field [pcpu] in line [ 1 6728 6728 0
S startup user 0.0 0.0 410266096 2528 ?? 11Apr24 0:16.95
501 501 0 /usr/sbin/distnoted]
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:164)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:228)
at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:566)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.lambda$returns$1(CalciteAssert.java:1495)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.withConnection(CalciteAssert.java:1434)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1493)
at
org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1483)
at
org.apache.calcite.adapter.os.OsAdapterTest.testPsDistinct(OsAdapterTest.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationI
[jira] [Created] (CALCITE-6387) Calcite build while compiliation with jdk11+
Sergey Nuyanzin created CALCITE-6387:
Summary: Calcite build while compiliation with jdk11+
Key: CALCITE-6387
URL: https://issues.apache.org/jira/browse/CALCITE-6387
Project: Calcite
Issue Type: Bug
Affects Versions: 1.36.0
Reporter: Sergey Nuyanzin
The issue appears with newly added Arrow adapter which requires
{noformat}
--add-opens=java.base/java.nio=ALL-UNNAMED
{noformat}
could be fixed with adding
{noformat}
plugins.withType {
tasks {
configureEach {
jvmArgs("-XX:+IgnoreUnrecognizedVMOptions")
jvmArgs("--add-opens=java.base/java.nio=ALL-UNNAMED")
}
}
}
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6386) NPE when using ES adapter with model.json and no specified username, password or pathPrefix
guluo created CALCITE-6386:
--
Summary: NPE when using ES adapter with model.json and no
specified username, password or pathPrefix
Key: CALCITE-6386
URL: https://issues.apache.org/jira/browse/CALCITE-6386
Project: Calcite
Issue Type: Bug
Components: elasticsearch-adapter
Affects Versions: 1.36.0
Reporter: guluo
Reproduction steps：
1 Creating model.json, according to the calcite doc about [Elasticsearch
adapter (apache.org)|https://calcite.apache.org/docs/elasticsearch_adapter.html]
{
"version": "1.0",
"defaultSchema": "elasticsearch",
"schemas": [
{
"type": "custom",
"name": "elasticsearch",
"factory":
"org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory",
"operand": {
"coordinates": "\{'127.0.0.1': 9200}"
}
}
]
}
2 Connecting es by sqlline
sqlline> !connect jdbc:calcite:model=/root/build/calcite//model.json
3 We would get NPE,as follow.
Caused by: java.lang.NullPointerException: at index 1
at
com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:232)
at
com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:222)
at
com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:216)
at com.google.common.collect.ImmutableList.construct(ImmutableList.java:354)
at com.google.common.collect.ImmutableList.of(ImmutableList.java:128)
at
org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory.connect(ElasticsearchSchemaFactory.java:202)
at
org.apache.calcite.adapter.elasticsearch.ElasticsearchSchemaFactory.create(ElasticsearchSchemaFactory.java:176)
at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:275)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6385) LintTest fails when run in source distribution
Stamatis Zampetakis created CALCITE-6385:
Summary: LintTest fails when run in source distribution
Key: CALCITE-6385
URL: https://issues.apache.org/jira/browse/CALCITE-6385
Project: Calcite
Issue Type: Bug
Affects Versions: 1.36.0
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
Steps to reproduce:
# Download
[https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-1.37.0-rc0/apache-calcite-1.37.0-src.tar.gz]
# tar -xvf apache-calcite-1.37.0-src.tar.gz
# cd apache-calcite-1.37.0-src
# /opt/gradle/gradle-7.6.1/bin/gradle cleanTest :core:test --tests LintTest
{noformat}
FAILURE 0.1sec, org.apache.calcite.test.LintTest >
testContributorsFileIsSorted()
java.lang.RuntimeException: command [git, ls-files, *.bat, *.cmd, *.csv,
*.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap, *.md,
*.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: failed with exception
at org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:185)
at org.apache.calcite.util.TestUnsafe.getTextFiles(TestUnsafe.java:158)
at
org.apache.calcite.test.LintTest.testContributorsFileIsSorted(LintTest.java:400)
Caused by: java.lang.RuntimeException: command [git, ls-files, *.bat,
*.cmd, *.csv, *.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap,
*.md, *.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: exited with
status 128
at
org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:180)
... 2 more
FAILURE 0.0sec, org.apache.calcite.test.LintTest > testMailmapFile()
java.lang.RuntimeException: command [git, ls-files, *.bat, *.cmd, *.csv,
*.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap, *.md,
*.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: failed with exception
at org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:185)
at org.apache.calcite.util.TestUnsafe.getTextFiles(TestUnsafe.java:158)
at org.apache.calcite.test.LintTest.testMailmapFile(LintTest.java:419)
Caused by: java.lang.RuntimeException: command [git, ls-files, *.bat,
*.cmd, *.csv, *.fmpp, *.ftl, *.iq, *.java, *.json, *.jj, *.kt, *.kts, .mailmap,
*.md, *.properties, *.sh, *.sql, *.txt, *.xml, *.yaml, *.yml]: exited with
status 128
at
org.apache.calcite.util.TestUnsafe.getGitFiles(TestUnsafe.java:180)
... 2 more
FAILURE 0.1sec,6 completed, 2 failed, 2 skipped,
org.apache.calcite.test.LintTest
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6384) Missing ASF header from buildcache.yml, gradle-wrapper-validation.yml
Stamatis Zampetakis created CALCITE-6384: Summary: Missing ASF header from buildcache.yml, gradle-wrapper-validation.yml Key: CALCITE-6384 URL: https://issues.apache.org/jira/browse/CALCITE-6384 Project: Calcite Issue Type: Task Affects Versions: 1.36.0 Reporter: Stamatis Zampetakis Assignee: Stamatis Zampetakis The header is required as per [ASF policy|https://www.apache.org/legal/src-headers.html]. * [https://github.com/apache/calcite/blob/153494207c24318d4c1d2c908df4a15c4aa4/.github/workflows/buildcache.yml] * [https://github.com/apache/calcite/blob/153494207c24318d4c1d2c908df4a15c4aa4/.github/workflows/gradle-wrapper-validation.yml] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6383) The class SameOperandTypeChecker is incorrectly named and documented
Mihai Budiu created CALCITE-6383:
Summary: The class SameOperandTypeChecker is incorrectly named and
documented
Key: CALCITE-6383
URL: https://issues.apache.org/jira/browse/CALCITE-6383
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The SameOperandTypeChecker claims that it checks whether operands have the same
type (the class name suggests this, as does the JavaDoc).
{code:java}
/**
* Parameter type-checking strategy where all operand types must be the same.
*/
public class SameOperandTypeChecker implements SqlSingleOperandTypeChecker {
{code}
But the code does something this:
{code:java}
for (int i : operandList) {
if (prev >= 0) {
if (!SqlTypeUtil.isComparable(types[i], types[prev])) {
{code}
The documentation for isComparable says:
{code:java}
/**
* Returns whether two types are comparable. They need to be scalar types of
* the same family, or struct types whose fields are pairwise comparable.
{code}
Thus the class only checks that the operands have the same type *family*, not
the same *type*.
I am not sure what the right fix is, though, since changing the class name
would be a pretty big breaking change. But I suspect this confusion is a source
of a few bugs. An instance is [CALCITE-6382]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6382) Type inference for SqlLeadLagAggFunction is incorrect
Mihai Budiu created CALCITE-6382: Summary: Type inference for SqlLeadLagAggFunction is incorrect Key: CALCITE-6382 URL: https://issues.apache.org/jira/browse/CALCITE-6382 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.36.0 Reporter: Mihai Budiu Currently the LeadLag operator does not use the default value when inferring the type of a column. For the following example: ``` SELECT lead(c * 2, 1, -1.4) OVER (PARTITION BY x ORDER BY c) FROM t ``` The column 'c' has type INTEGER, and Calcite infers a type of INTEGER for the result. However, the default value for the lead is -1.4, which is DECIMAL, so the result type should be DECIMAL. Currently Calcite only uses the nullability of the default value, but not its type in the result type inference. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6381) DateTruncFunctionCall customization support in parser
Robert Zych created CALCITE-6381: Summary: DateTruncFunctionCall customization support in parser Key: CALCITE-6381 URL: https://issues.apache.org/jira/browse/CALCITE-6381 Project: Calcite Issue Type: Improvement Reporter: Robert Zych Apache Pinot recently upgraded to version 1.36.0 of Apache Calcite. The parser (Parser.jj) was copied and [[customized|https://github.com/apache/pinot/blob/master/pinot-common/src/main/codegen/templates/Parser.jj#L7007]|https://github.com/apache/pinot/blob/master/pinot-common/src/main/codegen/templates/Parser.jj#L7007] to support Pinot's DATE_TRUNC in the upgrade process. Since builtinFunctionCallMethods is injected after DateTruncFunctionCall() it doesn't appear to support the customizations required. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6380) Casts from INTERVAL and STRING to DECIMAL are incorrect
Mihai Budiu created CALCITE-6380: Summary: Casts from INTERVAL and STRING to DECIMAL are incorrect Key: CALCITE-6380 URL: https://issues.apache.org/jira/browse/CALCITE-6380 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.36.0 Reporter: Mihai Budiu This is a follow-up from [CALCITE-6322], which is about casts from numeric types to DECIMAL values. There are two tests in SqlOperatorTest which are disabled due to this bug: - testCastStringToDecimal - testCastIntervalToNumeric -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6379) Arithmetic in Calcite does not produce results with the expected type
Mihai Budiu created CALCITE-6379: Summary: Arithmetic in Calcite does not produce results with the expected type Key: CALCITE-6379 URL: https://issues.apache.org/jira/browse/CALCITE-6379 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.36.0 Reporter: Mihai Budiu This seems to be a very old bug in Calcite, which appears in SqlOperatorTest under the name FNL25. It has been around for at least 10 years, but it affects other parts of the code as well. I could not find a more precise description of this bug in the JIRA, so I am filing a new one. I think this is at least a critical bug. The problem affects the expression evaluator in Calcite. After evaluating an arithmetic operation the result should be cast to the expected result type. For example, adding two SMALLINT values should cast the result to SMALLINT. Currently the code generated by the RexToLixTranslator is missing this cast. Since Java evaluation rules are different from SQL, many results are incorrect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6378) Since docker 26.x pushing and pulling with image manifest v2 schema 1 is disabled by default
Sergey Nuyanzin created CALCITE-6378:
Summary: Since docker 26.x pushing and pulling with image manifest
v2 schema 1 is disabled by default
Key: CALCITE-6378
URL: https://issues.apache.org/jira/browse/CALCITE-6378
Project: Calcite
Issue Type: Bug
Reporter: Sergey Nuyanzin
the source [1] also tells that it is going to be removed.
In Calcite it comes together with redis adapter which uses pretty old redis
docker image (2.8.19)
{noformat}
./gradlew clean build
{noformat}
is enough to reproduce (assuming there is docker 26+)
The fix is pretty straightforward: bumping docker image to e.g. 7.2.4 helps
[1]
https://docs.docker.com/engine/deprecated/#pushing-and-pulling-with-image-manifest-v2-schema-1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6377) Time expression causes IllegalStateException
Mihai Budiu created CALCITE-6377:
Summary: Time expression causes IllegalStateException
Key: CALCITE-6377
URL: https://issues.apache.org/jira/browse/CALCITE-6377
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The following SqlOperatorTest causes an exception:
{code:java}
final SqlOperatorFixture f = fixture();
f.checkScalar("time '12:03:01' + interval '25' day",
"12:03:01", "TIME(0) NOT NULL");
{code}
The exception is:
{code}
Caused by: java.lang.IllegalStateException: Unable to implement
EnumerableCalc(expr#0=[{inputs}], expr#1=[12:03:01],
expr#2=[216000:INTERVAL DAY], expr#3=[+($t1, $t2)], EXPR$0=[$t3]): rowcount
= 1.0, cumulative cost = {2.0 rows, 6.0 cpu, 0.0 io}, id = 20
EnumerableValues(tuples=[[{ 0 }]]): rowcount = 1.0, cumulative cost = {1.0
rows, 1.0 cpu, 0.0 io}, id = 13
at
org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:117)
at
org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:112)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1171)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:326)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:666)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:519)
...
Suppressed: java.lang.ArithmeticException: Value 216000 out of range
at
org.apache.calcite.linq4j.tree.Primitive.checkRoundedRange(Primitive.java:383)
at
org.apache.calcite.linq4j.tree.Primitive.numberValue(Primitive.java:412)
at
org.apache.calcite.linq4j.tree.Expressions.constant(Expressions.java:575)
at
org.apache.calcite.linq4j.tree.OptimizeShuttle.visit(OptimizeShuttle.java:305)
at
org.apache.calcite.linq4j.tree.UnaryExpression.accept(UnaryExpression.java:39)
at
org.apache.calcite.linq4j.tree.BinaryExpression.accept(BinaryExpression.java:47)
{code}
This seems to happen because the implementation insists in evaluating the
expression by converting the 25 days interval to milliseconds, which overflows.
However, adding a days interval to a time should be a noop. Replacing 'days'
with 'months', for example, works fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6376) Filtering CTE of at least 6 columns with QUALIFY operation results in exception
Austin Richardson created CALCITE-6376:
--
Summary: Filtering CTE of at least 6 columns with QUALIFY
operation results in exception
Key: CALCITE-6376
URL: https://issues.apache.org/jira/browse/CALCITE-6376
Project: Calcite
Issue Type: Bug
Affects Versions: 1.36.0
Reporter: Austin Richardson
Example query:
{code:java}
WITH MyCTE AS (
SELECT
column1,
column2,
column3,
column4,
column5,
column6
FROM (
VALUES
('value1', 10, 5.0, 'data1', 'info1', 'test1'),
('value2', 20, 4.0, 'data2', 'info2', 'test2'),
('value3', 30, 3.0, 'data3', 'info3', 'test3'),
('value4', 40, 2.0, 'data4', 'info4', 'test4'),
('value5', 50, 1.0, 'data5', 'info5', 'test5')
) AS t(column1, column2, column3, column4, column5, column6)
)
SELECT *
FROM MyCTE
QUALIFY RANK() OVER (ORDER BY column3) = 1{code}
Either removing one of the columns or the QUALIFY filter results in a
successful query.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6375) Fix failing quidem tests for FORMAT in CAST
Jerin John created CALCITE-6375: --- Summary: Fix failing quidem tests for FORMAT in CAST Key: CALCITE-6375 URL: https://issues.apache.org/jira/browse/CALCITE-6375 Project: Calcite Issue Type: Bug Reporter: Jerin John CALCITE-2980 implemented the FORMAT clause used inside the CAST operator. We had imported a large number of quidem tests from Apache Impala for the same, stored in the file: [cast-with-format.iq|https://github.com/apache/calcite/blob/main/core/src/test/resources/sql/cast-with-format.iq]. Many of these tests are currently disabled as they're either unsupported or incorrectly implemented, some tests outputs may also need to be changed as per dialect behavior. Creating this placeholder issue to track and fix the pending tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6374) lattice throw npe when agg call covered with cast
allendang created CALCITE-6374: -- Summary: lattice throw npe when agg call covered with cast Key: CALCITE-6374 URL: https://issues.apache.org/jira/browse/CALCITE-6374 Project: Calcite Issue Type: Bug Components: core Reporter: allendang Attachments: image-2024-04-18-21-44-04-262.png !image-2024-04-18-21-44-04-262.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6373) Distinct optimization produces broken sql query
Corvin Kuebler created CALCITE-6373:
---
Summary: Distinct optimization produces broken sql query
Key: CALCITE-6373
URL: https://issues.apache.org/jira/browse/CALCITE-6373
Project: Calcite
Issue Type: Bug
Components: core, jdbc-adapter
Affects Versions: 1.36.0
Reporter: Corvin Kuebler
Can be reproduced with the following test in JDBCTests:
{code:java}
String statement = "SELECT\n" +
" DISTINCT \"case-column\"\n" +
"FROM (\n" +
" SELECT \n" +
" CASE\n" +
" WHEN CAST(? AS VARCHAR) = \"ENAME\" THEN CAST(?
AS INTEGER)\n" +
" ELSE CAST(? AS INTEGER)\n" +
" END AS \"case-column\"\n" +
" FROM \"EMP\")";
CalciteAssert.model(JdbcTest.SCOTT_MODEL)
.query(statement)
.consumesPreparedStatement(p -> {
p.setString(1, "name");
p.setInt(2, 2);
p.setInt(3, 1);
})
.planHasSql("");
{code}
Lets assume the following statement is passed through calcite:
Before:
{code:java}
SELECT
DISTINCT "case-column"
FROM
SELECT
CASE
WHEN CAST(? AS VARCHAR) = "store_name" THEN CAST(? AS INTEGER)
ELSE CAST(? AS INTEGER)
END AS "case-column"
FROM "foodmart"."store"
{code}
After:
{code:java}
SELECT
CASE
WHEN ? = "ENAME" THEN ?
ELSE ?
END AS "case-column"
FROM
"SCOTT"."EMP"
GROUP BY
CASE
WHEN ? = "ENAME" THEN ?
ELSE ?
END
{code}
The produced statement hast two issues:
1. The missing casts (see also
https://issues.apache.org/jira/browse/CALCITE-6346)
2. Instead of pushing the DISTINCT it is replaced with a GROUP BY. This is
usually fine *but* since the field is a case statement containing dynamic
parameters it is not.
During sql syntax evaluation the database will give an error (the field in the
select is not contained in group by). This is because the dynamic parameters
are not evaluated during sql syntax evaluation.
I think this could be fixed by adding an alias to the field in the select
clause and referencing it in the group by clause instead of duplicating the
case statement and the dynamic parameters.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6372) Support ASOF joins
Mihai Budiu created CALCITE-6372: Summary: Support ASOF joins Key: CALCITE-6372 URL: https://issues.apache.org/jira/browse/CALCITE-6372 Project: Calcite Issue Type: New Feature Components: core Affects Versions: 1.36.0 Reporter: Mihai Budiu Seems that this new kind of JOIN named AS OF is very useful for processing time-series data. Here is some example documentation from Snowflake: https://docs.snowflake.com/en/sql-reference/constructs/asof-join The semantics is similar to a traditional join, but the result always contains at most one record from the left side, with the last matching record on the right side (where "time" is any value that can be compared for inequality). This can be expressed in SQL, but it looks very cumbersome, using a JOIN, a GROUP BY, and then an aggregation to keep the last value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6371) Add ability to configure RexExecutable behavior on exceptions
Claude Brisson created CALCITE-6371:
---
Summary: Add ability to configure RexExecutable behavior on
exceptions
Key: CALCITE-6371
URL: https://issues.apache.org/jira/browse/CALCITE-6371
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.36.0
Reporter: Claude Brisson
The current behavior of Calcite's {{RexExecutable}} constant folding class is
to eat all exceptions encountered during reduction. The underlying reason, as
far as I understand it, is that even if Calcite can't reduce an expression,
maybe the evaluation engine will be able to do it. Fair enough.
But it would be useful to be able to chose to let some exceptions go through.
One good example is {{ArithmeticException}} errors. They occur on divisions by
zero, negative logarithms, etc. and we may want to avoid the downstream
evaluation engine to receive such invalid expressions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6370) AS operator problems with USING clause
Claude Brisson created CALCITE-6370:
---
Summary: AS operator problems with USING clause
Key: CALCITE-6370
URL: https://issues.apache.org/jira/browse/CALCITE-6370
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Claude Brisson
In some cases, with the {{USING}} clause, the used column generates ambiguity
exceptions with the {{AS}} operator.
OK : {{select r_regionkey from region r0 join region r1 using (r_regionkey)}}
OK : {{select r_regionkey * 2 as rk2 from region r0 join region r1 using
(r_regionkey)}}
KO : {{select r_regionkey as rk from region r0 join region r1 using
(r_regionkey)}}
The last query generates the following error:
{code:java}
org.apache.calcite.sql.validate.SqlValidatorException: Column 'r_regionkey' is
ambiguous
at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance
(DirectConstructorHandleAccessor.java:62)
at java.lang.reflect.Constructor.newInstanceWithCaller
(Constructor.java:502)
at java.lang.reflect.Constructor.newInstance (Constructor.java:486)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex
(Resources.java:507)
at org.apache.calcite.runtime.Resources$ExInst.ex (Resources.java:601)
at org.apache.calcite.sql.SqlUtil.newContextException (SqlUtil.java:948)
at org.apache.calcite.sql.SqlUtil.newContextException (SqlUtil.java:933)
at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError
(SqlValidatorImpl.java:5517)
at org.apache.calcite.sql.validate.DelegatingScope.fullyQualify
(DelegatingScope.java:296)
at org.apache.calcite.sql.validate.SqlValidatorImpl.findTableColumnPair
(SqlValidatorImpl.java:3989)
at org.apache.calcite.sql.validate.SqlValidatorImpl.isRolledUpColumn
(SqlValidatorImpl.java:4032)
at org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp
(SqlValidatorImpl.java:3945)
at org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp
(SqlValidatorImpl.java:3940)
at org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUp
(SqlValidatorImpl.java:3959)
at org.apache.calcite.sql.validate.SqlValidatorImpl.checkRollUpInSelectList
(SqlValidatorImpl.java:3861)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect
(SqlValidatorImpl.java:3849)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6369) Expanding "start" gives ArrayIndexOutOfBoundsException with redundant columns and USING
Claude Brisson created CALCITE-6369:
---
Summary: Expanding "start" gives ArrayIndexOutOfBoundsException
with redundant columns and USING
Key: CALCITE-6369
URL: https://issues.apache.org/jira/browse/CALCITE-6369
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Claude Brisson
The query
{code}
select r_regionkey, * from region r0 join region r1 using (r_regionkey)
{code}
produces
{code}
java.lang.ArrayIndexOutOfBoundsException: Index 14 out of bounds for length 14
at org.apache.calcite.runtime.PairLists$ArrayImmutablePairList.get
(PairLists.java:573)
at org.apache.calcite.runtime.PairLists$ArrayImmutablePairList.get
(PairLists.java:550)
at org.apache.calcite.sql.validate.SqlValidatorImpl$Permute.permute
(SqlValidatorImpl.java:7443)
at org.apache.calcite.sql.validate.SqlValidatorImpl.expandStar
(SqlValidatorImpl.java:697)
at org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem
(SqlValidatorImpl.java:453)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList
(SqlValidatorImpl.java:4658)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect
(SqlValidatorImpl.java:3840)
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl
(SelectNamespace.java:61)
at org.apache.calcite.sql.validate.AbstractNamespace.validate
(AbstractNamespace.java:88)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace
(SqlValidatorImpl.java:1154)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery
(SqlValidatorImpl.java:1125)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6368) Inconsistent handling of COALESCE during validation
Claude Brisson created CALCITE-6368:
---
Summary: Inconsistent handling of COALESCE during validation
Key: CALCITE-6368
URL: https://issues.apache.org/jira/browse/CALCITE-6368
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Claude Brisson
{{SqlCoalesceFunction}} has a {{rewriteCall()}} method which will replace the
call with an {{SqlCase}} call during the unconditional rewrites stage.
But expanding natural joins and {{USING}} conditions is done using {{COALESCE}}
calls in a later validation stage.
As a result, the following query:
{code}
WITH A AS (SELECT 1 AS X)
SELECT
COALESCE(A0.X, A1.X),
*
FROM A AS A0 NATURAL JOIN A AS A1
{code}
is transformed by the validation step into:
{code}
WITH `A` AS (SELECT 1 AS `X`)
SELECT
CASE WHEN `A0`.`X` IS NOT NULL THEN `A0`.`X` ELSE `A1`.`X` END,
COALESCE(`A0`.`X`, `A1`.`X` AS `X0`) AS `X`
FROM `A` AS `A0` NATURAL INNER JOIN `A` AS `A1`
{code}
Maybe, the expanded results of {{*}} with natural joins and of {{USING}} should
pass through unconditional rewrites at creation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6367) Add timezone support for FORMAT clause in CAST (enabled in BigQuery)
Jerin John created CALCITE-6367:
---
Summary: Add timezone support for FORMAT clause in CAST (enabled
in BigQuery)
Key: CALCITE-6367
URL: https://issues.apache.org/jira/browse/CALCITE-6367
Project: Calcite
Issue Type: Bug
Reporter: Jerin John
This issue is a followup on CALCITE-6269 that fixes some of Calcite's existing
format elements implementation to be aligned to BQ functionality. Two major
formats that might require a bit more rework is adding support for the TZH/TZM
elements along with time zone areas as described below:
* [Parsing timestamp
literals|https://cloud.google.com/bigquery/docs/reference/standard-sql/format-elements#format_tz_as_string]
with timezones as used by BQ does not seem to be supported yet (format element
TZR is unimplemented, BQ has TZH, TZM for hour and minute offsets)
(eg: {{cast('2020.06.03 00:00:53+00' as timestamp format '.MM.DD
HH:MI:SSTZH')}}
* BQ format [timezone as string
|https://cloud.google.com/bigquery/docs/reference/standard-sql/format-elements#format_tz_as_string]
can take an additional argument {{{}AT TIME ZONE 'Asia/Kolkata'{}}}, which
would require additional parser changes and time zone parameter to be plumbed
in to the cast operator call.
One important thing to consider, is that the {{SimpleDateFormat}} class which
currently stores the datetime object, may not fully support timezone features
as described and might warrant a broader refactoring of this code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6366) Code generated by EnumUtils#convert should throw an exception if the target type is overflowed
Ruben Q L created CALCITE-6366: -- Summary: Code generated by EnumUtils#convert should throw an exception if the target type is overflowed Key: CALCITE-6366 URL: https://issues.apache.org/jira/browse/CALCITE-6366 Project: Calcite Issue Type: Improvement Components: core Reporter: Ruben Q L Code generated by EnumUtils#convert should throw an exception if the target type is overflowed (consider using Expressions#convertChecked) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6365) Support for RETURNING clause of JSON_QUERY
Dawid Wysakowicz created CALCITE-6365:
-
Summary: Support for RETURNING clause of JSON_QUERY
Key: CALCITE-6365
URL: https://issues.apache.org/jira/browse/CALCITE-6365
Project: Calcite
Issue Type: New Feature
Reporter: Dawid Wysakowicz
SQL standard says {{JSON_QUERY}} should support {{RETURNING}} clause similar to
{{JSON_VALUE}}. Calcite supports the clause for JSON_VALUE already, but not for
the JSON_QUERY.
{code}
::=
JSON_QUERY
[ ]
[ WRAPPER ]
[ QUOTES [ ON SCALAR STRING ] ]
[ ON EMPTY ]
[ ON ERROR ]
::=
RETURNING
[ FORMAT ]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6364) HttpClient SPENGO support is deprecated
Istvan Toth created CALCITE-6364: Summary: HttpClient SPENGO support is deprecated Key: CALCITE-6364 URL: https://issues.apache.org/jira/browse/CALCITE-6364 Project: Calcite Issue Type: Bug Components: avatica Reporter: Istvan Toth The Avatica Java client depends on Apache HttpClient's Kerberos/SPNEGO implementation. According to HTTPCLIENT-1625 that implementation is not secure, and is deprecated in newer versions. Unfortunately, HTTPCLIENT-1625 is very scant on details, and since the reason given for deprecation is the lack of time to fix it, it is likely not a trivial fix. Unfortunately, Avatica depends heavily on httpclient, and replacing it would it would be a big job. While Avatica in theory has a configurable Http Client implementation, the only non-httpclient implementation is more of a POC, and does not support ANY authentication methods. I can see these options: 1. Find an another http client library, and use it in Avatica 2. Copy the SPENGO auth code from httpclient, and fix it in Avatica 3. Fix the SPENGO auth code in httpclient. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6363) Introduce a rule to derive more filters from inner join condition
ruanhui created CALCITE-6363:
Summary: Introduce a rule to derive more filters from inner join
condition
Key: CALCITE-6363
URL: https://issues.apache.org/jira/browse/CALCITE-6363
Project: Calcite
Issue Type: New Feature
Components: core
Reporter: ruanhui
Sometimes we can infer more predicates from inner Join , for example, in the
query
SELECT * FROM ta INNER JOIN tb ON ta.x = tb.y WHERE ta.x > 10
we can infer condition tb.y > 10 and we can push it down to the table tb.
In this way, it is possible to reduce the amount of data involved in the Join.
To achieve this, here is my idea.
The core data strucature is two Multimap:
predicateMap : a map for inputRef to corresponding predicate such as: $1 -> [$1
> 10, $1 < 20, $1 = $2]
equivalenceMap : a map for inputRef to corresponding equivalent values or
inputRefs such as: $1 -> [$2, 1]
The filter derivation is divided into 4 steps:
1. construct predicate map and equivalence map by traversing all conjunctions
in the condition
2. search map and rewrite predicates with equivalent inputRefs or literals
2.1 find all inputRefs that are equivalent to the current inputRef, and then
rewrite all predicates involving equivalent inputRefs using inputRef, for
example if we have inputRef $1 = equivInputRef $2, then we can rewrite \{$2 =
10} to \{$1 = 10}.
2.2 find all predicates involving current inputRef. If any predicate refers to
another inputRef, rewrite the predicate with the literal/constant equivalent to
that inputRef, such as: if we have inputRef \{$1 > $2} and \{$2 = 10} then we
can infer new condition \{$1 > 10}.
2.3 derive new predicates based on equivalence relation in equivalenceMultimap
3. compose all original predicates and derived predicates
4. simplify expression such as range merging, like \{$1 > 10 AND $1 > 20} =>
\{$1 > 20}, \{$1 > $2 AND $1 > $2} => \{$1 > $2}
Anyone interested in this, please feel free to comment on this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6362) JSON_OBJECT. Internal representation DATE values leaks into resulting JSON object
Maksim Zhuravkov created CALCITE-6362:
-
Summary: JSON_OBJECT. Internal representation DATE values leaks
into resulting JSON object
Key: CALCITE-6362
URL: https://issues.apache.org/jira/browse/CALCITE-6362
Project: Calcite
Issue Type: Bug
Reporter: Maksim Zhuravkov
When a CAST function is used inside a JSON_OBJECT, then resulting JSON contains
a number of days instead
{code}
SELECT JSON_OBJECT('a': CAST('2010-01-01' AS DATE)) as c1;
{"a":14610}
{code}
I expected to get a date in a string form as many databases do in such case:
Bigquery:
{code}
SELECT JSON_OBJECT('a', CAST('2010-01-10' AS DATE))
{"a":"2010-01-10"}
{code}
PostgreSQL
{code}
SELECT json_build_object('a', '2010-01-01'::DATE)
{"a":"2010-01-01"}
{code}
MySql
{code}
SELECT json_object('a', DATE '2010-01-01')
{"a": "2010-01-01"}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6361) Uncollect.deriveUncollectRowType crashes if the input data is not a collection
Mihai Budiu created CALCITE-6361: Summary: Uncollect.deriveUncollectRowType crashes if the input data is not a collection Key: CALCITE-6361 URL: https://issues.apache.org/jira/browse/CALCITE-6361 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.37.0 Reporter: Mihai Budiu This happens because the type checker calls getComponentType() without checking first that the field type has components. It should report an error in such a case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6360) Add .asf.yaml to calcite-avatica-go repository
Francis Chuang created CALCITE-6360: --- Summary: Add .asf.yaml to calcite-avatica-go repository Key: CALCITE-6360 URL: https://issues.apache.org/jira/browse/CALCITE-6360 Project: Calcite Issue Type: Task Components: avatica-go Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-go-5.4.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6359) Update GitHub Actions workflows to use docker compose v2
Francis Chuang created CALCITE-6359: --- Summary: Update GitHub Actions workflows to use docker compose v2 Key: CALCITE-6359 URL: https://issues.apache.org/jira/browse/CALCITE-6359 Project: Calcite Issue Type: Task Components: avatica, avatica-go, core Reporter: Francis Chuang Assignee: Francis Chuang Fix For: 1.37.0, avatica-go-5.4.0, avatica-1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6358) Support all PostgreSQL 14 date/time patterns
Norman Jordan created CALCITE-6358: -- Summary: Support all PostgreSQL 14 date/time patterns Key: CALCITE-6358 URL: https://issues.apache.org/jira/browse/CALCITE-6358 Project: Calcite Issue Type: Sub-task Reporter: Norman Jordan Many of the date/time format patterns supported by PostgreSQL 14 are not supported in Calcite. * HH * US * * S * AM * A.M. * am * a.m. * PM * P.M. * pm * p.m. * Y,YYY * YYY * Y * IYYY * IYY * IY * I * BC * B.C. * bc * b.c. * AD * A.D. * ad * a.d. * MONTH * month * MON * mon * DAY * day * Dy * dy * IDDD * ID * TZH * TZM * OF Some format patterns in Calcite behave differently from PostgreSQL 14. * FF1 * FF2 * FF4 * FF5 * FF6 Also verify that the other existing format strings produce results that match PostgreSQL 14. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6357) Calcite enforces select argument count to be same as row schema fields which causes aliases to be ignored
Brachi Packter created CALCITE-6357:
---
Summary: Calcite enforces select argument count to be same as row
schema fields which causes aliases to be ignored
Key: CALCITE-6357
URL: https://issues.apache.org/jira/browse/CALCITE-6357
Project: Calcite
Issue Type: Bug
Reporter: Brachi Packter
Calcite RelBuilder.ProjectNamed cehcks if row size in the select is identical
to schema fields, if no, it creates a project with fields as they appear in the
select , meaning if they have aliases, they are returning with their aliases.
Here it checks if they are identical:
https://github.com/apache/calcite/blob/f14cf4c32b9079984a988bbad40230aa6a59b127/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2063
using RexUtil.isIdentity method:
```
public static boolean isIdentity(List exps,
RelDataType inputRowType) {
return inputRowType.getFieldCount() == exps.size()
&& containIdentity(exps, inputRowType, Litmus.IGNORE);
}
```
This is the problematic part `inputRowType.getFieldCount() == exps.size()`
And then it is ignored in the "rename" method later on
https://github.com/apache/calcite/blob/f14cf4c32b9079984a988bbad40230aa6a59b127/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2125
and alias is skipped
https://github.com/apache/calcite/blob/f14cf4c32b9079984a988bbad40230aa6a59b127/core/src/main/java/org/apache/calcite/tools/RelBuilder.java#L2137
This doesn't impact calcite queries, but in Apache Beam they are doing some
optimization on top of it,
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregateProjectMergeRule.java
which cause aliases to be ignored, and data is returning suddenly without
correct column field.
I believe the isIdentity check can causes more issues if not fixed, we need to
understand why is it enforced? isn't it valid to have different size of fields
in select from what we have in the schema?
In our case we have a one big row and we run on it different queries, each with
different fields in the select.
Beam issue
https://github.com/apache/beam/issues/30498
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6356) Upgrade Calcite to Avatica 1.25.0
Sergey Nuyanzin created CALCITE-6356:
Summary: Upgrade Calcite to Avatica 1.25.0
Key: CALCITE-6356
URL: https://issues.apache.org/jira/browse/CALCITE-6356
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Sergey Nuyanzin
Upgrade Calcite to Avatica 1.25.0
Also, as mentioned in comments for CALCITE-6053
{quote}
When the upgrade happens there are going to be some test failures in
SqlOperatorTest (CALCITE-5923) highlighting the part of the code that needs to
be updated. We left some comments to aid the cleanup.
{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6355) RelToSqlConverter[ORDER BY] generates an incorrect order by when NULLS LAST is used in non-projected field
Bruno Volpato created CALCITE-6355:
--
Summary: RelToSqlConverter[ORDER BY] generates an incorrect order
by when NULLS LAST is used in non-projected field
Key: CALCITE-6355
URL: https://issues.apache.org/jira/browse/CALCITE-6355
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Bruno Volpato
We are using RelToSqlConverter, and seeing issues with it generating invalid
queries when using _DESC NULLS LAST,_ specifically.
For example, this test query:
{code:java}
select "product_id"
from "product"
where "net_weight" is not null
group by "product_id"
order by MAX("net_weight") desc {code}
Gets resolved correctly, with a subquery, to:
{code:java}
SELECT "product_id"
FROM (SELECT "product_id", MAX("net_weight") AS "EXPR$1"
FROM "foodmart"."product"
WHERE "net_weight" IS NOT NULL
GROUP BY "product_id"
ORDER BY 2 DESC) AS "t3" {code}
However, if I specify `desc nulls last`:
{code:java}
select "product_id"
from "product"
where "net_weight" is not null
group by "product_id"
order by MAX("net_weight") desc nulls last {code}
It creates an invalid query (order by 2, but only one field was projected):
{code:java}
SELECT "product_id"
FROM "foodmart"."product"
WHERE "net_weight" IS NOT NULL
GROUP BY "product_id"
ORDER BY 2 DESC NULLS LAST {code}
Trying to troubleshoot it, it appears that without the `NULLS LAST`, we have
the following instance:
{code:java}
SqlBasicCall -> SqlNumericLiteral {code}
But when including it, it gets wrapped in another call:
{code:java}
SqlBasicCall -> SqlBasicCall -> SqlNumericLiteral {code}
So the [hasSortByOrdinal
method|https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L1938C21-L1958]
ends up returning {_}false{_}, which causes `needNewSubQuery` to incorrectly
report _false_ too.
It appears that the best way to deal with this is by using a recursion to find
numeric literals - but let me know if there are better ideas.
I plan to take a stab at this since I got enough context.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6354) Use gradle docker container directly when running tests using docker compose
Francis Chuang created CALCITE-6354: --- Summary: Use gradle docker container directly when running tests using docker compose Key: CALCITE-6354 URL: https://issues.apache.org/jira/browse/CALCITE-6354 Project: Calcite Issue Type: Task Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6353) Optimization CoreRules.PROJECT_REDUCE_EXPRESSIONS crashes while optimizing ARRAY_CONCAT expression
Mihai Budiu created CALCITE-6353:
Summary: Optimization CoreRules.PROJECT_REDUCE_EXPRESSIONS crashes
while optimizing ARRAY_CONCAT expression
Key: CALCITE-6353
URL: https://issues.apache.org/jira/browse/CALCITE-6353
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The following RelOptRulesTest
{code:java}
@Test void testArrayConcat() {
final String sql = "select array_concat(ARRAY [1, 2], ARRAY [3, 4])";
sql(sql).withFactory(
t -> t.withOperatorTable(
opTab ->
SqlLibraryOperatorTableFactory.INSTANCE.getOperatorTable(
SqlLibrary.STANDARD, SqlLibrary.BIG_QUERY)))
.withRule(CoreRules.PROJECT_REDUCE_EXPRESSIONS)
.check();
}
{code}
crashes with the following stack trace:
{code:java}
java.lang.RuntimeException: While compiling [ARRAY_CONCAT(ARRAY(1, 2), ARRAY(3,
4))]
at org.apache.calcite.rex.RexExecutable.compile(RexExecutable.java:73)
at org.apache.calcite.rex.RexExecutable.(RexExecutable.java:53)
at
org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:145)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressionsInternal(ReduceExpressionsRule.java:774)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:714)
{code}
It seems that the generated code passed to Janino is invalid:
Line 10, Column 5: Assignment conversion not possible from type
"java.util.ArrayList" to type "java.lang.Object[]"
org.codehaus.commons.compiler.CompileException: Line 10, Column 5: Assignment
conversion not possible from type "java.util.ArrayList" to type
"java.lang.Object[]"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6352) The map_contains_key function may return true when the key and mapkeytype types are different.
Caican Cai created CALCITE-6352:
---
Summary: The map_contains_key function may return true when the
key and mapkeytype types are different.
Key: CALCITE-6352
URL: https://issues.apache.org/jira/browse/CALCITE-6352
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Caican Cai
Fix For: 1.37.0
{code:java}
scala> val df = spark.sql("select map_contains_key(map(1, 'a', 2, 'b'), 2.0)")
val df: org.apache.spark.sql.DataFrame = [map_contains_key(map(1, a, 2, b),
2.0): boolean]
scala> df.show()
+--+
|map_contains_key(map(1, a, 2, b), 2.0)|
+--+
| true|
+--+
{code}
calcite return false
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6351) Update LICENSE and NOTICE for Jekyll website template
Francis Chuang created CALCITE-6351: --- Summary: Update LICENSE and NOTICE for Jekyll website template Key: CALCITE-6351 URL: https://issues.apache.org/jira/browse/CALCITE-6351 Project: Calcite Issue Type: Task Components: avatica Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6350) Unexpected result from UNION with literals expression
Evgeny Stanilovsky created CALCITE-6350:
---
Summary: Unexpected result from UNION with literals expression
Key: CALCITE-6350
URL: https://issues.apache.org/jira/browse/CALCITE-6350
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Evgeny Stanilovsky
Near expression need to return 'w' but empty result found.
{noformat}
select * from (select 'word' i union all select 'w' i) t1 where i='w'
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6349) CoreRules.PROJECT_REDUCE_EXPRESSIONS crashes on expression with ARRAY_REPEAT
Mihai Budiu created CALCITE-6349:
Summary: CoreRules.PROJECT_REDUCE_EXPRESSIONS crashes on
expression with ARRAY_REPEAT
Key: CALCITE-6349
URL: https://issues.apache.org/jira/browse/CALCITE-6349
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The following test in RelOptRulesTest causes a crash:
{code:java}
@Test void testArrayRepeat() {
final String sql = "select array_repeat(1, null)";
sql(sql)
.withFactory(
t -> t.withOperatorTable(
opTab ->
SqlLibraryOperatorTableFactory.INSTANCE.getOperatorTable(
SqlLibrary.STANDARD, SqlLibrary.SPARK)))
.withRule(CoreRules.PROJECT_REDUCE_EXPRESSIONS)
.check();
}
{code}
The crash is:
{code:java}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(INTEGER NOT NULL ARRAY NOT NULL EXPR$0) NOT NULL
expression type is RecordType(INTEGER NOT NULL ARRAY EXPR$0) NOT NULL
set is rel#4:LogicalProject.(input=HepRelVertex#3,exprs=[ARRAY_REPEAT(1,
null:DECIMAL(19, 9))])
expression is LogicalProject(EXPR$0=[null:INTEGER NOT NULL ARRAY])
LogicalValues(tuples=[[{ 0 }]])
Type mismatch:
rowtype of original rel: RecordType(INTEGER NOT NULL ARRAY NOT NULL EXPR$0) NOT
NULL
rowtype of new rel: RecordType(INTEGER NOT NULL ARRAY EXPR$0) NOT NULL
Difference:
EXPR$0: INTEGER NOT NULL ARRAY NOT NULL -> INTEGER NOT NULL ARRAY
at
org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:419)
at
org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:60)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:273)
at
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:288)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:317)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6348) ARRAY_OVERLAP with a NULL argument crashes the compiler
Mihai Budiu created CALCITE-6348:
Summary: ARRAY_OVERLAP with a NULL argument crashes the compiler
Key: CALCITE-6348
URL: https://issues.apache.org/jira/browse/CALCITE-6348
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The following SqlOperatorTest:
{code:java}
f.checkScalar("arrays_overlap(null, null)", true,
"NULL");
{code}
causes a crash:
{code:java}
at java.base/java.util.Objects.requireNonNull(Objects.java:222)
at
org.apache.calcite.sql.type.SqlTypeTransforms.lambda$static$6(SqlTypeTransforms.java:124)
at
org.apache.calcite.sql.type.SqlTypeTransformCascade.inferReturnType(SqlTypeTransformCascade.java:66)
at
org.apache.calcite.sql.SqlOperator.inferReturnType(SqlOperator.java:534)
at
org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:503)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:347)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6714)
{code}
This issue is similar to [CALCITE-6283]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6347) ARRAY_REPEAT with a string argument causes a compiler crash
Mihai Budiu created CALCITE-6347:
Summary: ARRAY_REPEAT with a string argument causes a compiler
crash
Key: CALCITE-6347
URL: https://issues.apache.org/jira/browse/CALCITE-6347
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The following SqlOperatorTest:
{code:java}
f.checkScalar("array_repeat('1', 2)", "['1', '1']",
"CHAR(1) NOT NULL ARRAY NOT NULL");
{code}
causes a compiler error:
{code}
Error while compiling generated Java code:
...
static final String $L4J$C$org_apache_calcite_runtime_SqlFunctions_repeat_1_2_
= org.apache.calcite.runtime.SqlFunctions.repeat("1", 2);
...
at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
at
org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:128)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1171)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:326)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:220)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:666)
{code}
This happens because the "repeat" function in SqlFunctions is overloaded to
implement both ARRAY_REPEAT and REPEAT.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6346) JdbcAdapter: Cast for dynamic filter arguments is lost
Corvin Kuebler created CALCITE-6346:
---
Summary: JdbcAdapter: Cast for dynamic filter arguments is lost
Key: CALCITE-6346
URL: https://issues.apache.org/jira/browse/CALCITE-6346
Project: Calcite
Issue Type: Improvement
Components: jdbc-adapter
Affects Versions: 1.36.0
Reporter: Corvin Kuebler
Hey,
today we encouraged some issues with the jdbc adapter behaviour.
We have a statement like this:
SELECT *
SELECT CASE WHEN CAST(? AS VARCHAR) = CAST(? AS VARCHAR) THEN "NAME" END
FROM "AUTHORS"
The parameters are ["some", 1].
The statement that is pushed via JDBC Adapter looks like this:
SELECT *
SELECT CASE WHEN ? = ? THEN "NAME" END
FROM "AUTHORS"
The cast in the resulting statement is lost and therefore we get:
ERROR: operator does not exist: character varying = integer Hint: No operator
matches the given name and argument types. You might need to add explicit type
casts.
As this example is not minimal I was able to reproduce it with the following
test in JdbcTests:
@Test void testFilterPush() {
CalciteAssert.that()
.with(CalciteAssert.Config.FOODMART_CLONE)
.query("SELECT * FROM \"foodmart\".\"sales_fact_1997\"" +
" WHERE cast(? as varchar) = cast(? as varchar)")
.planHasSql("SELECT *\n" +
"FROM \"foodmart\".\"sales_fact_1997\"\n" +
"WHERE cast(? as varchar) = cast(? as varchar)");
}
The test shows the exact behaviour as it fails with the following:
Caused by: java.sql.SQLSyntaxErrorException: data type cast needed for
parameter or null literal in statement [SELECT *
FROM "foodmart"."sales_fact_1997"
WHERE ? = ?
As can be seen the cast is also lost here.
To me it seems like an obvious bug, but maybe I am missing something here.
I hope some1 can help :)
BR,
Corvin
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6345) Intervals with more than 100 years are not supported
Mihai Budiu created CALCITE-6345:
Summary: Intervals with more than 100 years are not supported
Key: CALCITE-6345
URL: https://issues.apache.org/jira/browse/CALCITE-6345
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
Adding the following SqlValidatorTest:
{code:java}
expr("INTERVAL '100-2' YEAR TO MONTH").assertInterval(is(122L));
{code}
causes the following exception:
{code}
org.apache.calcite.runtime.CalciteContextException: From line 1, column 9 to
line 1, column 38: Interval field value 100 exceeds precision of YEAR(2) field
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:507)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:948)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:933)
at
org.apache.calcite.sql.SqlIntervalQualifier.fieldExceedsPrecisionException(SqlIntervalQualifier.java:1355)
at
org.apache.calcite.sql.SqlIntervalQualifier.checkLeadFieldInRange(SqlIntervalQualifier.java:475)
at
org.apache.calcite.sql.SqlIntervalQualifier.evaluateIntervalLiteralAsYearToMonth(SqlIntervalQualifier.java:626)
at
org.apache.calcite.sql.SqlIntervalQualifier.evaluateIntervalLiteral(SqlIntervalQualifier.java:1293)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateLiteral(SqlValidatorImpl.java:3429)
{code}
The spec does not limit years to 2 digits, so I don't know where the YEAR(2)
time is coming from.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6344) RelToSqlConverter invalid quotation for arrays and item operator(ansi dialect)
Anton Kovalevsky created CALCITE-6344:
-
Summary: RelToSqlConverter invalid quotation for arrays and item
operator(ansi dialect)
Key: CALCITE-6344
URL: https://issues.apache.org/jira/browse/CALCITE-6344
Project: Calcite
Issue Type: Bug
Reporter: Anton Kovalevsky
The context:
* Ansi Dialect
* having array of objects
* get an object by id and get property
"books"[0]."title" turns into ```books``[0]`.`title`
I added test case to RelToSqlConverterTest to show what I met:
{code:java}
@Test
void testSqlItemOperator2() {
new Sql(CalciteAssert.SchemaSpec.BOOKSTORE,
"SELECT MAP['title', \"books\"[0].\"title\"] from
\"bookstore\".\"authors\"",
AnsiSqlDialect.DEFAULT, SqlParser.Config.DEFAULT, ImmutableSet.of(),
UnaryOperator.identity(), null, ImmutableList.of())
.withSql("SELECT \"books\"[0].\"title\" from \"bookstore\".\"authors\"")
.schema(CalciteAssert.SchemaSpec.BOOKSTORE)
.ok("SELECT `books`[0].`title` FROM `bookstore`.`authors`");
} {code}
{noformat}
java.lang.AssertionError:
Expected: is "SELECT `books`[0].`title` FROM `bookstore`.`authors`"
but: was "SELECT ```books``[0]`.`title`\nFROM
`bookstore`.`authors`"{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6343) AS alias operator strips MEASUREness from measures
Barry Kelly created CALCITE-6343:
Summary: AS alias operator strips MEASUREness from measures
Key: CALCITE-6343
URL: https://issues.apache.org/jira/browse/CALCITE-6343
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Barry Kelly
Fix For: 1.37.0
[CALCITE-5869|https://issues.apache.org/jira/browse/CALCITE-5869] introduced a
change which removes MEASURE when inferring the return type when an operator is
applied.
The {{AS}} keyword for aliases is implemented as an operator in the SQL AST.
Using {{AS}} removes MEASURE when typing expressions that have an alias in the
{{SELECT}} clause.
Thus, the type of {{SELECT m}} and {{SELECT m AS m}} are different when {{m}}
is a measure. This is not desirable.
Proposed fix: don't change type when using the {{AS}} operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6342) Use highest input precision for datetime with/without local time zone
Dawid Wysakowicz created CALCITE-6342:
-
Summary: Use highest input precision for datetime with/without
local time zone
Key: CALCITE-6342
URL: https://issues.apache.org/jira/browse/CALCITE-6342
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Dawid Wysakowicz
In [CALCITE-5478] it was correctly improved that the precision should be taken
into account when calculating {{leastRestrictive}} type of {{Datetime}} types.
Unfortunately it still does not behave well for a call like:
{{leastRestrictive(TIMESTAMP_LTZ(0), TIMESTAMP(3)}}. It produces
{{TIMESTAMP_LTZ(0)}}.
In my opinion this should produce at least {{TIMESTAMP(3)}}.
I'd suggest modifying the code a little bit to:
{code}
if (type.getSqlTypeName().getFamily() ==
resultType.getSqlTypeName().getFamily()
&& type.getSqlTypeName().allowsPrec()
&& type.getPrecision() != resultType.getPrecision()) {
final int precision =
SqlTypeUtil.maxPrecision(resultType.getPrecision(),
type.getPrecision());
resultType = createSqlType(type.getSqlTypeName(),
precision);
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6341) DynamicRecordType join query will generate uncompailable code
Yonatan Graber created CALCITE-6341:
---
Summary: DynamicRecordType join query will generate uncompailable
code
Key: CALCITE-6341
URL: https://issues.apache.org/jira/browse/CALCITE-6341
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0, 1.35.0, 1.34.0, 1.33.0, 1.32.0, 1.31.0, 1.30.0,
1.29.0, 1.28.0, 1.27.0, 1.26.0, 1.25.0, 1.24.0, 1.23.0
Reporter: Yonatan Graber
See full minimal reproduction in
[https://github.com/yonatang/calcite-join-issue/|https://github.com/yonatang/calcite-join-issue/tree/main]
—
When creating a table with {{DynamicRecordType}} rowType, a joined query will
generate a code that won't get compiled.
For example, with the relevant schema, this query:
{{SELECT d1.a ,d2.a FROM static_table d1 join static_table d2 on d1.a=d2.a}}
Will not execute, because the generated code will contain a Baz class with the
following Comparator class:
{code:java}
new java.util.Comparator(){
public int compare(Object v0, Object v1) {
final int c;
c =
org.apache.calcite.runtime.Utilities.compareNullsLastForMergeJoin((Comparable)
v0, (Comparable) v1);
if (c != 0) {
return c;
}
return 0;
}
public int compare(Object o0, Object o1) {
return this.compare(o0, o1);
}
} {code}
This fails because it has two {{int compare(Object o0, Object o1)}} methods.
Notice this issue was introduced at Calcite 1.23.0. The query will work as
expected in versions 1.22 and below.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6340) RelBuilder always creates Project with Convention.NONE during aggregate_
Adam Kennedy created CALCITE-6340:
-
Summary: RelBuilder always creates Project with Convention.NONE
during aggregate_
Key: CALCITE-6340
URL: https://issues.apache.org/jira/browse/CALCITE-6340
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Adam Kennedy
In the RelBuilder method aggregate_, when (config.pruneInputOfAggregate() && r
instanceof Project) line 2443 the Project will be rewritten to remove unused
columns.
When this happens, the new Project will be created with the following line
{{{}2487: r =
{}}}{{{}2488: project.copy(cluster.traitSet(), project.getInput(),
newProjects,{}}}
{{2489: builder.build());}}
The use of cluster.traitSet() returns emptyTraitSet which is always going to
use Convention.NONE regardless of the Rebuilder's ProjectFactory.
In the case of a query plan using a non-Logical convention FOO, with FooProject
nodes that require the FOO convention, RelBuilder will normally happily produce
FooProject nodes with FOO convention, allowing many CoreRules to be easily
reused for custom Conventions.
However, while RelBuilder will produce FooProject with FOO convention in the
majority of cases, for the one specific case of column pruning a Project input
to an aggregate, it will instead product a FooProject with NONE convention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6339) Replace hashicorp/go-uuid with google/uuid
Francis Chuang created CALCITE-6339: --- Summary: Replace hashicorp/go-uuid with google/uuid Key: CALCITE-6339 URL: https://issues.apache.org/jira/browse/CALCITE-6339 Project: Calcite Issue Type: Task Components: avatica-go Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-go-5.4.0 The hashicorp/uuid library generates strings that look like uuids but are purely random. It is also not as popular or as active as the google/uuid library. The google/uuid library generates real uuids that are RFC compliant. The library is also more popular and actively maintained. These ids are used to generate connection ids, so whether they are RFC compliant or not is inconsequential, however, it is a good idea to generate compliant uuids if downstream users behind avatica expects correct and compliant uuids. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6338) RelMdCollation#project can return an incomplete list of collations
Ruben Q L created CALCITE-6338:
--
Summary: RelMdCollation#project can return an incomplete list of
collations
Key: CALCITE-6338
URL: https://issues.apache.org/jira/browse/CALCITE-6338
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Ruben Q L
Assignee: Ruben Q L
{{RelMdCollation#project}} can return an incomplete list of collations.
(I'll try to produce a unit test, for now I'll just describe the situation)
Let us say we have a Project that projects the following expressions (notice
that $2 will become $1 and $2 after the projection): $0, $2, $2, $3
The Project's input has collation [2, 3]
In order to calculate the Project's own collation, {{RelMdCollation#project}}
will be called, and a MultiMap targets will be computed because, as in this
case, a certain "source field" (e.g. 2) can have multiple project targets (e.g.
1 and 2). However, when the collation is being computed, *only the first target
will be considered* (and the rest will be discarded):
{code}
public static @Nullable List project(RelMetadataQuery mq,
RelNode input, List projects) {
...
for (RelFieldCollation ifc : ic.getFieldCollations()) {
final Collection integers = targets.get(ifc.getFieldIndex());
if (integers.isEmpty()) {
continue loop; // cannot do this collation
}
fieldCollations.add(ifc.withFieldIndex(integers.iterator().next()));
// <-- HERE!!
}
{code}
Because of this, the Project's collation will be [1 3], but there is also
another valid one ([2 3]), so the correct (complete) result should be: [1 3] [2
3]
This seems a minor problem, but it can be the root cause of more relevant
issues. For instance, at the moment I have a scenario (not so easy to reproduce
with a unit test) where a certain plan with a certain combination of rules in a
HepPlanner results in a StackOverflow due to SortJoinTransposeRule being fired
infinitely. The root cause is that, after the first application, the rule does
not detect that the Join's left input is already sorted (due to the previous
application of the rule), because there is a "problematic" Project on it (that
shows the problem described above), which returns only one collation, whereas
the second collation (the one being discarded) is the Sort's collation, so it
would be one that would prevent the SortJoinTransposeRule from being re-applied
over and over.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6337) Distinguish naked measure support between inside and outside aggregation
Barry Kelly created CALCITE-6337:
Summary: Distinguish naked measure support between inside and
outside aggregation
Key: CALCITE-6337
URL: https://issues.apache.org/jira/browse/CALCITE-6337
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: Barry Kelly
Measure type and AGGREGATE function (CALCITE-5105) introduced a configuration
flag for naked measures.
Naked measures are measure-typed columns that are referenced outside an
{{AGGREGATE()}} function call.
At Looker, we're trying to support a specific semantic using measure-typed
columns:
- selecting from a naked measure-typed column outside an aggregating query
evaluates to {{NULL}}
This permits basic introspection of the schema like `{{SELECT * FROM foo
LIMIT 1}}`
For this, we need naked measures outside a grouping context.
- selecting from a naked measure-typed column inside an aggregating query is an
error
We want all expressions in a grouping query to either be part of the grouping
key or to have
an aggregation function applied. For this, we don't want naked measures
inside a grouping context.
This change proposes:
- {{nakedMeasuresOutsideAggregatingQuery}} - boolean flag permitting measure
references outside aggregating query
- {{nakedMeasuresInsideAggregatingQuery}} - boolean flag permitting measure
references inside aggregating query
- deprecating {{nakedMeasures}} flag, which is now implemented by setting both
of the above to the same value
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6336) When inserting an insert SQL statement containing non-ANSII characters into an Oracle database, an exception occurs
gunveda created CALCITE-6336:
Summary: When inserting an insert SQL statement containing
non-ANSII characters into an Oracle database, an exception occurs
Key: CALCITE-6336
URL: https://issues.apache.org/jira/browse/CALCITE-6336
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: gunveda
*The following content is a Chinese translation*
Use the u& prefix for non-ANSII characters. This method is not supported in
Oracle, and should be replaced with the unistr function.
Modify as follows
{code:java}
public class CustomOracleSqlDialect extends OracleSqlDialect {
private static final char[] HEXITS = {
'0', '1', '2', '3', '4', '5', '6', '7',
'8', '9', 'a', 'b', 'c', 'd', 'e', 'f',
};
public static final SqlDialect DEFAULT = new
CustomOracleSqlDialect(DEFAULT_CONTEXT);
public CustomOracleSqlDialect(Context context) {
super(context);
}
@Override
public void quoteStringLiteralUnicode(StringBuilder buf, String val) {
buf.append("unistr('");
for (int i = 0; i < val.length(); i++) {
char c = val.charAt(i);
if (c < 32 || c >= 128) {
buf.append('\\');
buf.append(HEXITS[(c >> 12) & 0xf]);
buf.append(HEXITS[(c >> 8) & 0xf]);
buf.append(HEXITS[(c >> 4) & 0xf]);
buf.append(HEXITS[c & 0xf]);
} else if (c == '\'' || c == '\\') {
buf.append(c);
buf.append(c);
} else {
buf.append(c);
}
}
buf.append("')");
}
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6335) Quidem tests should allow specifying optimization passes to apply to programs
Mihai Budiu created CALCITE-6335: Summary: Quidem tests should allow specifying optimization passes to apply to programs Key: CALCITE-6335 URL: https://issues.apache.org/jira/browse/CALCITE-6335 Project: Calcite Issue Type: Improvement Components: core Affects Versions: 1.36.0 Reporter: Mihai Budiu This facility would make it much easier to test various optimization passes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6334) Release Avatica 1.25.0
Francis Chuang created CALCITE-6334: --- Summary: Release Avatica 1.25.0 Key: CALCITE-6334 URL: https://issues.apache.org/jira/browse/CALCITE-6334 Project: Calcite Issue Type: Task Reporter: Francis Chuang Assignee: Francis Chuang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6333) Queries with distinct aggregations with filter throw NPE when planned using joins
Abhishek Agarwal created CALCITE-6333:
-
Summary: Queries with distinct aggregations with filter throw NPE
when planned using joins
Key: CALCITE-6333
URL: https://issues.apache.org/jira/browse/CALCITE-6333
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Abhishek Agarwal
If I changed the test "testDistinctWithFilterWithoutGroupBy" and use the
AGGREGATE_EXPAND_DISTINCT_AGGREGATES_TO_JOIN rule instead of
AGGREGATE_EXPAND_DISTINCT_AGGREGATES
{code:java}
@Test void testDistinctWithFilterWithoutGroupBy() {
final String sql = "SELECT SUM(comm), COUNT(DISTINCT sal) FILTER (WHERE sal
> 1000)\n"
+ "FROM emp";
sql(sql)
.withRule(CoreRules.AGGREGATE_EXPAND_DISTINCT_AGGREGATES_TO_JOIN)
.check();
}
{code}
the test fails with the exception below
{code:java}
java.lang.NullPointerException: sourceOf.get(2)
at java.base/java.util.Objects.requireNonNull(Objects.java:347)
at
org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule.doRewrite(AggregateExpandDistinctAggregatesRule.java:740)
at
org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule.onMatch(AggregateExpandDistinctAggregatesRule.java:260)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
at
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420)
at
org.apache.calcite.plan.hep.HepPlanner.executeRuleInstance(HepPlanner.java:243)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance$State.execute(HepInstruction.java:178)
at
org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211)
at
com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
at
org.apache.calcite.plan.hep.HepProgram$State.execute(HepProgram.java:118)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:205)
at
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:191)
at
org.apache.calcite.test.RelOptFixture.checkPlanning(RelOptFixture.java:378)
at org.apache.calcite.test.RelOptFixture.check(RelOptFixture.java:330)
at org.apache.calcite.test.RelOptFixture.check(RelOptFixture.java:314)
at
org.apache.calcite.test.RelOptRulesTest.testDistinctWithFilterWithoutGroupBy(RelOptRulesTest.java:1462)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(
[jira] [Created] (CALCITE-6332) Optimization CoreRules.AGGREGATE_EXPAND_DISTINCT_AGGREGATES_TO_JOIN produces incorrect results for aggregates with groupSets
Mihai Budiu created CALCITE-6332:
Summary: Optimization
CoreRules.AGGREGATE_EXPAND_DISTINCT_AGGREGATES_TO_JOIN produces incorrect
results for aggregates with groupSets
Key: CALCITE-6332
URL: https://issues.apache.org/jira/browse/CALCITE-6332
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The optimization rule does not seem to consider the groupSets at all.
The following two queries produce the same resulting plan:
{code:sql}
select count(distinct deptno) as cd, count(*) as c
from emp
group by cube(deptno)
{code}
{code:sql}
select count(distinct deptno) as cd, count(*) as c
from emp
group by deptno
{code}
(Notice that one query has a cube, while the other one doesn't)
The produced plan is:
{code}
LogicalProject(CD=[$1], C=[$2]), id = 196
LogicalAggregate(group=[{0}], CD=[COUNT($0)], C=[$SUM0($1)]), id = 201
LogicalAggregate(group=[{0}], C=[COUNT()]), id = 198
LogicalProject(DEPTNO=[$8]), id = 192
LogicalTableScan(table=[[schema, EMP]]), id = 163
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6331) The third-party source code file doesn't have a License file.
Calvin Kirs created CALCITE-6331: Summary: The third-party source code file doesn't have a License file. Key: CALCITE-6331 URL: https://issues.apache.org/jira/browse/CALCITE-6331 Project: Calcite Issue Type: Improvement Reporter: Calvin Kirs [https://github.com/apache/calcite/blob/main/LICENSE#L184-L196] We reference these third-party source code files but don't provide the corresponding LICENSE files. FYI : https://www.apache.org/legal/release-policy.html#license-file -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6330) Print the average row size when when explaining an operator
mengdou created CALCITE-6330:
Summary: Print the average row size when when explaining an
operator
Key: CALCITE-6330
URL: https://issues.apache.org/jira/browse/CALCITE-6330
Project: Calcite
Issue Type: Improvement
Components: core
Reporter: mengdou
Assignee: mengdou
In this time, when we dump the plan of a RelNode Tree, there is no average row
size exported in the output string, even if SqlExplainLevel.ALL_ATTRIBUTES is
specified.
Because the implementation in explain_() in class RelWriterImpl doesn't include
the metric average_row_size:
{code:java}
switch (detailLevel) {
case ALL_ATTRIBUTES:
s.append(": rowcount = ")
.append(mq.getRowCount(rel))
.append(", cumulative cost = ")
.append(mq.getCumulativeCost(rel));
}
switch (detailLevel) {
case NON_COST_ATTRIBUTES:
case ALL_ATTRIBUTES:
if (!withIdPrefix) {
// If we didn't print the rel id at the start of the line, print
// it at the end.
s.append(", id = ").append(rel.getId());
}
break;
} {code}
So I'd like to add this metric by calling md.getAverageRowSize(rel)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6329) Use weighted-average calculation for the columns in Union operator
mengdou created CALCITE-6329:
Summary: Use weighted-average calculation for the columns in Union
operator
Key: CALCITE-6329
URL: https://issues.apache.org/jira/browse/CALCITE-6329
Project: Calcite
Issue Type: Improvement
Components: core
Affects Versions: 1.37.0
Reporter: mengdou
Assignee: mengdou
In the method averageColumnSizes(Union rel, RelMetadataQuery mq) of class
RelMdSize, it uses a simple average for every column of the current Union
operator to calculate column_size, so does row_size, which is calculated
according to all the column sizes.
{code:java}
public List averageColumnSizes(Union rel, RelMetadataQuery mq) {
final int fieldCount = rel.getRowType().getFieldCount();
List> inputColumnSizeList = new ArrayList<>();
for (RelNode input : rel.getInputs()) {
final List inputSizes = mq.getAverageColumnSizes(input);
if (inputSizes != null) {
inputColumnSizeList.add(inputSizes);
}
}
switch (inputColumnSizeList.size()) {
case 0:
return null; // all were null
case 1:
return inputColumnSizeList.get(0); // all but one were null
}
final ImmutableNullableList.Builder sizes =
ImmutableNullableList.builder();
int nn = 0;
for (int i = 0; i < fieldCount; i++) {
double d = 0d;
int n = 0;
for (List inputColumnSizes : inputColumnSizeList) {
Double d2 = inputColumnSizes.get(i);
if (d2 != null) {
d += d2;
++n;
++nn;
}
}
sizes.add(n > 0 ? d / n : null);
}
if (nn == 0) {
return null; // all columns are null
}
return sizes.build();
} {code}
But it doesn't take the rowCount of each input into account, which may
introduce a bad case and make a bad impact on the downstream operators. for
example:
{code:java}
# We have two tables A and B
# Logical Plan
ShuffleWrite
Union
TableScan(table=A)
TableScan(table=B)
# stats
row_count(A) = 1E9, row_size(A) = 10
row_count(B) = 1E5, row_size(B) = 100
row_count(Union) = 1.0001E10, row_size(Union) = 55 # using simple average
row_count(ShuffleWrite) = row_count(Union) = 1.0001E10 # inherits from Union
row_size(ShuffleWrite) = row_size(Union) = 55 # inherits from Union
# cost estimation of ShuffleWrite, which is more larger than real input bytes
input_bytes(ShuffleWrite) = 55 * 1.0001E10 = 5.50055E11
input_bytes(Union) = 1E9 * 10 + 1E5 * 100 = 1.001E11{code}
So I suggest that we can take row count of Union into consideration and use
weighted average to calculate every column sizes and the final row size instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6328) The BigQuery functions SAFE_* do not match the BigQuery specification
Mihai Budiu created CALCITE-6328:
Summary: The BigQuery functions SAFE_* do not match the BigQuery
specification
Key: CALCITE-6328
URL: https://issues.apache.org/jira/browse/CALCITE-6328
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The BigQuery dialect does not support DECIMAL values with arbitrary types: it
only supports two fixed types: NUMERIC and BIGNUMERIC, both with fixed
precision and scale.
The runtime implementation of the SAFE_* functions uses the following helper in
SqlFunctions:
{code:java}
/** Returns whether a BigDecimal value is safe (that is, has not overflowed).
* According to BigQuery, BigDecimal overflow occurs if the precision is
greater
* than 76 or the scale is greater than 38. */
private static boolean safeDecimal(BigDecimal b) {
return b.scale() <= 38 && b.precision() <= 76;
}
{code}
This helper does not handle correctly NUMERIC value, only BIGNUMERIC.
Moreover, all the tests in SqlOperatorTests use a type system which doesn't
even support DECIMAL values wider than 38 digits. So a test like the following:
{code:java}
f.checkNull("safe_add(cast(-9.9e75 as DECIMAL(76, 0)), "
+ "cast(-9.9e75 as DECIMAL(76, 0)))");
{code}
cannot even create the expected BigDecimal value correctly.
This surfaced during the attempt to fix [CALCITE-6322]: once the casts to
DECIMAL are implemented, some of these tests break.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6327) getValidatedNodeTypeIfKnown should never throw
Claude Brisson created CALCITE-6327: --- Summary: getValidatedNodeTypeIfKnown should never throw Key: CALCITE-6327 URL: https://issues.apache.org/jira/browse/CALCITE-6327 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.36.0 Reporter: Claude Brisson During validation, when a SqlNode has been rewritten (for instance when a COALESCE call has been rewritten as a CASE call) but does not yet have a RelDataType, the method SqlValidatorImpl.getValidatedNodeTypeIfKnown() throws an exception because it relies on SqlValidatorImpl.getValidatedNodeType(originalExpr), not on SqlValidatorImpl.getValidatedNodeTypeIfKnown(originalExpr). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6326) Add LOG10 and LN function (enabled in Mysql, Spark library)
Caican Cai created CALCITE-6326: --- Summary: Add LOG10 and LN function (enabled in Mysql, Spark library) Key: CALCITE-6326 URL: https://issues.apache.org/jira/browse/CALCITE-6326 Project: Calcite Issue Type: Sub-task Components: core Affects Versions: 1.36.0 Reporter: Caican Cai Fix For: 1.37.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6325) Add LOG function (enabled in Mysql, Spark library)
Caican Cai created CALCITE-6325: --- Summary: Add LOG function (enabled in Mysql, Spark library) Key: CALCITE-6325 URL: https://issues.apache.org/jira/browse/CALCITE-6325 Project: Calcite Issue Type: Sub-task Reporter: Caican Cai -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6324) Type inferred for result of STDDEV, VAR_SAMP, etc. is incorrect
Mihai Budiu created CALCITE-6324: Summary: Type inferred for result of STDDEV, VAR_SAMP, etc. is incorrect Key: CALCITE-6324 URL: https://issues.apache.org/jira/browse/CALCITE-6324 Project: Calcite Issue Type: Bug Components: core Affects Versions: 1.36.0 Reporter: Mihai Budiu These functions are all use the same type inference algorithm, essentially the algorithm used by AVG. But if the values processed are decimal, STDDEV (and others) need much higher precision to represent the result. (I am not sure that the inference is right for integer types either, btw.) This surfaced during the implementation of a fix for [CALCITE-6322]: if we use the type inferred for these functions, the result overflows and causes a runtime exception. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6323) Serialize return type during RelJson.toJson(RexNode node) for SqlKind.SAFE_CAST
Tanner Clary created CALCITE-6323: - Summary: Serialize return type during RelJson.toJson(RexNode node) for SqlKind.SAFE_CAST Key: CALCITE-6323 URL: https://issues.apache.org/jira/browse/CALCITE-6323 Project: Calcite Issue Type: Bug Reporter: Tanner Clary Assignee: Tanner Clary This is essentially the exact same as [CALCITE-5607] for SqlKind.SAFE_CAST[1]. We need to preserve the desired cast type so that when going from json->rex, we know what type we want to cast to. We do this for the standard CAST, so we're really just trying to align the two functions. [1] https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rel/externalize/RelJson.java#L635 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6322) Casts to DECIMAL types are ignored
Mihai Budiu created CALCITE-6322:
Summary: Casts to DECIMAL types are ignored
Key: CALCITE-6322
URL: https://issues.apache.org/jira/browse/CALCITE-6322
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
The following SqlOperatorTest fails:
{code:java}
f.checkScalar("CAST(1.123 AS DECIMAL(4, 0))", "1.0", "DECIMAL(4, 0) NOT NULL");
{code}
The result computed by Calcite is 1.123, ignoring the scale of the DECIMAL
result.
Spark, Postgres, MySQL all return 1.0.
I have marked this as a major bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6321) Add copy(List constants) method to Window class.
Thomas D'Silva created CALCITE-6321: --- Summary: Add copy(List constants) method to Window class. Key: CALCITE-6321 URL: https://issues.apache.org/jira/browse/CALCITE-6321 Project: Calcite Issue Type: Improvement Reporter: Thomas D'Silva -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6320) Parse SQL with Hive-style and Spark-style unescaped character literals
EveyWu created CALCITE-6320: Summary: Parse SQL with Hive-style and Spark-style unescaped character literals Key: CALCITE-6320 URL: https://issues.apache.org/jira/browse/CALCITE-6320 Project: Calcite Issue Type: Bug Reporter: EveyWu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6319) Add negative test to pow and power functions
Caican Cai created CALCITE-6319: --- Summary: Add negative test to pow and power functions Key: CALCITE-6319 URL: https://issues.apache.org/jira/browse/CALCITE-6319 Project: Calcite Issue Type: Test Affects Versions: 1.36.0 Reporter: Caican Cai Fix For: 1.37.0 Add negative test to pow and power functions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6318) Add some rules that WeTune discovers and verifies
Jiajun Xie created CALCITE-6318: --- Summary: Add some rules that WeTune discovers and verifies Key: CALCITE-6318 URL: https://issues.apache.org/jira/browse/CALCITE-6318 Project: Calcite Issue Type: Improvement Components: core Reporter: Jiajun Xie [WeTune|https://ipads.se.sjtu.edu.cn/_media/publications/wtune_extend.pdf] is a rule generator that can automatically discover new query rewrite rules for SQL query optimization. The paper was published in 2022, and some optimization rules have recently been implemented in Calcite. e.g. CALCITE-6214 is rule No.2 in Table.7 of the paper. But there are still many rules that have not been implemented, which are worth discussing. Here are all the discovered rewrite [rules|https://ipads.se.sjtu.edu.cn:1312/opensource/wetune/-/blob/main/wtune_data/rules/rules.test.txt?ref_type=heads]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6317) Optimization CoreRules.PROJECT_REDUCE_EXPRESSIONS is unsound
Mihai Budiu created CALCITE-6317:
Summary: Optimization CoreRules.PROJECT_REDUCE_EXPRESSIONS is
unsound
Key: CALCITE-6317
URL: https://issues.apache.org/jira/browse/CALCITE-6317
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
Here is a query taken from agg.iq:
{code:sql}
select deptno, gender, grouping_id(deptno, gender, deptno), count(*) as c
from emp
where deptno = 10
group by rollup(gender, deptno)
{code}
The query plan initially is
{code}
LogicalProject(DEPTNO=[$1], GENDER=[$0], EXPR$2=[$2], C=[$3]), id = 72
LogicalAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
EXPR$2=[GROUPING_ID($1, $0, $1)], C=[COUNT()]), id = 71
LogicalProject(GENDER=[$2], DEPTNO=[$1]), id = 70
LogicalFilter(condition=[=($1, 10)]), id = 66
LogicalTableScan(table=[[schema, EMP]]), id = 65
{code}
After applying PROJECT_REDUCE_EXPRESSIONS the plan looks like:
{code}
LogicalProject(DEPTNO=[CAST(10):INTEGER], GENDER=[$0], EXPR$2=[$2],
C=[$3]), id = 82
LogicalAggregate(group=[{0, 1}], groups=[[{0, 1}, {0}, {}]],
EXPR$2=[GROUPING_ID($1, $0, $1)], C=[COUNT()]), id = 78
LogicalProject(GENDER=[$2], DEPTNO=[CAST(10):INTEGER]), id = 84
LogicalFilter(condition=[=($1, 10)]), id = 74
LogicalTableScan(table=[[schema, EMP]]), id = 65
{code}
The problem is in the outer LogicalProject, where the value 10 has replaced
DEPTNO.
However, DEPTNO can also be NULL, because of the groups in the LogicalAggregate.
The constant should not be pushed past the aggregation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6316) Update Javadoc for CALCITE-5607
Oliver Lee created CALCITE-6316: --- Summary: Update Javadoc for CALCITE-5607 Key: CALCITE-6316 URL: https://issues.apache.org/jira/browse/CALCITE-6316 Project: Calcite Issue Type: Task Reporter: Oliver Lee Assignee: Oliver Lee -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6314) Support PostgreSQL RANDOM
James Duong created CALCITE-6314: Summary: Support PostgreSQL RANDOM Key: CALCITE-6314 URL: https://issues.apache.org/jira/browse/CALCITE-6314 Project: Calcite Issue Type: Sub-task Reporter: James Duong This is an alias for RAND(), except it should not support passing in a seed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6315) Support PostgreSQL TO_CHAR, TO_DATE, TO_TIMESTAMP
James Duong created CALCITE-6315: Summary: Support PostgreSQL TO_CHAR, TO_DATE, TO_TIMESTAMP Key: CALCITE-6315 URL: https://issues.apache.org/jira/browse/CALCITE-6315 Project: Calcite Issue Type: Sub-task Reporter: James Duong PostgreSQL supports different format strings than the version we have implemented. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6312) Support PostgreSQL LOG
James Duong created CALCITE-6312: Summary: Support PostgreSQL LOG Key: CALCITE-6312 URL: https://issues.apache.org/jira/browse/CALCITE-6312 Project: Calcite Issue Type: Sub-task Reporter: James Duong * We have the BigQuery implementation which has a 1-arg overload that uses base e (ln), and a 2-arg overload which is LOG(value, base) * PostgreSQL's 1-arg version uses base 10 and the 2-arg overload has the order reversed – LOG(base, value) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-6311) Support PostgreSQL DATE_PART
James Duong created CALCITE-6311:
Summary: Support PostgreSQL DATE_PART
Key: CALCITE-6311
URL: https://issues.apache.org/jira/browse/CALCITE-6311
Project: Calcite
Issue Type: Sub-task
Reporter: James Duong
* PostgreSQL and Redshift let the date_part parameter be a string instead of a
just an enum-like identifier (eg DATE_PART('year', ...) and DATE_PART(year,
...) are both supported.
* SQL Server does not support using a string here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (CALCITE-6313) Support PostgreSQL POW
James Duong created CALCITE-6313: Summary: Support PostgreSQL POW Key: CALCITE-6313 URL: https://issues.apache.org/jira/browse/CALCITE-6313 Project: Calcite Issue Type: Sub-task Reporter: James Duong * Our existing implementation always returns double. * PostgreSQL allows returning a numeric instead of a double when inputs are numeric. Not sure if this makes sense though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
