Ran Tao created CALCITE-6163:
--------------------------------
Summary: The spark ARRAY function gives exception when elements
contain row(struct) type and NULL
Key: CALCITE-6163
URL: https://issues.apache.org/jira/browse/CALCITE-6163
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Ran Tao
when we run
{noformat}
array(row(1,2), null)
{noformat}
It will give 'Parameters must be of the same type' exception.
{noformat}
org.apache.calcite.runtime.CalciteContextException: Parameters must be of the
same type
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:507)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:948)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:933)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5517)
at
org.apache.calcite.sql.SqlCallBinding.newValidationError(SqlCallBinding.java:414)
at
org.apache.calcite.sql.type.SameOperandTypeChecker.checkOperandTypesImpl(SameOperandTypeChecker.java:100)
at
org.apache.calcite.sql.type.SameOperandTypeChecker.checkOperandTypes(SameOperandTypeChecker.java:59)
at
org.apache.calcite.sql.SqlOperator.checkOperandTypes(SqlOperator.java:761)
at org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:498)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:347)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6575)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6562)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:166)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1926)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1911)
at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:276)
at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
at org.apache.calcite.sql.SqlNode.validateExpr(SqlNode.java:275)
at org.apache.calcite.sql.SqlOperator.validateCall(SqlOperator.java:475)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateCall(SqlValidatorImpl.java:6232)
at org.apache.calcite.sql.SqlCall.validate(SqlCall.java:143)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1091)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:797)
at
org.apache.calcite.sql.test.AbstractSqlTester.parseAndValidate(AbstractSqlTester.java:161)
at
org.apache.calcite.sql.test.AbstractSqlTester.validateAndThen(AbstractSqlTester.java:249)
at
org.apache.calcite.test.SqlOperatorFixtureImpl.lambda$forEachQueryValidateAndThen$1(SqlOperatorFixtureImpl.java:154)
at
org.apache.calcite.sql.test.AbstractSqlTester.forEachQuery(AbstractSqlTester.java:446)
at
org.apache.calcite.test.SqlOperatorFixtureImpl.forEachQueryValidateAndThen(SqlOperatorFixtureImpl.java:153)
at
org.apache.calcite.test.SqlOperatorFixtureImpl.checkType(SqlOperatorFixtureImpl.java:130)
at
org.apache.calcite.sql.test.SqlOperatorFixture.checkScalar(SqlOperatorFixture.java:226)
at
org.apache.calcite.test.SqlOperatorTest.testArrayFunction(SqlOperatorTest.java:10551)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
{noformat}
however, the calcite std array constructor and spark array, it shows like below:
{noformat}
0: jdbc:calcite:model=src/test/resources/mode> select array[row(1,2), null];
+----------------+
| EXPR$0 |
+----------------+
| [{1, 2}, null] |
+----------------+
{noformat}
{noformat}
spark-sql (default)> select array((1,2), null);
[{"col1":1,"col2":2},null]
{noformat}
They supported row and null literal to coexist(the generated component type is
nullable row/struct).
The reason is because of current SameOperandTypeChecker of spark array will
call *SqlTypeUtil.isComparable* like below:
{noformat}
if (type1.isStruct() != type2.isStruct()) {return false;}
{noformat}
It will reject row(struct) type and NULL, then throw exception.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
