soumyakanti3578 commented on code in PR #5196:
URL: https://github.com/apache/hive/pull/5196#discussion_r1940290564
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java:
##########
@@ -182,12 +191,75 @@ public RexNode visitCall(RexCall inputCall) {
switch (call.getKind()) {
case IN:
return transformIntoOrAndClause(rexBuilder, call);
+ case SEARCH:
+ return expandSearchAndRemoveRowOperator(rexBuilder, call);
default:
break;
}
}
return RexUtil.isFlat(node) ? node : RexUtil.flatten(rexBuilder, node);
}
+
+ private RexNode expandSearchAndRemoveRowOperator(RexBuilder rexBuilder,
RexCall expression) {
+ RexLiteral literal = (RexLiteral) expression.getOperands().get(1);
+ Sarg<?> sarg = Objects.requireNonNull(literal.getValueAs(Sarg.class),
"Sarg");
+
+ if (sarg.isPoints()) {
+ // if it is a row operator, we'll have
+ // SEARCH(ROW($1, $2), Sarg[[1, 2], [3, 4]]
Review Comment:
If you run this query:
https://github.com/apache/hive/blob/84cdf6003678ed673d59d223ef7f4dac1aa6fdc0/ql/src/test/queries/clientpositive/external_jdbc_table2.q#L137
and check the log, you can see that just before applying the jdbc rules, the
CBO plan is:
```
2025-02-03T14:55:17,977 DEBUG [3918befe-f357-489b-8ad8-53780cdf7bb9 main]
parse.CalcitePlanner: Plan after view-based rewriting:
HiveProject(ikey=[$0])
HiveFilter(condition=[AND(SEARCH($1, Sarg[10L:BIGINT, 20L:BIGINT]:BIGINT),
SEARCH($2, Sarg[1.515E1:DOUBLE, 2.525E1:DOUBLE]:DOUBLE), SEARCH(ROW($1, $2),
Sarg[[10L:BIGINT, 1.515E1:DOUBLE]:RecordType(BIGINT NOT NULL EXPR$0, DOUBLE NOT
NULL EXPR$1), [20L:BIGINT, 2.525E1:DOUBLE]:RecordType(BIGINT NOT NULL EXPR$0,
DOUBLE NOT NULL EXPR$1)]:RecordType(BIGINT NOT NULL EXPR$0, DOUBLE NOT NULL
EXPR$1)), IS NOT NULL($0))])
HiveJdbcConverter(convention=[JDBC.DERBY])
JdbcProject(ikey=[$0], bkey=[$1], dkey=[$3])
JdbcHiveTableScan(table=[[default, db1_ext_auth1]], table:alias=[p])
```
After this point, we apply `JDBCExpandExpressionsRule` where we need to
handle:
```
SEARCH(ROW($1, $2), Sarg[[10L:BIGINT, 1.515E1:DOUBLE]:RecordType(BIGINT NOT
NULL EXPR$0, DOUBLE NOT NULL EXPR$1)
```
Calcite seems to be expanding the expression incorrectly when
`RexUtil.searchShuttle` is called. It treats the `SEARCH` as points, i.e.,
`isPoints()` returns true here:
https://github.com/apache/calcite/blob/calcite-1.33.0/core/src/main/java/org/apache/calcite/rex/RexUtil.java#L626
which produces the following, keeping `ROW` in the predicate:
```
OR(=(ROW($1, $2), [10:BIGINT, 1.515E1:DOUBLE]:RecordType(BIGINT NOT NULL
EXPR$0, DOUBLE NOT NULL EXPR$1)), =(ROW($1, $2), [20:BIGINT,
2.525E1:DOUBLE]:RecordType(BIGINT NOT NULL EXPR$0, DOUBLE NOT NULL EXPR$1)))
```
Derby doesn't like it and fails with:
```
java.lang.RuntimeException: java.io.IOException: java.io.IOException:
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Caught
exception while trying to execute query: 'ROW' is not recognized as a function
or procedure.
at
org.apache.hadoop.hive.ql.exec.FetchTask.executeInner(FetchTask.java:210)
at org.apache.hadoop.hive.ql.exec.FetchTask.execute(FetchTask.java:95)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:197)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:143)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:138)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:235)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:733)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:703)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:116)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
at
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
...
...
Caused by: ERROR 42Y03: 'ROW' is not recognized as a function or procedure.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at
org.apache.derby.impl.sql.compile.StaticMethodCallNode.bindExpressionMinion(Unknown
Source)
at
org.apache.derby.impl.sql.compile.StaticMethodCallNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.JavaToSQLValueNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryOperatorNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryComparisonOperatorNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryOperatorNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryLogicalOperatorNode.bindExpression(Unknown
Source)
at org.apache.derby.impl.sql.compile.OrNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryOperatorNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryLogicalOperatorNode.bindExpression(Unknown
Source)
at org.apache.derby.impl.sql.compile.AndNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryOperatorNode.bindExpression(Unknown
Source)
at
org.apache.derby.impl.sql.compile.BinaryLogicalOperatorNode.bindExpression(Unknown
Source)
at org.apache.derby.impl.sql.compile.AndNode.bindExpression(Unknown
Source)
at org.apache.derby.impl.sql.compile.SelectNode.bindExpressions(Unknown
Source)
at
org.apache.derby.impl.sql.compile.DMLStatementNode.bindExpressions(Unknown
Source)
at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(Unknown
Source)
at org.apache.derby.impl.sql.compile.CursorNode.bindStatement(Unknown
Source)
at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
at
org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown
Source)
... 68 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
