[
https://issues.apache.org/jira/browse/HIVE-28564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soumyakanti Das updated HIVE-28564:
-----------------------------------
Description:
To repro this, create four tpcds table in postgres by copying
q_test_tpcds_tables_schema.postgres.sql to {{hive/data/scripts/}} directory.
Add jdbc_non_flat_filter_condition.q to {{ql/src/test/queries/clientpositive/}}
and run
{noformat}
mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver
-Dtest.output.overwrite=true -Dqfile=jdbc_non_flat_filter_condition.q{noformat}
Error stack:
{noformat}
[INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.238
s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
[ERROR]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[jdbc_non_flat_filter_condition]
Time elapsed: 8.409 s <<< FAILURE!
java.lang.AssertionError: AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000),
OR(AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0))
at org.apache.calcite.rel.core.Filter.<init>(Filter.java:76)
at
org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.<init>(JdbcRules.java:629)
at
org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.copy(JdbcRules.java:635)
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCExpandExpressionsRule$FilterCondition.onMatch(JDBCExpandExpressionsRule.java:85)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271)
at
org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2469)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2428)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPostJoinOrderingTransform(CalcitePlanner.java:2347)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1739)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1573)
at
org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1325)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:573)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13164)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:466)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:109)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:499)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:451)
at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:415){noformat}
In non jdbc Hive query
([cbo_query47.q.out|https://github.com/apache/hive/blob/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d/ql/src/test/results/clientpositive/perf/tpcds30tb/tez/cbo_query47.q.out#L59]),
the condition is:
{noformat}
AND(IN($6, 2000, 1999, 2001), OR(=($6, 2000), IN(ROW($6, $8), ROW(1999, 12),
ROW(2001, 1)))){noformat}
[JDBCExpandExpressionsRule|https://github.com/apache/hive/blob/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java#L209]
converts it to:
{noformat}
AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000), OR(AND(=($1, 1999), =($2, 12)),
AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0)){noformat}
which is not flat as there is an OR within an OR which can be simplified using
{{RexSimplify}} to:
{noformat}
AND(OR(=($1, 2000), AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1))),
IN($1, 2000, 1999, 2001), IS NOT NULL($0)){noformat}
was:
To repro this, create four tpcds table in postgres by copying
q_test_tpcds_tables_schema.postgres.sql to {{hive/data/scripts/}} directory.
Add jdbc_non_flat_filter_condition.q to {{ql/src/test/queries/clientpositive/}}
and run
{noformat}
mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver
-Dtest.output.overwrite=true -Dqfile=jdbc_non_flat_filter_condition.q{noformat}
Error stack:
{noformat}
[INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.238
s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
[ERROR]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[jdbc_non_flat_filter_condition]
Time elapsed: 8.409 s <<< FAILURE!
java.lang.AssertionError: AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000),
OR(AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0))
at org.apache.calcite.rel.core.Filter.<init>(Filter.java:76)
at
org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.<init>(JdbcRules.java:629)
at
org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.copy(JdbcRules.java:635)
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCExpandExpressionsRule$FilterCondition.onMatch(JDBCExpandExpressionsRule.java:85)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271)
at
org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2469)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2428)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPostJoinOrderingTransform(CalcitePlanner.java:2347)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1739)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1573)
at
org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1325)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:573)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13164)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:466)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:109)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:499)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:451)
at
org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:415){noformat}
In non jdbc Hive queries, the condition is:
{noformat}
AND(IN($6, 2000, 1999, 2001), OR(=($6, 2000), IN(ROW($6, $8), ROW(1999, 12),
ROW(2001, 1)))){noformat}
[JDBCExpandExpressionsRule|https://github.com/apache/hive/blob/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java#L209]
converts it to:
{noformat}
AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000), OR(AND(=($1, 1999), =($2, 12)),
AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0)){noformat}
which is not flat as there is an OR within an OR which can be simplified using
{{RexSimplify}} to:
{noformat}
AND(OR(=($1, 2000), AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1))),
IN($1, 2000, 1999, 2001), IS NOT NULL($0)){noformat}
> AssertionError for jdbc queries when JDBCExpandExpressionsRule produces non
> flat Filter conditions
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-28564
> URL: https://issues.apache.org/jira/browse/HIVE-28564
> Project: Hive
> Issue Type: Bug
> Security Level: Public(Viewable by anyone)
> Components: CBO, JDBC
> Affects Versions: 4.1.0
> Reporter: Soumyakanti Das
> Assignee: Soumyakanti Das
> Priority: Major
> Attachments: jdbc_non_flat_filter_condition.q,
> q_test_tpcds_tables_schema.postgres.sql
>
>
> To repro this, create four tpcds table in postgres by copying
> q_test_tpcds_tables_schema.postgres.sql to {{hive/data/scripts/}} directory.
> Add jdbc_non_flat_filter_condition.q to
> {{ql/src/test/queries/clientpositive/}} and run
> {noformat}
> mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver
> -Dtest.output.overwrite=true
> -Dqfile=jdbc_non_flat_filter_condition.q{noformat}
> Error stack:
> {noformat}
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 15.238 s <<< FAILURE! - in
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
> [ERROR]
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[jdbc_non_flat_filter_condition]
> Time elapsed: 8.409 s <<< FAILURE!
> java.lang.AssertionError: AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000),
> OR(AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0))
> at org.apache.calcite.rel.core.Filter.<init>(Filter.java:76)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.<init>(JdbcRules.java:629)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.copy(JdbcRules.java:635)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCExpandExpressionsRule$FilterCondition.onMatch(JDBCExpandExpressionsRule.java:85)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2469)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2428)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPostJoinOrderingTransform(CalcitePlanner.java:2347)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1739)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1573)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1325)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:573)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13164)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:466)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:109)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:499)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:451)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:415){noformat}
>
> In non jdbc Hive query
> ([cbo_query47.q.out|https://github.com/apache/hive/blob/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d/ql/src/test/results/clientpositive/perf/tpcds30tb/tez/cbo_query47.q.out#L59]),
> the condition is:
> {noformat}
> AND(IN($6, 2000, 1999, 2001), OR(=($6, 2000), IN(ROW($6, $8), ROW(1999, 12),
> ROW(2001, 1)))){noformat}
> [JDBCExpandExpressionsRule|https://github.com/apache/hive/blob/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java#L209]
> converts it to:
> {noformat}
> AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000), OR(AND(=($1, 1999), =($2, 12)),
> AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0)){noformat}
> which is not flat as there is an OR within an OR which can be simplified
> using {{RexSimplify}} to:
> {noformat}
> AND(OR(=($1, 2000), AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1))),
> IN($1, 2000, 1999, 2001), IS NOT NULL($0)){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
