Pengcheng Xiong created HIVE-8997:
-------------------------------------
Summary: Groupby index will fail if an indexed group by operator
is followed by a non-indexed group by operator
Key: HIVE-8997
URL: https://issues.apache.org/jira/browse/HIVE-8997
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
following ql_rewrite_gbtoidx.q, if we run
explain
select ckeysum, count(ckeysum)
from
(select l_shipdate, count(l_shipdate) as ckeysum
from lineitem_ix
group by l_shipdate) tabA
group by ckeysum
We will get an error:
junit.framework.AssertionFailedError: Client Execution failed with error code =
40000 running
The trace is
MismatchedTokenException(-1!=12)
at
org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.charSetStringLiteral(HiveParser_IdentifiersParser.java:6099)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.constant(HiveParser_IdentifiersParser.java:5891)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6478)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6641)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:7026)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:7086)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7270)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7430)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7590)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7750)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7909)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8439)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9452)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9571)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9730)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6363)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.groupByExpression(HiveParser_IdentifiersParser.java:1386)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.groupByClause(HiveParser_IdentifiersParser.java:774)
at
org.apache.hadoop.hive.ql.parse.HiveParser.groupByClause(HiveParser.java:44007)
at
org.apache.hadoop.hive.ql.parse.HiveParser.singleSelectStatement(HiveParser.java:41504)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41135)
at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41072)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40125)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40001)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1519)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at
org.apache.hadoop.hive.ql.optimizer.index.RewriteParseContextGenerator.generateOperatorTree(RewriteParseContextGenerator.java:67)
at
org.apache.hadoop.hive.ql.optimizer.index.RewriteQueryUsingAggregateIndex$NewQueryGroupbySchemaProc.process(RewriteQueryUsingAggregateIndex.java:255)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at
org.apache.hadoop.hive.ql.optimizer.index.RewriteQueryUsingAggregateIndexCtx.invokeRewriteQueryProc(RewriteQueryUsingAggregateIndexCtx.java:151)
at
org.apache.hadoop.hive.ql.optimizer.index.RewriteGBUsingIndex.rewriteOriginalQuery(RewriteGBUsingIndex.java:382)
at
org.apache.hadoop.hive.ql.optimizer.index.RewriteGBUsingIndex.transform(RewriteGBUsingIndex.java:135)
at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:177)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10177)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1169)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:199)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:362)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:297)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:849)
at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx_notwork(TestCliDriver.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Exception: Client Execution failed with error code = 40000 running
The main reason is that
After Optimizer,org.apache.hadoop.hive.ql.optimizer.ColumnPruner, we should get
the operator tree as
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-SEL[6]-GBY[7]-RS[8]-GBY[9]-SEL[10]-FS[11]
Here, TS[0], SEL[1], GBY[2], GBY[4] should be rewritten. However, because the
default graph walker is used in the code, GBY[9] and GBY[7] are going to be
rewritten. This causes the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
