[
https://issues.apache.org/jira/browse/HIVE-28680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930255#comment-17930255
]
Krisztian Kasa commented on HIVE-28680:
---------------------------------------
[~Aggarwal_Raghav]
1. The stacktrace in the description is seems to be a 3rd issue: it seems that
there is a compilation error while compiling the MV definition query. Could you
please check if hive.log contains an exception with a message like
{code:java}
CBO failed, skipping CBO.
{code}
2. Based on the stack trace in [^NPE_Stacktrace.txt] the issue is related to
authorization
{code:java}
at
org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewUtils.checkPrivilegeForMaterializedViews(HiveMaterializedViewUtils.java:492)
{code}
These are the major steps of MV based query rewrite:
1. Collect all MVs available for rewrite and add their plan to the
{{{}RelOptPlanner{}}}. By default these are coming from
{{{}HiveMaterializedViewsRegistry{}}}. This registry contains all the
precompiled plans of the MVs which are enabled for query rewrite. The
compilation is done at HS2 start in a background thread.
2. Start the planner to find a possible rewrite. When a possible rewrite found
the planner makes a cost based decision whether the rewritten plan is better
than the original. To do it stats has to be loaded and If an MV stored by
Iceberg cached in {{HiveMaterializedViewsRegistry}} does not exists an
exception is thrown because in case of Iceberg the stats are provided by the
IcebergStorageHandler which tries to load them from the filesystem. This issue
is tracked by HIVE-28773.
3. Check if any rewrite is made and return the original query plan if not.
4. If there is a possible rewrite collect the MVs from the rewritten plan.
5. Check the privilege of these MVs. - This is the point where the exception
was thrown in your case.
I run all 3 attached sql scripts as q test using apache master locally and all
of them passed.
Are you running these scripts on a cluster with multiple HS2 instances? Are
these issues intermittent?
> Invalid function when querying MV with rewrite enable
> -----------------------------------------------------
>
> Key: HIVE-28680
> URL: https://issues.apache.org/jira/browse/HIVE-28680
> Project: Hive
> Issue Type: Bug
> Reporter: Raghav Aggarwal
> Assignee: Raghav Aggarwal
> Priority: Major
> Attachments: NPE_Stacktrace.txt, failure_case.sql, failure_case2.sql,
> failure_case3.sql
>
>
> Attaching the q file to reproduce the issue. The same select query passes
> when re-write is disabled and fails when re-write is by default enabled.
>
> Here is the stacktrace:
> {code:java}
> 2024-12-23 21:49:29,423 ERROR thrift.ThriftCLIService: Failed to execute
> statement [request:
> TExecuteStatementReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:36
> 97 08 39 80 BD 45 B3 94 96 1A B3 26 96 87 77, secret:20 1F E2 CA 1E EA 41 16
> A3 E0 27 81 41 02 25 4F)), statement:select * from (select * from emps where
> empid < 120) t join depts using (deptno), confOverlay:{}, runAsync:true,
> queryTimeout:0)]
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: SemanticException [Error 10011]: Line 1:73 Invalid
> function 'deptno'; Query ID:
> raghav_20241223214929_d7f8ee22-45d6-4992-8493-590b9ff23867
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:212)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:268)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:558)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:543)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:316)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:652)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:73
> Invalid function 'deptno'
> at
> org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:953)
> at
> org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1481)
> at
> org.apache.hadoop.hive.ql.lib.CostLessRuleDispatcher.dispatch(CostLessRuleDispatcher.java:66)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:101)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:231)
> at
> org.apache.hadoop.hive.ql.parse.type.ExprNodeTypeCheck.genExprNode(ExprNodeTypeCheck.java:49)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:13606)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13561)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13529)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3792)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.pushJoinFilters(SemanticAnalyzer.java:10234)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.pushJoinFilters(SemanticAnalyzer.java:10212)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12421)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12298)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:13051)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13170)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12676)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:109)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:499)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:451)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:415)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
