[
https://issues.apache.org/jira/browse/HIVE-25565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vipin updated HIVE-25565:
-------------------------
Environment:
Aws EMR - 6.3.0
Spark - 3.1.1
Hive - 3.1.2
was:
```
EMR - 6.3.0
Spark - 3.1.1
Hive - 3.1.2
```
> Materialized view Rebuild issue Aws EMR
> ---------------------------------------
>
> Key: HIVE-25565
> URL: https://issues.apache.org/jira/browse/HIVE-25565
> Project: Hive
> Issue Type: Bug
> Environment: Aws EMR - 6.3.0
> Spark - 3.1.1
> Hive - 3.1.2
> Reporter: Vipin
> Priority: Major
>
> We have Materialized views built on top of Hudi tables which are
> hive-sync'd.We have Materialized views built on top of Hudi tables which are
> hive-sync'd.
> Hive uses AWS Glue for its metastore catalog. We are running into issue
> whenever we are trying to "**rebuild**" Hive materialized views. Please note,
> creation of materialized views works fine. It's only rebuild which is
> failing. However, it does seem the rebuild actually seems to work behind the
> scenes but its throws some exception causing EMR steps to fail.
> Can anyone please guide us here, about any config changes that we need to do
> or anything. Any help will be great.
> ```
> EMR - 6.3.0
> Spark - 3.1.1
> Hive - 3.1.2
> ```
>
> The stack trace of the exception -
> > FAILED: Hive Internal Error:
> > org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking
> > FailureHook. hooks: java.lang.NullPointerException at
> > org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)
> > at
> > org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)
> > at
> > org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)
> > at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616) at
> > org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386) at
> > org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) at
> > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at
> > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> > at
> > org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
> > at java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:422) at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748))
> > org.apache.hadoop.hive.ql.metadata.HiveException: Error while invoking
> > FailureHook. hooks:
> > java.lang.NullPointerException
> > at
> > org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)>
> > at
> > org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)>
> > at
> > org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)>
> > at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)>
> > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at
> > org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at
> > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)>
> > at java.security.AccessController.doPrivileged(Native Method)> at
> > javax.security.auth.Subject.doAs(Subject.java:422)> at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)>
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)> at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)>
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)>
> > at java.lang.Thread.run(Thread.java:748)> > at
> > org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:302)>
> > at
> > org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)>
> > at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)>
> > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at
> > org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at
> > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)>
> > at java.security.AccessController.doPrivileged(Native Method)> at
> > javax.security.auth.Subject.doAs(Subject.java:422)> at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)>
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)> at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)>
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)>
> > at java.lang.Thread.run(Thread.java:748)> Caused by:
> > java.lang.NullPointerException> at
> > org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)>
> > at
> > org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)>
> > ... 19 more
> > INFO : Completed executing
> > command(queryId=hive_20210927174227_c0ccf861-e7df-4ae4-be7d-b93774d29c25);
> > Time taken: 15.402 seconds> INFO : Concurrency mode is disabled, not
> > creating a lock manager> Error: Error while processing statement: FAILED:
> > Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error
> > while invoking FailureHook. hooks: java.lang.NullPointerException> at
> > org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)>
> > at
> > org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)>
> > at
> > org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)>
> > at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)>
> > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at
> > org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at
> > org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at
> > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)>
> > at java.security.AccessController.doPrivileged(Native Method)> at
> > javax.security.auth.Subject.doAs(Subject.java:422)> at
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)>
> > at
> > org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)>
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)> at
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)> at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)>
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)>
> > at java.lang.Thread.run(Thread.java:748)> ) (state=08S01,code=12)>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
