Vipin created HIVE-25565:
----------------------------
Summary: Materialized view Rebuild issue Aws EMR
Key: HIVE-25565
URL: https://issues.apache.org/jira/browse/HIVE-25565
Project: Hive
Issue Type: Bug
Environment: ```
EMR - 6.3.0
Spark - 3.1.1
Hive - 3.1.2
```
Reporter: Vipin
We have Materialized views built on top of Hudi tables which are hive-sync'd.We
have Materialized views built on top of Hudi tables which are hive-sync'd.
Hive uses AWS Glue for its metastore catalog. We are running into issue
whenever we are trying to "**rebuild**" Hive materialized views. Please note,
creation of materialized views works fine. It's only rebuild which is failing.
However, it does seem the rebuild actually seems to work behind the scenes but
its throws some exception causing EMR steps to fail.
Can anyone please guide us here, about any config changes that we need to do or
anything. Any help will be great.
```
EMR - 6.3.0
Spark - 3.1.1
Hive - 3.1.2
```
The stack trace of the exception -
> FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking
> FailureHook. hooks: java.lang.NullPointerException at
> org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)
> at
> org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)
> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)
> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616) at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748))
> org.apache.hadoop.hive.ql.metadata.HiveException: Error while invoking
> FailureHook. hooks:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)>
> at
> org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)>
> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)>
> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)>
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)>
> at java.security.AccessController.doPrivileged(Native Method)> at
> javax.security.auth.Subject.doAs(Subject.java:422)> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)>
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)>
> at java.lang.Thread.run(Thread.java:748)> > at
> org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:302)>
> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)>
> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)>
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)>
> at java.security.AccessController.doPrivileged(Native Method)> at
> javax.security.auth.Subject.doAs(Subject.java:422)> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)>
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)>
> at java.lang.Thread.run(Thread.java:748)> Caused by:
> java.lang.NullPointerException> at
> org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)>
> at
> org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)>
> ... 19 more
> INFO : Completed executing
> command(queryId=hive_20210927174227_c0ccf861-e7df-4ae4-be7d-b93774d29c25);
> Time taken: 15.402 seconds> INFO : Concurrency mode is disabled, not
> creating a lock manager> Error: Error while processing statement: FAILED:
> Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error
> while invoking FailureHook. hooks: java.lang.NullPointerException> at
> org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)>
> at
> org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)>
> at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)>
> at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2616)> at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2386)> at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)> at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)> at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)>
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)>
> at java.security.AccessController.doPrivileged(Native Method)> at
> javax.security.auth.Subject.doAs(Subject.java:422)> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)>
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)>
> at java.lang.Thread.run(Thread.java:748)> ) (state=08S01,code=12)>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
