[
https://issues.apache.org/jira/browse/HIVE-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314440#comment-17314440
]
Ashish Kumar edited comment on HIVE-23710 at 4/4/21, 8:33 AM:
--------------------------------------------------------------
The given patch is not working for version 2.3.4
Modifying this file. -
metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
Getting NullPointerException -
{quote}2021-04-04T13:37:26,016 INFO [main] server.HiveServer2: Starting
HiveServer2
2021-04-04T13:37:26,237 INFO [main] service.AbstractService:
Service:OperationManager is inited.
2021-04-04T13:37:26,237 INFO [main] service.AbstractService:
Service:SessionManager is inited.
2021-04-04T13:37:26,237 INFO [main] service.AbstractService:
Service:CLIService is inited.
2021-04-04T13:37:26,240 INFO [main] service.AbstractService:
Service:ThriftBinaryCLIService is inited.
2021-04-04T13:37:26,240 INFO [main] service.AbstractService:
Service:HiveServer2 is inited.
2021-04-04T13:37:26,241 INFO [main] server.HiveServer2: Shutting down
HiveServer2
2021-04-04T13:37:26,242 WARN [main] server.HiveServer2: Error starting
HiveServer2 on attempt 2, will retry in 60000ms
java.lang.NullPointerException
at com.sun.proxy.$Proxy36.isCompatibleWith(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.ql.metadata.Hive.isCompatible(Hive.java:345)
~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:311)
~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:166)
~[hive-service-2.3.4.jar:2.3.4]
at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607)
[hive-service-2.3.4.jar:2.3.4]
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100)
[hive-service-2.3.4.jar:2.3.4]
at
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855)
[hive-service-2.3.4.jar:2.3.4]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724)
[hive-service-2.3.4.jar:2.3.4]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_282]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
[hadoop-common-2.7.3.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
[hadoop-common-2.7.3.jar:?]
{quote}
was (Author: ashubhumca):
The given patch is not working for version 2.4
Modifying this file. -
metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
Getting NullPointerException -
{quote}2021-04-04T13:37:26,016 INFO [main] server.HiveServer2: Starting
HiveServer2
2021-04-04T13:37:26,237 INFO [main] service.AbstractService:
Service:OperationManager is inited.
2021-04-04T13:37:26,237 INFO [main] service.AbstractService:
Service:SessionManager is inited.
2021-04-04T13:37:26,237 INFO [main] service.AbstractService:
Service:CLIService is inited.
2021-04-04T13:37:26,240 INFO [main] service.AbstractService:
Service:ThriftBinaryCLIService is inited.
2021-04-04T13:37:26,240 INFO [main] service.AbstractService:
Service:HiveServer2 is inited.
2021-04-04T13:37:26,241 INFO [main] server.HiveServer2: Shutting down
HiveServer2
2021-04-04T13:37:26,242 WARN [main] server.HiveServer2: Error starting
HiveServer2 on attempt 2, will retry in 60000ms
java.lang.NullPointerException
at com.sun.proxy.$Proxy36.isCompatibleWith(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.ql.metadata.Hive.isCompatible(Hive.java:345)
~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:311)
~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:166)
~[hive-service-2.3.4.jar:2.3.4]
at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607)
[hive-service-2.3.4.jar:2.3.4]
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100)
[hive-service-2.3.4.jar:2.3.4]
at
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855)
[hive-service-2.3.4.jar:2.3.4]
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724)
[hive-service-2.3.4.jar:2.3.4]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_282]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
[hadoop-common-2.7.3.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
[hadoop-common-2.7.3.jar:?]
{quote}
> Add table meta cache limit when starting Hive server2
> -----------------------------------------------------
>
> Key: HIVE-23710
> URL: https://issues.apache.org/jira/browse/HIVE-23710
> Project: Hive
> Issue Type: Improvement
> Environment: Hive 2.3.6
> Reporter: Deegue
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23710.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When we start up Hive server2, it will connect to metastore to get table meta
> info by database and cache them. If there are many tables in a database,
> however, will exceed `hive.metastore.client.socket.timeout`.
> Then exception thrown like:
> {noformat}
> 2020-06-17T11:38:27,595 WARN [main] metastore.RetryingMetaStoreClient:
> MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s.
> getTableObjectsByName
> org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name_req(ThriftHiveMetastore.java:1596)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name_req(ThriftHiveMetastore.java:1583)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:1370)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableObjectsByName(SessionHiveMetaStoreClient.java:238)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:206)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at com.sun.proxy.$Proxy38.getTableObjectsByName(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at com.sun.proxy.$Proxy38.getTableObjectsByName(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllTableObjects(Hive.java:1343)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry.init(HiveMaterializedViewsRegistry.java:127)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:167)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724)
> ~[hive-service-2.3.6.jar:2.3.6]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> ~[hadoop-common-2.6.0-cdh5.16.1.jar:?]
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> ~[hadoop-common-2.6.0-cdh5.16.1.jar:?]
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_121]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[?:1.8.0_121]
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> ~[?:1.8.0_121]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[?:1.8.0_121]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> ~[?:1.8.0_121]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> ~[?:1.8.0_121]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> ~[hive-exec-2.3.6.jar:2.3.6]
> ... 32 more
> {noformat}
> The solution is add `hive.server2.init.load.table.limit` to control max
> number of tables for one TCP request from Hive server2 to metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
