You should be able to see your connections with the netstat command. There might be a conncetion leak in the Hive code as well.
On Mon, Jun 5, 2017 at 11:20 PM, Diego Fustes Villadóniga <[email protected] > wrote: > Dear colleagues, > > We have installed a Cloudera kerberized cluster, version 5.11. We have > enabled Sentry, in order to ensure autorization for Hive, Impala, Solr, > etc. However, since then, we see downtimes in the Hive Metastore, > apparently because of the connection with the Sentry server. This is the > stacktrace of the metastore: > > 2017-05-31 12:10:15,539 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-50]: MetaException(message:Failed to connect to Sentry > service null) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.getSentryServiceClient( > SentryMetastorePostEventListener.java:308) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.dropSentryPrivileges( > SentryMetastorePostEventListener.java:351) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.dropSentryDbPrivileges( > SentryMetastorePostEventListener.java:318) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.onDropDatabase( > SentryMetastorePostEventListener.java:199) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.drop_database_core(HiveMetaStore.java:1180) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.drop_database(HiveMetaStore.java:1212) > at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler. > invokeInternal(RetryingHMSHandler.java:140) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler. > invoke(RetryingHMSHandler.java:99) > at com.sun.proxy.$Proxy14.drop_database(Unknown Source) > at org.apache.hadoop.hive.metastore.api. > ThriftHiveMetastore$Processor$drop_database.getResult( > ThriftHiveMetastore.java:9005) > at org.apache.hadoop.hive.metastore.api. > ThriftHiveMetastore$Processor$drop_database.getResult( > ThriftHiveMetastore.java:8989) > at org.apache.thrift.ProcessFunction.process( > ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process( > TBaseProcessor.java:39) > at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ > TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:735) > at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ > TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:730) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1920) > at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ > TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:730) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( > TThreadPoolServer.java:286) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > On the other hand, before the metastore goes down, Sentry starts to throw > this warning: > > 2017-05-31 12:26:35,619 WARN org.apache.thrift.server.TThreadPoolServer: > Task has been rejected by ExecutorService 10 times till timedout, reason: > java.util.concurrent.RejectedExecutionException: Task > org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b<mailto: > org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b> > rejected from java.util.concurrent.ThreadPoolExecutor@5c1d81c7[ > Running<mailto:java.util.concurrent.ThreadPoolExecutor@5c1d81c7[Running>, > pool size = 500, active threads = 500, queued tasks = 0, completed tasks = > 1107] > > It seems that there are too many open connections to Sentry. May this be > that some client is not closing them? > How can we overcome this error? We can't use the cluster given these > downtimes.... > > Regards, > > Diego > > > Diego Fustes Villadóniga, Arquitecto Big Data, CCIM > >
