Yes, that makes a lot of sense. Thanks a lot Alex Diego
-----Mensaje original----- De: Alexander Kolbasov [mailto:[email protected]] Enviado el: martes, 6 de junio de 2017 18:05 Para: [email protected] Asunto: Re: Problems with Sentry, possibly Denial of Service It looks like SENTRY-1759 - looks like it should be back-ported to 5.11. - Alex > On Jun 6, 2017, at 3:46 AM, Diego Fustes Villadóniga <[email protected]> > wrote: > > We have done this, and it seems that the service responsible for the high > number of connections is Kafka, even though it is not actively used at the > moment. It should happen when Kafka reloads the privileges cache, where it > looks like it's not closing previous connections. > > Should we open a bug for Kafka? > > Regards, > > Diego > > > -----Mensaje original----- > De: Alexander Kolbasov [mailto:[email protected]] Enviado el: martes, > 6 de junio de 2017 8:37 > Para: dev <[email protected]> > Asunto: Re: Problems with Sentry, possibly Denial of Service > > You should be able to see your connections with the netstat command. There > might be a conncetion leak in the Hive code as well. > > On Mon, Jun 5, 2017 at 11:20 PM, Diego Fustes Villadóniga > <[email protected] >> wrote: > >> Dear colleagues, >> >> We have installed a Cloudera kerberized cluster, version 5.11. We >> have enabled Sentry, in order to ensure autorization for Hive, >> Impala, Solr, etc. However, since then, we see downtimes in the Hive >> Metastore, apparently because of the connection with the Sentry >> server. This is the stacktrace of the metastore: >> >> 2017-05-31 12:10:15,539 ERROR >> org.apache.hadoop.hive.metastore.RetryingHMSHandler: >> [pool-5-thread-50]: MetaException(message:Failed to connect to Sentry >> service null) >> at org.apache.sentry.binding.metastore. >> SentryMetastorePostEventListener.getSentryServiceClient( >> SentryMetastorePostEventListener.java:308) >> at org.apache.sentry.binding.metastore. >> SentryMetastorePostEventListener.dropSentryPrivileges( >> SentryMetastorePostEventListener.java:351) >> at org.apache.sentry.binding.metastore. >> SentryMetastorePostEventListener.dropSentryDbPrivileges( >> SentryMetastorePostEventListener.java:318) >> at org.apache.sentry.binding.metastore. >> SentryMetastorePostEventListener.onDropDatabase( >> SentryMetastorePostEventListener.java:199) >> at org.apache.hadoop.hive.metastore.HiveMetaStore$ >> HMSHandler.drop_database_core(HiveMetaStore.java:1180) >> at org.apache.hadoop.hive.metastore.HiveMetaStore$ >> HMSHandler.drop_database(HiveMetaStore.java:1212) >> at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at org.apache.hadoop.hive.metastore.RetryingHMSHandler. >> invokeInternal(RetryingHMSHandler.java:140) >> at org.apache.hadoop.hive.metastore.RetryingHMSHandler. >> invoke(RetryingHMSHandler.java:99) >> at com.sun.proxy.$Proxy14.drop_database(Unknown Source) >> at org.apache.hadoop.hive.metastore.api. >> ThriftHiveMetastore$Processor$drop_database.getResult( >> ThriftHiveMetastore.java:9005) >> at org.apache.hadoop.hive.metastore.api. >> ThriftHiveMetastore$Processor$drop_database.getResult( >> ThriftHiveMetastore.java:8989) >> at org.apache.thrift.ProcessFunction.process( >> ProcessFunction.java:39) >> at org.apache.thrift.TBaseProcessor.process( >> TBaseProcessor.java:39) >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ >> TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:735) >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ >> TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:730) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at org.apache.hadoop.security.UserGroupInformation.doAs( >> UserGroupInformation.java:1920) >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ >> TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:730) >> at >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( >> TThreadPoolServer.java:286) >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1145) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> >> On the other hand, before the metastore goes down, Sentry starts to >> throw this warning: >> >> 2017-05-31 12:26:35,619 WARN org.apache.thrift.server.TThreadPoolServer: >> Task has been rejected by ExecutorService 10 times till timedout, reason: >> java.util.concurrent.RejectedExecutionException: Task >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b<mailto: >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b> >> rejected from java.util.concurrent.ThreadPoolExecutor@5c1d81c7[ >> Running<mailto:java.util.concurrent.ThreadPoolExecutor@5c1d81c7[Runni >> n >> g>, pool size = 500, active threads = 500, queued tasks = 0, >> g>completed >> tasks = 1107] >> >> It seems that there are too many open connections to Sentry. May this >> be that some client is not closing them? >> How can we overcome this error? We can't use the cluster given these >> downtimes.... >> >> Regards, >> >> Diego >> >> >> Diego Fustes Villadóniga, Arquitecto Big Data, CCIM >> >>
