[
https://issues.apache.org/jira/browse/SENTRY-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1780:
---------------------------------------
Component/s: Sentry
> FullUpdateInitializer does not kill the threads whenever getFullHMSSnapshot
> throws an exception
> -----------------------------------------------------------------------------------------------
>
> Key: SENTRY-1780
> URL: https://issues.apache.org/jira/browse/SENTRY-1780
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Sergio Peña
> Assignee: Sergio Peña
>
> Errors may happen inside the FullUpdateInitializer.getFullHMSSnapshot()
> causing to throw an exception. When
> this happens, the current threads running are not killed and continue
> consuming resources while they're alive.
> We should kill those threads. The information collected by them won't be used
> anymore.
> I noticed this issue on the logs:
> {noformat}
> ### HMSFollower 1st run ######
> 2017-05-23 13:57:56,976 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)]
> HMSFollower run is called.
> ### Starting to get a snapshot (2 databases found (db2, default) and
> submitted for execution) ###
> 2017-05-23 13:57:56,978 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:396)]
> get full snapshot
> 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)]
> submit database: db2
> 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)]
> submit database: default
> ### Thread 1 started: db2 ###
> 2017-05-23 13:57:56,979 (pool-150-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)]
> begin thread: db2
> ### Waiting for db2 thread-1 ###
> 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:416)]
> Waiting for call java.util.concurrent.FutureTa$
> k@3b30d168
>
>
> ### Thread 2 started: default ###
> 2017-05-23 13:57:56,979 (pool-150-thread-2) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)]
> begin thread: default
> ### Thread 1 failed ###
> 2017-05-23 13:57:57,007 (pool-150-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)]
> Failed to execute task on 1 attempt$
> . Sleeping for 1000 ms. Exception: org.apache.thrift.TApplicationException:
> get_database failed: out of sequence response
>
> 2017-05-23 13:57:58,011 (pool-150-thread-1) [ERROR -
> org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)]
> Task did not complete successfully $
> fter 1 tries
> ### Thread 1 failed, and throwing an exception from getFullHMSSnapshot() ###
> 2017-05-23 13:57:58,011 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:418)]
> Call ended org.apache.sentry.hdfs.FullUpdateIn$
> tializer$CallResult@7cf642e8
>
>
> 2017-05-23 13:57:58,012 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:421)]
> throwing exception
> ### HMSFollower 2nd run ######
> 2017-05-23 13:57:59,012 (pool-5-thread-1) [DEBUG -
> org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)]
> HMSFollower run is called.
> ### Thread 2 from 1st HMSFollower finished with a failure ####
> 2017-05-23 13:58:14,722 (pool-150-thread-2) [DEBUG -
> org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)]
> Failed to execute task on 1 attempts
> . Sleeping for 1000 ms. Exception:
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Socket closed
> 2017-05-23 13:58:14,725 (pool-150-thread-2) [ERROR -
> org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)]
> Task did not complete successfully a
> fter 2 tries
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
