Sergio Peña created SENTRY-1780:
-----------------------------------
Summary: FullUpdateInitializer does not kill the threads whenever
getFullHMSSnapshot throws an exception
Key: SENTRY-1780
URL: https://issues.apache.org/jira/browse/SENTRY-1780
Project: Sentry
Issue Type: Bug
Affects Versions: sentry-ha-redesign
Reporter: Sergio Peña
Assignee: Sergio Peña
Errors may happen inside the FullUpdateInitializer.getFullHMSSnapshot() causing
to throw an exception. When
this happens, the current threads running are not killed and continue consuming
resources while they're alive.
We should kill those threads. The information collected by them won't be used
anymore.
I noticed this issue on the logs:
{noformat}
### HMSFollower 1st run ######
2017-05-23 13:57:56,976 (pool-5-thread-1) [DEBUG -
org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)]
HMSFollower run is called.
### Starting to get a snapshot (2 databases found (db2, default) and submitted
for execution) ###
2017-05-23 13:57:56,978 (pool-5-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:396)]
get full snapshot
2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)]
submit database: db2
2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)]
submit database: default
### Thread 1 started: db2 ###
2017-05-23 13:57:56,979 (pool-150-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)]
begin thread: db2
### Waiting for db2 thread-1 ###
2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:416)]
Waiting for call java.util.concurrent.FutureTa$
k@3b30d168
### Thread 2 started: default ###
2017-05-23 13:57:56,979 (pool-150-thread-2) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)]
begin thread: default
### Thread 1 failed ###
2017-05-23 13:57:57,007 (pool-150-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)]
Failed to execute task on 1 attempt$
. Sleeping for 1000 ms. Exception: org.apache.thrift.TApplicationException:
get_database failed: out of sequence response
2017-05-23 13:57:58,011 (pool-150-thread-1) [ERROR -
org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)]
Task did not complete successfully $
fter 1 tries
### Thread 1 failed, and throwing an exception from getFullHMSSnapshot() ###
2017-05-23 13:57:58,011 (pool-5-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:418)]
Call ended org.apache.sentry.hdfs.FullUpdateIn$
tializer$CallResult@7cf642e8
2017-05-23 13:57:58,012 (pool-5-thread-1) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:421)]
throwing exception
### HMSFollower 2nd run ######
2017-05-23 13:57:59,012 (pool-5-thread-1) [DEBUG -
org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)]
HMSFollower run is called.
### Thread 2 from 1st HMSFollower finished with a failure ####
2017-05-23 13:58:14,722 (pool-150-thread-2) [DEBUG -
org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)]
Failed to execute task on 1 attempts
. Sleeping for 1000 ms. Exception:
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Socket closed
2017-05-23 13:58:14,725 (pool-150-thread-2) [ERROR -
org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)]
Task did not complete successfully a
fter 2 tries
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
