[
https://issues.apache.org/jira/browse/SENTRY-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753666#comment-15753666
]
Misha Dmitriev commented on SENTRY-1526:
----------------------------------------
In short, the problem can be fixed by removing 'synchronized' from the
SentryService.waitOnFuture() method. This method doesn't need synchronization,
because nobody calls it concurrently, nor we need calls to it be mutually
exclusive with other synchronized calls in SentryService class. And the fact
that it's synchronized prevents the shutdown mechanism from working properly,
as explained below.
To diagnose the problem, I ran 'jstack' after calling Ctrl+C. It turns out that
many threads still exist. Here are the three relevant threads:
{code}
"Thread-1" #274 prio=5 os_prio=31 tid=0x00007fc2e115b000 nid=0xa703 waiting for
monitor entry [0x0000700003f50000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.sentry.service.thrift.SentryService.stop(SentryService.java:276)
- waiting to lock <0x00000006c1850620> (a
org.apache.sentry.service.thrift.SentryService)
at
org.apache.sentry.service.thrift.SentryService$CommandImpl$1.run(SentryService.java:390)
"SIGINT handler" #311 daemon prio=9 os_prio=31 tid=0x00007fc2e1159000
nid=0xa183 in Object.wait() [0x0000700003b44000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1245)
- locked <0x00000006c160f320> (a
org.apache.sentry.service.thrift.SentryService$CommandImpl$1)
at java.lang.Thread.join(Thread.java:1319)
at
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at
java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000006c04baf88> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:745)
"org.apache.sentry.SentryMain.main()" #11 prio=5 os_prio=31
tid=0x00007f8f5cb4c000 nid=0x5603 waiting on condition [0x00007000015d4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c1b2a740> (a
java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.sentry.service.thrift.SentryService.waitOnFuture(SentryService.java:313)
- locked <0x00000006c1b2a778> (a
org.apache.sentry.service.thrift.SentryService)
at
org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:399)
at org.apache.sentry.SentryMain.main(SentryMain.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:291)
at java.lang.Thread.run(Thread.java:745)
{code}
The thread that's responsible for starting shutdown hook(s) (#311) waits for
the shutdown hook (that in turn runs in a separate thread), to finish. The
shutdown hook code runs in thread #274. That thread is in turn stuck. It cannot
enter synchronized method SentryService.stop(), waiting on lock
0x00000006c160f320, which is associated with the instance of SentryService
class. This lock, it turns out, is being held by the third thread (#11, the
main one), because this thread is inside a chain of calls with the
'synchronized waitOnFuture()' method in the middle. So, until the main thread
exits waitOnFuture(), stop() cannot proceed. In other words, thread A that
wants to stop thread B needs to grab lock X that B will never free until B is
stopped.
If B (the main thread) needed to hold this lock for some good reason, it could
be a non-trivial problem to solve. But fortunately, here it looks like the main
thread doesn't need it for any good reason. Perhaps someone just added that
'synchronized' to waitOnFuture() at some point in the past for perceived
consistency or some similar consideration, and accidentally broke shutdown.
When 'synchronized' is removed from waitOnFuture(), shutdown upon Ctrl+C
immediately works as expected.
> Sentry processed stays alive after being killed
> -----------------------------------------------
>
> Key: SENTRY-1526
> URL: https://issues.apache.org/jira/browse/SENTRY-1526
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
> Affects Versions: 1.8.0, sentry-ha-redesign
> Reporter: Alexander Kolbasov
>
> When a running Sentry daemon receives SIGINT it doesn't terminate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
