[
https://issues.apache.org/jira/browse/HIVE-23164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-23164:
--------------------------------
Description:
HiveServer2 which receives the deregister command is at first preparing
shutdown. If there's no remaining session, HiveServer2.stop() is called to shut
down. But I found the case that the HiveServer2 JVM is not terminated even if
HiveServer2.stop() has been called and processed. The case is always occurred
when the local(embedded) metastore is used.
I've attached the full thread dump describing the situation.
[^thread_dump_hiveserver2_is_not_terminated.txt]
In this thread dump, you can see some bunch of 'daemon' threads, NO main
thread, and some 'non-daemon' thread(or user thread)s. As specified by
[https://www.baeldung.com/java-daemon-thread], if there is at least one user
thread exists, JVM does not terminate. (Note that DestroyJavaVM thread is
non-daemon but it's special.)
{code:java}
"pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c
waiting on condition [0x00007f525c500000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003cfa057c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
{code}
The thread above is created by the ScheduledThreadPoolExecutor(int coreSize)
constructor with default ThreadFactory which always makes a thread non-daemon.
If such thread pool is not destroyed by calling
ScheduledThreadPoolExecutor.shutdown() method, JVM cannot terminate! The only
way to kill is TERM signal. If JVM receives TERM signal, it ignores non-daemon
threads and terminates.
So I have been digging modules which create ScheduledThreadPoolExecutor with
non-daemon threads and I got it. As you may guess, it's the local(embedded)
metastore. ScheduledThreadPoolExecutor is created by
org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads()
and ScheduledThreadPoolExecutor.shutdown() is never called.
Plus, I found another usage of creating such ScheduledThreadPoolExecutor and
not calling its shutdown.
was:
HiveServer2 which receives the deregister command is at first preparing
shutdown. If there's no remaining session, HiveServer2.stop() is called to shut
down. But I found the case that the HiveServer2 JVM is not terminated even if
HiveServer2.stop() has been called and processed. The case is always occurred
when the local(embedded) metastore is used.
I've attached the full thread dump describing the situation.
[^thread_dump_hiveserver2_is_not_terminated.txt]
In this thread dump, you can see some bunch of 'daemon' threads, NO main
thread, and some 'non-daemon' thread(or user thread)s. As specified by
[https://www.baeldung.com/java-daemon-thread], if there is at least one user
thread exists, JVM does not terminate. (Note that DestroyJavaVM thread is
non-daemon but it's special.)
{code:java}
"pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c
waiting on condition [0x00007f525c500000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003cfa057c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
{code}
The thread above is created by the ScheduledThreadPoolExecutor(int coreSize)
constructor with default ThreadFactory which always makes a thread non-daemon.
If such thread pool is not destroyed with
ScheduledThreadPoolExecutor.shutdown() method, JVM cannot terminate! The only
way to kill is TERM signal. If JVM receives TERM signal, it ignores non-daemon
threads and terminates.
So I have been digging modules which create ScheduledThreadPoolExecutor with
non-daemon threads and I got it. As you may guess, it's the local(embedded)
metastore. ScheduledThreadPoolExecutor is created by
org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads()
and ScheduledThreadPoolExecutor.shutdown() is never called.
Plus, I found another usage of creating such ScheduledThreadPoolExecutor and
not calling its shutdown.
> server is not properly terminated because of non-daemon threads
> ---------------------------------------------------------------
>
> Key: HIVE-23164
> URL: https://issues.apache.org/jira/browse/HIVE-23164
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Eugene Chung
> Assignee: Eugene Chung
> Priority: Major
> Attachments: HIVE-23164.01.patch, HIVE-23164.02.patch,
> HIVE-23164.03.patch, thread_dump_hiveserver2_is_not_terminated.txt
>
>
> HiveServer2 which receives the deregister command is at first preparing
> shutdown. If there's no remaining session, HiveServer2.stop() is called to
> shut down. But I found the case that the HiveServer2 JVM is not terminated
> even if HiveServer2.stop() has been called and processed. The case is always
> occurred when the local(embedded) metastore is used.
> I've attached the full thread dump describing the situation.
> [^thread_dump_hiveserver2_is_not_terminated.txt]
> In this thread dump, you can see some bunch of 'daemon' threads, NO main
> thread, and some 'non-daemon' thread(or user thread)s. As specified by
> [https://www.baeldung.com/java-daemon-thread], if there is at least one user
> thread exists, JVM does not terminate. (Note that DestroyJavaVM thread is
> non-daemon but it's special.)
>
> {code:java}
> "pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c
> waiting on condition [0x00007f525c500000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000003cfa057c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> {code}
> The thread above is created by the ScheduledThreadPoolExecutor(int coreSize)
> constructor with default ThreadFactory which always makes a thread
> non-daemon. If such thread pool is not destroyed by calling
> ScheduledThreadPoolExecutor.shutdown() method, JVM cannot terminate! The only
> way to kill is TERM signal. If JVM receives TERM signal, it ignores
> non-daemon threads and terminates.
> So I have been digging modules which create ScheduledThreadPoolExecutor with
> non-daemon threads and I got it. As you may guess, it's the local(embedded)
> metastore. ScheduledThreadPoolExecutor is created by
> org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads()
> and ScheduledThreadPoolExecutor.shutdown() is never called.
> Plus, I found another usage of creating such ScheduledThreadPoolExecutor and
> not calling its shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
