chenyuzhi created FLINK-34008:
---------------------------------
Summary: Taskmanager blocked with deserialization when starting
Key: FLINK-34008
URL: https://issues.apache.org/jira/browse/FLINK-34008
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 1.15.2
Reporter: chenyuzhi
When starting taskmanager in kubernetes cluster, some task will block with
deserialization.
However, th jobmanager will not consider the taskmanager heartbeat timeout, so
the entire Job is in a Block state and cannot be checkpointed.
Thread dumb(For taskmanager):
{code:java}
2024-01-06 11:37:53Full thread dump Java HotSpot(TM) 64-Bit Server VM
(25.202-b08 mixed mode):
"flink-akka.actor.default-dispatcher-19" #339 prio=5 os_prio=0
tid=0x00007f9ca84d6000 nid=0x5c8 waiting on condition [0x00007f9c831fd000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7d7d2d0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-akka.actor.default-dispatcher-18" #338 prio=5 os_prio=0
tid=0x00007f9c8c144000 nid=0x5c7 waiting on condition [0x00007f9c8857d000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7d7d2d0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"Attach Listener" #334 daemon prio=9 os_prio=0 tid=0x00007f9cbae06000 nid=0x59e
waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers: - None
"OkHttp WebSocket https://k596.elk.x.netease.com:6443/..."; #323 prio=5
os_prio=0 tid=0x00007f9ca9611800 nid=0x551 waiting on condition
[0x00007f9c837fc000] java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000e7101010> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"flink-taskexecutor-io-thread-3" #316 daemon prio=5 os_prio=0
tid=0x00007f9c88584800 nid=0x53c waiting on condition [0x00007f9cb0ffc000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d8027900> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"flink-taskexecutor-io-thread-2" #312 daemon prio=5 os_prio=0
tid=0x00007f9c9e00a800 nid=0x52d waiting on condition [0x00007f9c9dffc000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d8027900> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"flink-taskexecutor-io-thread-1" #311 daemon prio=5 os_prio=0
tid=0x00007f9c8c0bf800 nid=0x52c waiting on condition [0x00007f9ca95fc000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d8027900> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"prometheus-http-1-5" #123 prio=5 os_prio=0 tid=0x00007f9c9fa1a000 nid=0x14e
waiting on condition [0x00007f9ca377d000] java.lang.Thread.State: WAITING
(parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc44f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"prometheus-http-1-4" #122 prio=5 os_prio=0 tid=0x00007f9c9fa19800 nid=0x14c
waiting on condition [0x00007f9c83dfd000] java.lang.Thread.State: WAITING
(parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc44f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"prometheus-http-1-3" #115 prio=5 os_prio=0 tid=0x00007f9c9fa18800 nid=0x145
waiting on condition [0x00007f9ca6ffd000] java.lang.Thread.State: WAITING
(parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc44f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"prometheus-http-1-2" #114 prio=5 os_prio=0 tid=0x00007f9c9fa18000 nid=0x143
waiting on condition [0x00007f9c9f3fd000] java.lang.Thread.State: WAITING
(parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc44f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"org.apache.hadoop.hdfs.PeerCache@551ed293" #102 daemon prio=5 os_prio=0
tid=0x00007f9ca7023800 nid=0x137 sleeping[0x00007f9c867fd000]
java.lang.Thread.State: TIMED_WAITING (sleeping) at
java.lang.Thread.sleep(Native Method) at
org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:249) at
org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) at
org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"TumblingEventTimeWindows -> Sink: replay-sink-g126 (1/1)#0" #101 prio=5
os_prio=0 tid=0x00007f9c88581800 nid=0x136 in Object.wait()
[0x00007f9c86dfc000] java.lang.Thread.State: BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Sink: split-sink-g126 (1/1)#0" #99 prio=5 os_prio=0 tid=0x00007f9c88580000
nid=0x134 in Object.wait() [0x00007f9c879fc000] java.lang.Thread.State:
BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Config2LogFunction-olschema -> diffOLSchema -> Sink:
other-log-schema-sink-g126 (1/1)#0" #98 prio=5 os_prio=0 tid=0x00007f9c8c0be000
nid=0x133 in Object.wait() [0x00007f9c87ffc000] java.lang.Thread.State:
BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"other-log-schema (1/1)#0" #96 prio=5 os_prio=0 tid=0x00007f9c8c0bb800
nid=0x131 in Object.wait() [0x00007f9c88dfc000] java.lang.Thread.State:
BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"discover-schema-g126 -> schema-to-kafka-record-map -> null-filter -> Sink:
schema-sink-g126 (1/1)#0" #95 prio=5 os_prio=0 tid=0x00007f9c8c0bd000 nid=0x130
in Object.wait() [0x00007f9c893fc000] java.lang.Thread.State: BLOCKED (on
object monitor) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"aggreagate-window-logschema (1/1)#0" #94 prio=5 os_prio=0
tid=0x00007f9c8c0bc800 nid=0x12f in Object.wait() [0x00007f9c899fc000]
java.lang.Thread.State: BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Config2LogFunction -> filter-g126 -> parser-regex -> preprocess-g126 ->
splitter-json -> (null-log-filter -> (Filter, Filter), Sink: error-sink-g126,
log2event-g126 -> (Filter -> Sink: event-sink-g126, Sink: data-sample-g126))
(1/1)#0" #93 prio=5 os_prio=0 tid=0x00007f9c8c0b8000 nid=0x12e in Object.wait()
[0x00007f9c89ffc000] java.lang.Thread.State: BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Source: Custom Source (1/1)#0" #91 prio=5 os_prio=0 tid=0x00007f9c8c0ba000
nid=0x12b in Object.wait() [0x00007f9c8b1fc000] java.lang.Thread.State:
BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Source: Custom Source (1/1)#0" #90 prio=5 os_prio=0 tid=0x00007f9c8c0b9800
nid=0x12a in Object.wait() [0x00007f9c8b7fc000] java.lang.Thread.State:
BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Source: kafka-replay-source (1/1)#0" #88 prio=5 os_prio=0
tid=0x00007f9c8c0b8800 nid=0x129 in Object.wait() [0x00007f9ca75fc000]
java.lang.Thread.State: BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"CloseableReaperThread" #87 daemon prio=5 os_prio=0 tid=0x00007f9ca7022000
nid=0x128 in Object.wait() [0x00007f9c9ebfd000] java.lang.Thread.State:
WAITING (on object monitor) at java.lang.Object.wait(Native Method) -
waiting on <0x00000000d6b23f80> (a java.lang.ref.ReferenceQueue$Lock) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked
<0x00000000d6b23f80> (a java.lang.ref.ReferenceQueue$Lock) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at
org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:215)
Locked ownable synchronizers: - None
"Source: kafka-raw-metrics-source -> tuple-process-headers-value ->
String2AgentHeader -> Timestamps/Watermarks (1/1)#0" #86 prio=5 os_prio=0
tid=0x00007f9c8c0b7000 nid=0x127 in Object.wait() [0x00007f9ca13fc000]
java.lang.Thread.State: BLOCKED (on object monitor) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.lang.Class.getEnumConstantsShared(Class.java:3320) at
java.lang.Class.enumConstantDirectory(Class.java:3341) at
java.lang.Enum.valueOf(Enum.java:232) at
java.io.ObjectInputStream.readEnum(ObjectInputStream.java:2011) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570) at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:617)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:602)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:589)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:627) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"config-map-watch-handler-thread-2" #84 daemon prio=5 os_prio=0
tid=0x00007f9ca9c4f000 nid=0x125 waiting on condition [0x00007f9cb37d2000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7d7ce20> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"pool-9-thread-1" #83 prio=5 os_prio=0 tid=0x00007f9c9e00a000 nid=0x124 waiting
on condition [0x00007f9c8bffc000] java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7d7d068> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"flink-akka.actor.default-dispatcher-16" #82 prio=5 os_prio=0
tid=0x00007f9c9e009000 nid=0x123 waiting on condition [0x00007f9c8c5fd000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7d7d2d0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-akka.actor.default-dispatcher-15" #81 prio=5 os_prio=0
tid=0x00007f9ca6479800 nid=0x122 waiting on condition [0x00007f9c8cbfd000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7d7d2d0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-akka.actor.default-dispatcher-14" #80 prio=5 os_prio=0
tid=0x00007f9ca6479000 nid=0x121 waiting on condition [0x00007f9ca7bfd000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7d7d2d0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"Hashed wheel timer #1" #35 prio=5 os_prio=0 tid=0x00007f9ca4a06000 nid=0x11f
sleeping[0x00007f9c9e5fd000] java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method) at
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
at
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"prometheus-http-1-1" #78 prio=5 os_prio=0 tid=0x00007f9c9fa17000 nid=0x11d
waiting on condition [0x00007f9c8d3fd000] java.lang.Thread.State: WAITING
(parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc44f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Flink Netty Server (0) Thread 0" #73 daemon prio=5 os_prio=0
tid=0x00007f9cac612000 nid=0x11b runnable [0x00007f9c8db6b000]
java.lang.Thread.State: RUNNABLE at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.Native.epollWait(Native
Method) at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.Native.epollWait(Native.java:193)
at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.Native.epollWait(Native.java:186)
at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Flink-MetricRegistry-thread-1" #72 daemon prio=5 os_prio=0
tid=0x00007f9cac686000 nid=0x11a waiting on condition [0x00007f9c8e5fd000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc49e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"IOManager reader thread #1" #69 daemon prio=5 os_prio=0 tid=0x00007f9cac619800
nid=0x119 waiting on condition [0x00007f9c99dfc000] java.lang.Thread.State:
WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc4c50> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$ReaderThread.run(IOManagerAsync.java:372)
Locked ownable synchronizers: - None
"IOManager writer thread #1" #68 daemon prio=5 os_prio=0 tid=0x00007f9cac618800
nid=0x118 waiting on condition [0x00007f9c9a3fc000] java.lang.Thread.State:
WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc4e58> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:482)
Locked ownable synchronizers: - None
"Timer-1" #66 daemon prio=5 os_prio=0 tid=0x00007f9cac618000 nid=0x117 in
Object.wait() [0x00007f9c9a9fd000] java.lang.Thread.State: TIMED_WAITING (on
object monitor) at java.lang.Object.wait(Native Method) at
java.util.TimerThread.mainLoop(Timer.java:552) - locked
<0x00000000d7dc5060> (a java.util.TaskQueue) at
java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers: - None
"Timer-0" #64 daemon prio=5 os_prio=0 tid=0x00007f9cac617000 nid=0x116 in
Object.wait() [0x00007f9c9affd000] java.lang.Thread.State: TIMED_WAITING (on
object monitor) at java.lang.Object.wait(Native Method) at
java.util.TimerThread.mainLoop(Timer.java:552) - locked
<0x00000000d7dc5238> (a java.util.TaskQueue) at
java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers: - None
"New I/O server boss #8" #60 prio=5 os_prio=0 tid=0x00007f9c9ce34800 nid=0x113
runnable [0x00007f9c9bbfc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc5710> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc5720> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc56c8> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc5818> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"New I/O worker #7" #59 prio=5 os_prio=0 tid=0x00007f9c9ce33800 nid=0x112
runnable [0x00007f9c9c1fc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc5a60> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc5a50> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc5a08> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc5b60> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"New I/O boss #6" #58 prio=5 os_prio=0 tid=0x00007f9c9ce33000 nid=0x111
runnable [0x00007f9c9c7fc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc5da8> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc5d98> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc5d50> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc5ea8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"New I/O worker #5" #56 prio=5 os_prio=0 tid=0x00007f9c9ce32000 nid=0x110
runnable [0x00007f9c9cdfc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc60f0> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc60e0> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc6098> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc61f0> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"flink-metrics-akka.remote.default-remote-dispatcher-7" #55 prio=5 os_prio=0
tid=0x00007f9c9e007800 nid=0x10f waiting on condition [0x00007f9c9d3fd000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc63a0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-metrics-akka.remote.default-remote-dispatcher-6" #54 prio=5 os_prio=0
tid=0x00007f9c9e007000 nid=0x10e waiting on condition [0x00007f9c9d9fd000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7dc63a0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-metrics-scheduler-1" #49 prio=5 os_prio=0 tid=0x00007f9cac614000
nid=0x109 sleeping[0x00007f9c9f9fd000] java.lang.Thread.State: TIMED_WAITING
(sleeping) at java.lang.Thread.sleep(Native Method) at
akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:90)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:300)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Thread-7" #48 prio=5 os_prio=0 tid=0x00007f9cac613800 nid=0x108 runnable
[0x00007f9c9fffd000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc7300> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc72f0> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc72a8> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:352) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"server-timer" #47 daemon prio=5 os_prio=0 tid=0x00007f9cac612800 nid=0x107 in
Object.wait() [0x00007f9ca05fd000] java.lang.Thread.State: TIMED_WAITING (on
object monitor) at java.lang.Object.wait(Native Method) at
java.util.TimerThread.mainLoop(Timer.java:552) - locked
<0x00000000d7dc74a8> (a java.util.TaskQueue) at
java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers: - None
"RMI Scheduler(0)" #44 daemon prio=5 os_prio=0 tid=0x00007f9cac2bb800 nid=0x104
waiting on condition [0x00007f9ca19fc000] java.lang.Thread.State: WAITING
(parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc7910> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"GC Daemon" #42 daemon prio=2 os_prio=0 tid=0x00007f9cac611000 nid=0x102 in
Object.wait() [0x00007f9ca25fd000] java.lang.Thread.State: TIMED_WAITING (on
object monitor) at java.lang.Object.wait(Native Method) at
sun.misc.GC$Daemon.run(GC.java:117) - locked <0x00000000d7dc7d28> (a
sun.misc.GC$LatencyLock)
Locked ownable synchronizers: - None
"RMI Reaper" #41 prio=5 os_prio=0 tid=0x00007f9cac2bc800 nid=0x101 in
Object.wait() [0x00007f9ca2bfd000] java.lang.Thread.State: WAITING (on object
monitor) at java.lang.Object.wait(Native Method) - waiting on
<0x00000000d7dc7eb0> (a java.lang.ref.ReferenceQueue$Lock) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked
<0x00000000d7dc7eb0> (a java.lang.ref.ReferenceQueue$Lock) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at
sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"RMI TCP Accept-8800" #40 daemon prio=5 os_prio=0 tid=0x00007f9cb7972000
nid=0x100 runnable [0x00007f9ca31fc000] java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method) at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at
java.net.ServerSocket.implAccept(ServerSocket.java:545) at
java.net.ServerSocket.accept(ServerSocket.java:513) at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377) at
java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"New I/O server boss #4" #38 prio=5 os_prio=0 tid=0x00007f9ca560e000 nid=0xfe
runnable [0x00007f9ca43fc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc85b8> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc85c8> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc8570> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc86c0> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"New I/O worker #3" #37 prio=5 os_prio=0 tid=0x00007f9ca560d800 nid=0xfd
runnable [0x00007f9ca49fc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc8908> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc88f8> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc88b0> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc8a08> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"New I/O boss #2" #36 prio=5 os_prio=0 tid=0x00007f9ca560c800 nid=0xfc runnable
[0x00007f9ca4ffc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc8c50> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc8c40> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc8bf8> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc8d50> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"New I/O worker #1" #34 prio=5 os_prio=0 tid=0x00007f9ca560c000 nid=0xfb
runnable [0x00007f9ca55fc000] java.lang.Thread.State: RUNNABLE at
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked
<0x00000000d7dc8f98> (a sun.nio.ch.Util$3) - locked <0x00000000d7dc8f88> (a
java.util.Collections$UnmodifiableSet) - locked <0x00000000d7dc8f40> (a
sun.nio.ch.EPollSelectorImpl) at
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dc9098> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"flink-akka.remote.default-remote-dispatcher-7" #33 prio=5 os_prio=0
tid=0x00007f9ca76ba000 nid=0xfa waiting on condition [0x00007f9ca5bfd000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7dc9248> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-akka.remote.default-remote-dispatcher-6" #32 prio=5 os_prio=0
tid=0x00007f9ca76b9800 nid=0xf9 waiting on condition [0x00007f9ca63fd000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7dc9248> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-akka.actor.default-dispatcher-5" #31 prio=5 os_prio=0
tid=0x00007f9ca76b8800 nid=0xf8 waiting on condition [0x00007f9ca69fd000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7d7d2d0> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers: - None
"flink-scheduler-1" #27 prio=5 os_prio=0 tid=0x00007f9cac2f6800 nid=0xf4
sleeping[0x00007f9ca89fd000] java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method) at
akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:90)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:300)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"config-map-watch-handler-thread-1" #25 daemon prio=5 os_prio=0
tid=0x00007f9ca9c4e000 nid=0xf2 waiting on condition [0x00007f9ca8ffc000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7d7ce20> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"OkHttp https://k596.elk.x.netease.com:6443/..."; #23 prio=5 os_prio=0
tid=0x00007f9caa215000 nid=0xf0 runnable [0x00007f9ca9bfc000]
java.lang.Thread.State: RUNNABLE at
java.net.SocketInputStream.socketRead0(Native Method) at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at
java.net.SocketInputStream.read(SocketInputStream.java:171) at
java.net.SocketInputStream.read(SocketInputStream.java:141) at
sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at
sun.security.ssl.InputRecord.read(InputRecord.java:503) at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) - locked
<0x00000000e7105a50> (a java.lang.Object) at
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) at
sun.security.ssl.AppInputStream.read(AppInputStream.java:105) - locked
<0x00000000e7105b48> (a sun.security.ssl.AppInputStream) at
org.apache.flink.kubernetes.shaded.okio.Okio$2.read(Okio.java:140) at
org.apache.flink.kubernetes.shaded.okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at
org.apache.flink.kubernetes.shaded.okio.RealBufferedSource.request(RealBufferedSource.java:72)
at
org.apache.flink.kubernetes.shaded.okio.RealBufferedSource.require(RealBufferedSource.java:65)
at
org.apache.flink.kubernetes.shaded.okio.RealBufferedSource.readByte(RealBufferedSource.java:78)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:273)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:209)
at
org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7dcebc8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"KubernetesClient-Informer-thread-1" #22 daemon prio=5 os_prio=0
tid=0x00007f9cb8f18800 nid=0xef waiting on condition [0x00007f9caa1fd000]
java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native
Method) - parking to wait for <0x00000000d7dced70> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Okio Watchdog" #21 daemon prio=5 os_prio=0 tid=0x00007f9cbc71f800 nid=0xee in
Object.wait() [0x00007f9caeb7c000] java.lang.Thread.State: TIMED_WAITING (on
object monitor) at java.lang.Object.wait(Native Method) at
org.apache.flink.kubernetes.shaded.okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:348)
at
org.apache.flink.kubernetes.shaded.okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:313)
- locked <0x00000000d7d4d718> (a java.lang.Class for
org.apache.flink.kubernetes.shaded.okio.AsyncTimeout)
Locked ownable synchronizers: - None
"OkHttp ConnectionPool" #20 daemon prio=5 os_prio=0 tid=0x00007f9cbc71f000
nid=0xed in Object.wait() [0x00007f9caec7c000] java.lang.Thread.State:
TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at
java.lang.Object.wait(Object.java:460) at
org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnectionPool.lambda$new$0(RealConnectionPool.java:62)
- locked <0x00000000d7409b78> (a
org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnectionPool)
at
org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnectionPool$$Lambda$112/269892014.run(Unknown
Source) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - <0x00000000d7e19db0> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #17
daemon prio=5 os_prio=0 tid=0x00007f9cb8f16000 nid=0xea in Object.wait()
[0x00007f9cb87c3000] java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method) - waiting on <0x00000000d72b06e0> (a
java.lang.ref.ReferenceQueue$Lock) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked
<0x00000000d72b06e0> (a java.lang.ref.ReferenceQueue$Lock) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:2989)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: - None
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f9cbc6be000 nid=0xe3
runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers: - None
"C1 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f9cbc6bd000 nid=0xe2
waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers: - None
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f9cd345d800 nid=0xe1
waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers: - None
"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f9cd345c800 nid=0xe0
runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers: - None
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007f9cd3447000 nid=0xdf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers: - None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9cd345c000 nid=0xde in
Object.wait() [0x00007f9cbc1fd000] java.lang.Thread.State: WAITING (on object
monitor) at java.lang.Object.wait(Native Method) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked
<0x00000000d6869340> (a java.lang.ref.ReferenceQueue$Lock) at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
Locked ownable synchronizers: - None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9cd345b000 nid=0xdd
in Object.wait() [0x00007f9cbcf23000] java.lang.Thread.State: WAITING (on
object monitor) at java.lang.Object.wait(Native Method) at
java.lang.Object.wait(Object.java:502) at
java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked
<0x00000000d68694f8> (a java.lang.ref.Reference$Lock) at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers: - None
"main" #1 prio=5 os_prio=0 tid=0x00007f9cd3458000 nid=0xd6 waiting on condition
[0x00007f9cd5192000] java.lang.Thread.State: WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00000000d7e1a008> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:487)
at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerProcessSecurely$5(TaskManagerRunner.java:525)
at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner$$Lambda$87/1088104996.call(Unknown
Source) at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext$$Lambda$88/421637524.run(Unknown
Source) at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:525)
at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:505)
at
org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner.main(KubernetesTaskExecutorRunner.java:39)
Locked ownable synchronizers: - None
"VM Thread" os_prio=0 tid=0x00007f9cd34cf800 nid=0xdc runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f9cd34cd000 nid=0xd7
runnable
"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f9cd34ce000 nid=0xda
runnable
"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f9cd34ce800
nid=0xdb runnable
"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f9cd3459800 nid=0xd9
runnable
"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f9cd3459000 nid=0xd8
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f9cd34d0800 nid=0xe4 waiting on
condition
JNI global references: 1928
{code}
Environment:
Flink: 1.15.2
Flink-kubernetes-operator: 1.3.1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
