[
https://issues.apache.org/jira/browse/FLINK-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195735#comment-17195735
]
Stephan Ewen commented on FLINK-18128:
--------------------------------------
The problem is the following:
- When the fetching thread gets the fetch that signals "end of split" it
enqueues that into the handover queue
- Then there may be a context switch to the reader thread that checks this
fetch and finishes its own split handling and checks whether the split fetcher
is idle. It it not at that point.
- The result is that the reader eventually goes to MORE_AVAILABLE_LATER
- The split fetcher thread continues and marks the split fetcher as idle.
However, it will never be checked again, because the reader thread is never
notified again of any availability so it never checks.
*Solution part 1:*
The split fetcher needs to notify the availability future when it turns idle. I
think there is no way to guarantee that the fetcher gets checked for being idle
otherwise, save for introducing locking around the enqueuing and the idleness
check. That is more expensive, the notification is rare and cheap.
*Solution part 2:*
The current model how the availability future is implemented makes this still
dead-lock dangerous: The reader thread (mailbox thread) creates the
availability future when obtaining it. It may be that the notification comes
before the thread obtains the future and the notification is lost.
Consider this scenario:
- Fetcher Thread gets "end of split" it enqueues that into the handover queue
- Reader Thread obtains this fetch and removes the split locally. Split
Reader is not idle, yet, Reader Thread concludes MORE_AVAILABLE_LATER
- Fetcher Thread notifies the FutureNotifier - no future has been requested
so far, the notification does nothing.
- Reader Thread gets the future from the FutureNotifier, which will now never
be complete.
We could fix this by adding a dummy element into the
{{FutureCompletingBlockingQueue}}, but I think that FLINK-19223 is a better
solution when looking at the bigger picture.
> CoordinatedSourceITCase.testMultipleSources gets stuck
> ------------------------------------------------------
>
> Key: FLINK-18128
> URL: https://issues.apache.org/jira/browse/FLINK-18128
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task, Tests
> Affects Versions: 1.11.0
> Reporter: Robert Metzger
> Assignee: Stephan Ewen
> Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2705&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=6b04ca5f-0b52-511d-19c9-52bf0d9fbdfa
> {code}
> 2020-06-04T11:19:39.6335702Z [INFO]
> 2020-06-04T11:19:39.6337440Z [INFO]
> -------------------------------------------------------
> 2020-06-04T11:19:39.6338176Z [INFO] T E S T S
> 2020-06-04T11:19:39.6339305Z [INFO]
> -------------------------------------------------------
> 2020-06-04T11:19:40.1906157Z [INFO] Running
> org.apache.flink.connector.base.source.reader.CoordinatedSourceITCase
> 2020-06-04T11:34:51.0599860Z
> ==============================================================================
> 2020-06-04T11:34:51.0603015Z Maven produced no output for 900 seconds.
> 2020-06-04T11:34:51.0604174Z
> ==============================================================================
> 2020-06-04T11:34:51.0613908Z
> ==============================================================================
> 2020-06-04T11:34:51.0615097Z The following Java processes are running (JPS)
> 2020-06-04T11:34:51.0616043Z
> ==============================================================================
> 2020-06-04T11:34:51.0762007Z Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> 2020-06-04T11:34:51.2775341Z 29635 surefirebooter5307550588991461882.jar
> 2020-06-04T11:34:51.2931264Z 2100 Launcher
> 2020-06-04T11:34:51.3012583Z 32203 Jps
> 2020-06-04T11:34:51.3258038Z Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> 2020-06-04T11:34:51.5443730Z
> ==============================================================================
> 2020-06-04T11:34:51.5445134Z Printing stack trace of Java process 29635
> 2020-06-04T11:34:51.5445984Z
> ==============================================================================
> 2020-06-04T11:34:51.5528602Z Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> 2020-06-04T11:34:51.9617670Z 2020-06-04 11:34:51
> 2020-06-04T11:34:51.9619131Z Full thread dump OpenJDK 64-Bit Server VM
> (25.242-b08 mixed mode):
> 2020-06-04T11:34:51.9619732Z
> 2020-06-04T11:34:51.9620618Z "Attach Listener" #299 daemon prio=9 os_prio=0
> tid=0x00007f4d60001000 nid=0x7e59 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:51.9621720Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:51.9622190Z
> 2020-06-04T11:34:51.9623631Z "flink-akka.actor.default-dispatcher-185" #297
> prio=5 os_prio=0 tid=0x00007f4ca0003000 nid=0x7db4 waiting on condition
> [0x00007f4d10136000]
> 2020-06-04T11:34:51.9624972Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9625716Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9627072Z - parking to wait for <0x0000000080c557f0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 2020-06-04T11:34:51.9628593Z at
> akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 2020-06-04T11:34:51.9629649Z at
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 2020-06-04T11:34:51.9630825Z at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2020-06-04T11:34:51.9631559Z
> 2020-06-04T11:34:51.9633020Z "flink-akka.actor.default-dispatcher-186" #298
> prio=5 os_prio=0 tid=0x00007f4d08006800 nid=0x7db3 waiting on condition
> [0x00007f4d12974000]
> 2020-06-04T11:34:51.9634074Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9634965Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9636384Z - parking to wait for <0x0000000080c557f0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 2020-06-04T11:34:51.9637683Z at
> akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 2020-06-04T11:34:51.9638795Z at
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 2020-06-04T11:34:51.9639845Z at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2020-06-04T11:34:51.9640475Z
> 2020-06-04T11:34:51.9642008Z "flink-akka.actor.default-dispatcher-182" #293
> prio=5 os_prio=0 tid=0x00007f4ce4007000 nid=0x7d7f waiting on condition
> [0x00007f4d1063b000]
> 2020-06-04T11:34:51.9643266Z java.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-06-04T11:34:51.9644081Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9645784Z - parking to wait for <0x0000000080c557f0> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> 2020-06-04T11:34:51.9647361Z at
> akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> 2020-06-04T11:34:51.9648516Z at
> akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> 2020-06-04T11:34:51.9649692Z at
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 2020-06-04T11:34:51.9651016Z at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2020-06-04T11:34:51.9651739Z
> 2020-06-04T11:34:51.9653311Z "jobmanager-future-thread-16" #236 daemon prio=5
> os_prio=0 tid=0x00007f4c94002000 nid=0x7a99 waiting on condition
> [0x00007f4d12873000]
> 2020-06-04T11:34:51.9654578Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9655376Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9656968Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9658360Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9659770Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9661421Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9663047Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9664605Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9665917Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9667296Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9668392Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9668914Z
> 2020-06-04T11:34:51.9670561Z "Flink-DispatcherRestEndpoint-thread-4" #119
> daemon prio=5 os_prio=0 tid=0x00007f4cc0004800 nid=0x749b waiting on
> condition [0x00007f4d1328f000]
> 2020-06-04T11:34:51.9671805Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9672620Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9674216Z - parking to wait for <0x0000000087803238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9675411Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9676984Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9678652Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9680193Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9681672Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9682904Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9684149Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9685116Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9685563Z
> 2020-06-04T11:34:51.9687028Z "Flink-DispatcherRestEndpoint-thread-3" #118
> daemon prio=5 os_prio=0 tid=0x00007f4cc0003800 nid=0x7490 waiting on
> condition [0x00007f4c825e4000]
> 2020-06-04T11:34:51.9688144Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9688792Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9690067Z - parking to wait for <0x0000000087803238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9691152Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9692335Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9693745Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9695311Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9696509Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9697640Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9698724Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9699618Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9700045Z
> 2020-06-04T11:34:51.9701396Z "Flink-DispatcherRestEndpoint-thread-2" #115
> daemon prio=5 os_prio=0 tid=0x00007f4cc0002800 nid=0x7478 waiting on
> condition [0x00007f4d12f78000]
> 2020-06-04T11:34:51.9702417Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9703084Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9704565Z - parking to wait for <0x0000000087803238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9705653Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9706804Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9708230Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9709574Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9710779Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9711890Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9713162Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9714226Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9714854Z
> 2020-06-04T11:34:51.9715760Z "SourceFetcher" #112 prio=5 os_prio=0
> tid=0x00007f4c10006800 nid=0x746b waiting on condition [0x00007f4c812d3000]
> 2020-06-04T11:34:51.9716852Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9717754Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9719392Z - parking to wait for <0x000000008777dc80> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9726297Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9727728Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9729254Z at
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
> 2020-06-04T11:34:51.9730516Z at
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
> 2020-06-04T11:34:51.9731853Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:113)
> 2020-06-04T11:34:51.9733084Z at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:94)
> 2020-06-04T11:34:51.9734253Z at
> org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:42)
> 2020-06-04T11:34:51.9735413Z at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2020-06-04T11:34:51.9736573Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-06-04T11:34:51.9737863Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-06-04T11:34:51.9739214Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9740349Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9740883Z
> 2020-06-04T11:34:51.9743025Z "SourceCoordinator-Source: TestingSource2" #110
> prio=5 os_prio=0 tid=0x00007f4cf406a000 nid=0x7469 waiting on condition
> [0x00007f4c814d5000]
> 2020-06-04T11:34:51.9744816Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9745675Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9747526Z - parking to wait for <0x0000000087a7aff8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9749566Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9751072Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9752591Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:51.9753926Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9755414Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9756776Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9757999Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9758523Z
> 2020-06-04T11:34:51.9759948Z "SourceCoordinator-Source: TestingSource1" #109
> prio=5 os_prio=0 tid=0x00007f4cf4068800 nid=0x7468 waiting on condition
> [0x00007f4c815d6000]
> 2020-06-04T11:34:51.9760998Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9761673Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9762927Z - parking to wait for <0x0000000087200178> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9764012Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9765377Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9766639Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:51.9767882Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9769195Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9770567Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9771666Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9772235Z
> 2020-06-04T11:34:51.9773191Z "Sink: Unnamed (2/4)" #105 prio=5 os_prio=0
> tid=0x00007f4cf4060800 nid=0x7464 waiting on condition [0x00007f4c81bda000]
> 2020-06-04T11:34:51.9774776Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9775591Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9777459Z - parking to wait for <0x0000000087a7b278> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9778794Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9780275Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9782114Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:146)
> 2020-06-04T11:34:51.9783693Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:279)
> 2020-06-04T11:34:51.9785319Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:190)
> 2020-06-04T11:34:51.9786679Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)
> 2020-06-04T11:34:51.9788033Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:557)
> 2020-06-04T11:34:51.9789165Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:529)
> 2020-06-04T11:34:51.9790155Z at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:720)
> 2020-06-04T11:34:51.9791088Z at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:545)
> 2020-06-04T11:34:51.9791879Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9792357Z
> 2020-06-04T11:34:51.9793428Z "OutputFlusher for Source: TestingSource2" #98
> daemon prio=5 os_prio=0 tid=0x00007f4c10001800 nid=0x745d waiting on
> condition [0x00007f4c822e1000]
> 2020-06-04T11:34:51.9794635Z java.lang.Thread.State: TIMED_WAITING
> (sleeping)
> 2020-06-04T11:34:51.9795319Z at java.lang.Thread.sleep(Native Method)
> 2020-06-04T11:34:51.9796321Z at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter$OutputFlusher.run(RecordWriter.java:334)
> 2020-06-04T11:34:51.9797036Z
> 2020-06-04T11:34:51.9797921Z "Source: TestingSource2 (2/4)" #94 prio=5
> os_prio=0 tid=0x00007f4cf4055800 nid=0x7459 waiting on condition
> [0x00007f4c826e5000]
> 2020-06-04T11:34:51.9798908Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9799555Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9800954Z - parking to wait for <0x0000000087500178> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9802006Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9803187Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9804559Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take(TaskMailboxImpl.java:146)
> 2020-06-04T11:34:51.9805858Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:279)
> 2020-06-04T11:34:51.9807228Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:190)
> 2020-06-04T11:34:51.9808529Z at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)
> 2020-06-04T11:34:51.9809766Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:557)
> 2020-06-04T11:34:51.9810856Z at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:529)
> 2020-06-04T11:34:51.9811862Z at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:720)
> 2020-06-04T11:34:51.9812794Z at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:545)
> 2020-06-04T11:34:51.9813580Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9814025Z
> 2020-06-04T11:34:51.9814876Z "CloseableReaperThread" #89 daemon prio=5
> os_prio=0 tid=0x00007f4c28004800 nid=0x7454 in Object.wait()
> [0x00007f4c82bea000]
> 2020-06-04T11:34:51.9815918Z java.lang.Thread.State: WAITING (on object
> monitor)
> 2020-06-04T11:34:51.9816734Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:51.9817885Z - waiting on <0x00000000878034a0> (a
> java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:51.9818759Z at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-06-04T11:34:51.9819928Z - locked <0x00000000878034a0> (a
> java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:51.9820808Z at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> 2020-06-04T11:34:51.9821937Z at
> org.apache.flink.core.fs.SafetyNetCloseableRegistry$CloseableReaperThread.run(SafetyNetCloseableRegistry.java:204)
> 2020-06-04T11:34:51.9822706Z
> 2020-06-04T11:34:51.9823940Z "Flink-MetricRegistry-thread-1" #87 daemon
> prio=5 os_prio=0 tid=0x00007f4cf403a000 nid=0x7452 waiting on condition
> [0x00007f4c82dec000]
> 2020-06-04T11:34:51.9825063Z java.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-06-04T11:34:51.9825768Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9827061Z - parking to wait for <0x0000000080c46498> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9828195Z at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:51.9829405Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:51.9830790Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:51.9832105Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9833408Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9834537Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9835667Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9836568Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9837003Z
> 2020-06-04T11:34:51.9838290Z "jobmanager-future-thread-15" #86 daemon prio=5
> os_prio=0 tid=0x00007f4d000e4800 nid=0x7451 waiting on condition
> [0x00007f4c82eed000]
> 2020-06-04T11:34:51.9839293Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9839970Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9841223Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9842294Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9843451Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9844893Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:51.9846266Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9847511Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9848582Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9849641Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9850531Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9850957Z
> 2020-06-04T11:34:51.9852196Z "jobmanager-future-thread-14" #85 daemon prio=5
> os_prio=0 tid=0x00007f4d000e2800 nid=0x7450 waiting on condition
> [0x00007f4c82fee000]
> 2020-06-04T11:34:51.9853177Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9853845Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9855174Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9856287Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9857618Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9858956Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9860293Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9861465Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9862544Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9863626Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9864587Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9865040Z
> 2020-06-04T11:34:51.9866307Z "jobmanager-future-thread-13" #84 daemon prio=5
> os_prio=0 tid=0x00007f4d000e1000 nid=0x744f waiting on condition
> [0x00007f4c830ef000]
> 2020-06-04T11:34:51.9867374Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9868019Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9869286Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9870338Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9871506Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9872859Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:51.9874275Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9875582Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9876638Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9877780Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9878683Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9879109Z
> 2020-06-04T11:34:51.9880327Z "jobmanager-future-thread-12" #83 daemon prio=5
> os_prio=0 tid=0x00007f4d000df800 nid=0x744e waiting on condition
> [0x00007f4c831f0000]
> 2020-06-04T11:34:51.9881409Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9882126Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9883427Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9884608Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9885796Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9887213Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:51.9888559Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9889735Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9890801Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9891859Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9892754Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9893187Z
> 2020-06-04T11:34:51.9894424Z "jobmanager-future-thread-11" #82 daemon prio=5
> os_prio=0 tid=0x00007f4d000dd800 nid=0x744d waiting on condition
> [0x00007f4c832f1000]
> 2020-06-04T11:34:51.9895512Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9896193Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9897529Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9898711Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9899878Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9901271Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9902744Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9904043Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9905324Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9906518Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9907468Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9907922Z
> 2020-06-04T11:34:51.9909177Z "jobmanager-future-thread-10" #81 daemon prio=5
> os_prio=0 tid=0x00007f4d000db800 nid=0x744c waiting on condition
> [0x00007f4c833f2000]
> 2020-06-04T11:34:51.9910178Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9910825Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9912091Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9913164Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9914314Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9915996Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9917386Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9918590Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9919657Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9920718Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9921606Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9922033Z
> 2020-06-04T11:34:51.9923282Z "jobmanager-future-thread-9" #80 daemon prio=5
> os_prio=0 tid=0x00007f4d000da000 nid=0x744b waiting on condition
> [0x00007f4c834f3000]
> 2020-06-04T11:34:51.9924257Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9925107Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9926445Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9927613Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9928764Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9930131Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9931469Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9932647Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9933721Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9934914Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9935852Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9936281Z
> 2020-06-04T11:34:51.9937591Z "jobmanager-future-thread-8" #79 daemon prio=5
> os_prio=0 tid=0x00007f4d000d8000 nid=0x744a waiting on condition
> [0x00007f4c835f4000]
> 2020-06-04T11:34:51.9938593Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9939359Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9940645Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9941698Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9942880Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9944220Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9945780Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9946985Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9948097Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9949185Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9950060Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9950506Z
> 2020-06-04T11:34:51.9951732Z "jobmanager-future-thread-7" #78 daemon prio=5
> os_prio=0 tid=0x00007f4d000d6800 nid=0x7449 waiting on condition
> [0x00007f4c836f5000]
> 2020-06-04T11:34:51.9952738Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9953383Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9954784Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9956012Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9957224Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9958589Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9959909Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9961100Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9962169Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9963229Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9964122Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9964684Z
> 2020-06-04T11:34:51.9965965Z "jobmanager-future-thread-6" #77 daemon prio=5
> os_prio=0 tid=0x00007f4d000d5000 nid=0x7448 waiting on condition
> [0x00007f4c837f6000]
> 2020-06-04T11:34:51.9966947Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9967675Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9968927Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9970017Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9971191Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:51.9972532Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:51.9973873Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9975208Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9976321Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9977472Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9978347Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9978776Z
> 2020-06-04T11:34:51.9980010Z "jobmanager-future-thread-5" #76 daemon prio=5
> os_prio=0 tid=0x00007f4d000d3800 nid=0x7447 waiting on condition
> [0x00007f4c838f7000]
> 2020-06-04T11:34:51.9981144Z java.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-06-04T11:34:51.9981812Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9983091Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9984161Z at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:51.9985490Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:51.9986902Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:51.9988288Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:51.9989486Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:51.9990539Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:51.9991623Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:51.9992519Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:51.9992945Z
> 2020-06-04T11:34:51.9994158Z "jobmanager-future-thread-4" #75 daemon prio=5
> os_prio=0 tid=0x00007f4d000d1000 nid=0x7446 waiting on condition
> [0x00007f4c839f8000]
> 2020-06-04T11:34:51.9995240Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:51.9996082Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:51.9997404Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:51.9998474Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:51.9999624Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0000989Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0002324Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0003501Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0004656Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0005750Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0006656Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0007083Z
> 2020-06-04T11:34:52.0008361Z "mini-cluster-io-thread-14" #74 daemon prio=5
> os_prio=0 tid=0x00007f4cf402b000 nid=0x7445 waiting on condition
> [0x00007f4c83af9000]
> 2020-06-04T11:34:52.0009334Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0010002Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0011256Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0012329Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0013501Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0014777Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0015971Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0017254Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0018345Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0019239Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0019668Z
> 2020-06-04T11:34:52.0020806Z "pool-2-thread-1" #73 prio=5 os_prio=0
> tid=0x00007f4cf4028800 nid=0x7444 waiting on condition [0x00007f4c83bfa000]
> 2020-06-04T11:34:52.0021869Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0022542Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0023806Z - parking to wait for <0x000000008797ae48> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0024972Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0026160Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0027597Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:52.0028931Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0030107Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0031193Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0032255Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0033149Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0033574Z
> 2020-06-04T11:34:52.0034882Z "jobmanager-future-thread-3" #72 daemon prio=5
> os_prio=0 tid=0x00007f4cf4021000 nid=0x7441 waiting on condition
> [0x00007f4c83cfb000]
> 2020-06-04T11:34:52.0035891Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0036563Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0038036Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0039090Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0040263Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0041614Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0042956Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0044161Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0045427Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0046545Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0047480Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0047941Z
> 2020-06-04T11:34:52.0049160Z "mini-cluster-io-thread-13" #71 daemon prio=5
> os_prio=0 tid=0x00007f4d000c2000 nid=0x7440 waiting on condition
> [0x00007f4c83dfc000]
> 2020-06-04T11:34:52.0050151Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0050796Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0052074Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0053149Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0054301Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0055639Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0056678Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0057821Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0058882Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0059783Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0060209Z
> 2020-06-04T11:34:52.0061450Z "jobmanager-future-thread-2" #70 daemon prio=5
> os_prio=0 tid=0x00007f4d0802a800 nid=0x743f waiting on condition
> [0x00007f4c83efd000]
> 2020-06-04T11:34:52.0062567Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0063214Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0064577Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0065663Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0066842Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0068250Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0069584Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0070782Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0071836Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0072916Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0073789Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0074242Z
> 2020-06-04T11:34:52.0075755Z "FlinkCompletableFutureDelayScheduler-thread-1"
> #69 daemon prio=5 os_prio=0 tid=0x00007f4d08021000 nid=0x743e waiting on
> condition [0x00007f4c83ffe000]
> 2020-06-04T11:34:52.0076828Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0077536Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0078926Z - parking to wait for <0x000000008777e698> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0079998Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0081149Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0082514Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> 2020-06-04T11:34:52.0083830Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0085239Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0086384Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0087523Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0088428Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0088855Z
> 2020-06-04T11:34:52.0090097Z "mini-cluster-io-thread-12" #68 daemon prio=5
> os_prio=0 tid=0x00007f4d0800d000 nid=0x743d waiting on condition
> [0x00007f4d115da000]
> 2020-06-04T11:34:52.0091065Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0091736Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0092994Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0094069Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0095322Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0096546Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0097702Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0098758Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0099837Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0100733Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0101161Z
> 2020-06-04T11:34:52.0102374Z "mini-cluster-io-thread-11" #67 daemon prio=5
> os_prio=0 tid=0x00007f4d000b4800 nid=0x743c waiting on condition
> [0x00007f4d1196c000]
> 2020-06-04T11:34:52.0103496Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0104166Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0105557Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0106636Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0107863Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0109078Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0110111Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0111187Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0112263Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0113141Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0113590Z
> 2020-06-04T11:34:52.0114888Z "mini-cluster-io-thread-10" #64 daemon prio=5
> os_prio=0 tid=0x00007f4cb000b000 nid=0x7439 waiting on condition
> [0x00007f4d10338000]
> 2020-06-04T11:34:52.0115908Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0116557Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0117907Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0125989Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0127262Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0128486Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0129522Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0130608Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0131667Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0132564Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0132994Z
> 2020-06-04T11:34:52.0134343Z "mini-cluster-io-thread-9" #63 daemon prio=5
> os_prio=0 tid=0x00007f4cb0006800 nid=0x7438 waiting on condition
> [0x00007f4d10439000]
> 2020-06-04T11:34:52.0135607Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0136372Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0137749Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0138798Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0139975Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0141174Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0142229Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0143301Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0144361Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0145444Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0145909Z
> 2020-06-04T11:34:52.0147243Z "jobmanager-future-thread-1" #62 daemon prio=5
> os_prio=0 tid=0x00007f4cb001c000 nid=0x7437 waiting on condition
> [0x00007f4d1053a000]
> 2020-06-04T11:34:52.0148220Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0148887Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0150135Z - parking to wait for <0x000000008797abe8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0151379Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0152526Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0153885Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> 2020-06-04T11:34:52.0155318Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0156536Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0157700Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0158762Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0159663Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0160089Z
> 2020-06-04T11:34:52.0161328Z "mini-cluster-io-thread-8" #60 daemon prio=5
> os_prio=0 tid=0x00007f4cfc017800 nid=0x7435 waiting on condition
> [0x00007f4d1093c000]
> 2020-06-04T11:34:52.0162317Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0162964Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0164236Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0165367Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0166572Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0168520Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0169554Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0170631Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0171697Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0172596Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0173024Z
> 2020-06-04T11:34:52.0174262Z "mini-cluster-io-thread-7" #59 daemon prio=5
> os_prio=0 tid=0x00007f4cfc016000 nid=0x7434 waiting on condition
> [0x00007f4d10a3d000]
> 2020-06-04T11:34:52.0175318Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0176020Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0177340Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0178416Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0179587Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0180777Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0181838Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0182888Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0183966Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0185038Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0185530Z
> 2020-06-04T11:34:52.0186770Z "mini-cluster-io-thread-6" #58 daemon prio=5
> os_prio=0 tid=0x00007f4cfc014800 nid=0x7433 waiting on condition
> [0x00007f4d10b3e000]
> 2020-06-04T11:34:52.0187836Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0188506Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0189757Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0190825Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0191976Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0193303Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0194355Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0195678Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0197319Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0198196Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0198648Z
> 2020-06-04T11:34:52.0199892Z "mini-cluster-io-thread-5" #57 daemon prio=5
> os_prio=0 tid=0x00007f4cb4003800 nid=0x7432 waiting on condition
> [0x00007f4d10c3f000]
> 2020-06-04T11:34:52.0200877Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0201523Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0202784Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0203856Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0205091Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0206342Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0207441Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0208512Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0209704Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0210593Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0211018Z
> 2020-06-04T11:34:52.0212247Z "mini-cluster-io-thread-4" #56 daemon prio=5
> os_prio=0 tid=0x00007f4cc616e000 nid=0x7431 waiting on condition
> [0x00007f4d10d40000]
> 2020-06-04T11:34:52.0213243Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0213887Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0215292Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0216467Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0217738Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0218929Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0219990Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0221055Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0222114Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0223005Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0223439Z
> 2020-06-04T11:34:52.0224768Z "mini-cluster-io-thread-3" #55 daemon prio=5
> os_prio=0 tid=0x00007f4cf4014800 nid=0x7430 waiting on condition
> [0x00007f4d10e41000]
> 2020-06-04T11:34:52.0225773Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0226441Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0227766Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0228837Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0230003Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0231194Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0232248Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0233294Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0234581Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0235493Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0235938Z
> 2020-06-04T11:34:52.0237207Z "mini-cluster-io-thread-2" #54 daemon prio=5
> os_prio=0 tid=0x00007f4cc6141800 nid=0x742f waiting on condition
> [0x00007f4d10f42000]
> 2020-06-04T11:34:52.0238195Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0238860Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0240116Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0241187Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0242335Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0243547Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0244662Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0245767Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0246845Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0247775Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0248219Z
> 2020-06-04T11:34:52.0249480Z "Flink-DispatcherRestEndpoint-thread-1" #53
> daemon prio=5 os_prio=0 tid=0x00007f4cc60c4800 nid=0x742e waiting on
> condition [0x00007f4d11043000]
> 2020-06-04T11:34:52.0250644Z java.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-06-04T11:34:52.0251313Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0252604Z - parking to wait for <0x0000000087803238> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0253674Z at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:52.0254977Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:52.0256397Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:52.0257767Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0258964Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0260021Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0261102Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0261991Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0262415Z
> 2020-06-04T11:34:52.0263611Z "mini-cluster-io-thread-1" #52 daemon prio=5
> os_prio=0 tid=0x00007f4cc60c3000 nid=0x742d waiting on condition
> [0x00007f4d11144000]
> 2020-06-04T11:34:52.0264670Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0265335Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0266619Z - parking to wait for <0x000000008797b270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0267750Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0268900Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0270115Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0271168Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0272219Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0273298Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0274314Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0274838Z
> 2020-06-04T11:34:52.0276095Z "flink-rest-server-netty-boss-thread-1" #51
> daemon prio=5 os_prio=0 tid=0x00007f4cc6062000 nid=0x742c runnable
> [0x00007f4d11245000]
> 2020-06-04T11:34:52.0277062Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0277813Z at
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 2020-06-04T11:34:52.0278653Z at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 2020-06-04T11:34:52.0279581Z at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> 2020-06-04T11:34:52.0280504Z at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> 2020-06-04T11:34:52.0281888Z - locked <0x00000000879040b0> (a
> org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySet)
> 2020-06-04T11:34:52.0283156Z - locked <0x00000000879040a0> (a
> java.util.Collections$UnmodifiableSet)
> 2020-06-04T11:34:52.0284265Z - locked <0x0000000087904058> (a
> sun.nio.ch.EPollSelectorImpl)
> 2020-06-04T11:34:52.0285156Z at
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> 2020-06-04T11:34:52.0286383Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
> 2020-06-04T11:34:52.0287813Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)
> 2020-06-04T11:34:52.0288990Z at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)
> 2020-06-04T11:34:52.0290320Z at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> 2020-06-04T11:34:52.0291783Z at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2020-06-04T11:34:52.0292788Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0293213Z
> 2020-06-04T11:34:52.0294026Z "IOManager reader thread #1" #46 daemon prio=5
> os_prio=0 tid=0x00007f4e711b9000 nid=0x7429 waiting on condition
> [0x00007f4d1206d000]
> 2020-06-04T11:34:52.0295102Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0295781Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0297076Z - parking to wait for <0x000000008797ba50> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0298195Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0299365Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0300565Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0301714Z at
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$ReaderThread.run(IOManagerAsync.java:354)
> 2020-06-04T11:34:52.0302413Z
> 2020-06-04T11:34:52.0303226Z "IOManager writer thread #1" #45 daemon prio=5
> os_prio=0 tid=0x00007f4e711b7000 nid=0x7428 waiting on condition
> [0x00007f4d1216e000]
> 2020-06-04T11:34:52.0304216Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0304942Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0306264Z - parking to wait for <0x000000008777ee58> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0307417Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0308735Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> 2020-06-04T11:34:52.0309996Z at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 2020-06-04T11:34:52.0311166Z at
> org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:460)
> 2020-06-04T11:34:52.0311865Z
> 2020-06-04T11:34:52.0312999Z "Timer-2" #43 daemon prio=5 os_prio=0
> tid=0x00007f4e7117f000 nid=0x7427 in Object.wait() [0x00007f4d1226f000]
> 2020-06-04T11:34:52.0313971Z java.lang.Thread.State: TIMED_WAITING (on
> object monitor)
> 2020-06-04T11:34:52.0314967Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0316159Z - waiting on <0x000000008797bc58> (a
> java.util.TaskQueue)
> 2020-06-04T11:34:52.0316929Z at
> java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-06-04T11:34:52.0318069Z - locked <0x000000008797bc58> (a
> java.util.TaskQueue)
> 2020-06-04T11:34:52.0318790Z at java.util.TimerThread.run(Timer.java:505)
> 2020-06-04T11:34:52.0319253Z
> 2020-06-04T11:34:52.0320336Z "Timer-1" #41 daemon prio=5 os_prio=0
> tid=0x00007f4e7117c800 nid=0x7426 in Object.wait() [0x00007f4d12370000]
> 2020-06-04T11:34:52.0321313Z java.lang.Thread.State: TIMED_WAITING (on
> object monitor)
> 2020-06-04T11:34:52.0322018Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0322990Z - waiting on <0x0000000087803a80> (a
> java.util.TaskQueue)
> 2020-06-04T11:34:52.0323768Z at
> java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-06-04T11:34:52.0324863Z - locked <0x0000000087803a80> (a
> java.util.TaskQueue)
> 2020-06-04T11:34:52.0325652Z at java.util.TimerThread.run(Timer.java:505)
> 2020-06-04T11:34:52.0326096Z
> 2020-06-04T11:34:52.0326861Z "BLOB Server listener at 41173" #37 daemon
> prio=5 os_prio=0 tid=0x00007f4e7117a000 nid=0x7425 runnable
> [0x00007f4d12471000]
> 2020-06-04T11:34:52.0327864Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0328567Z at java.net.PlainSocketImpl.socketAccept(Native
> Method)
> 2020-06-04T11:34:52.0329437Z at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> 2020-06-04T11:34:52.0330392Z at
> java.net.ServerSocket.implAccept(ServerSocket.java:560)
> 2020-06-04T11:34:52.0331352Z at
> java.net.ServerSocket.accept(ServerSocket.java:528)
> 2020-06-04T11:34:52.0332267Z at
> org.apache.flink.runtime.blob.BlobServer.run(BlobServer.java:262)
> 2020-06-04T11:34:52.0332818Z
> 2020-06-04T11:34:52.0333941Z "Timer-0" #38 daemon prio=5 os_prio=0
> tid=0x00007f4e71160000 nid=0x7424 in Object.wait() [0x00007f4d12772000]
> 2020-06-04T11:34:52.0334980Z java.lang.Thread.State: TIMED_WAITING (on
> object monitor)
> 2020-06-04T11:34:52.0335747Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0336733Z - waiting on <0x0000000087803a68> (a
> java.util.TaskQueue)
> 2020-06-04T11:34:52.0337565Z at
> java.util.TimerThread.mainLoop(Timer.java:552)
> 2020-06-04T11:34:52.0338608Z - locked <0x0000000087803a68> (a
> java.util.TaskQueue)
> 2020-06-04T11:34:52.0339327Z at java.util.TimerThread.run(Timer.java:505)
> 2020-06-04T11:34:52.0339785Z
> 2020-06-04T11:34:52.0341535Z "flink-metrics-scheduler-1" #33 prio=5 os_prio=0
> tid=0x00007f4e71126000 nid=0x7420 waiting on condition [0x00007f4d12b76000]
> 2020-06-04T11:34:52.0342549Z java.lang.Thread.State: TIMED_WAITING
> (sleeping)
> 2020-06-04T11:34:52.0343227Z at java.lang.Thread.sleep(Native Method)
> 2020-06-04T11:34:52.0344121Z at
> akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
> 2020-06-04T11:34:52.0345384Z at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
> 2020-06-04T11:34:52.0346626Z at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
> 2020-06-04T11:34:52.0347630Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0348057Z
> 2020-06-04T11:34:52.0349204Z "flink-scheduler-1" #27 prio=5 os_prio=0
> tid=0x00007f4e70d14800 nid=0x7418 waiting on condition [0x00007f4d13390000]
> 2020-06-04T11:34:52.0350172Z java.lang.Thread.State: TIMED_WAITING
> (sleeping)
> 2020-06-04T11:34:52.0350872Z at java.lang.Thread.sleep(Native Method)
> 2020-06-04T11:34:52.0351746Z at
> akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
> 2020-06-04T11:34:52.0352929Z at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
> 2020-06-04T11:34:52.0354102Z at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
> 2020-06-04T11:34:52.0355209Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0355675Z
> 2020-06-04T11:34:52.0356453Z "process reaper" #24 daemon prio=10 os_prio=0
> tid=0x00007f4d18048800 nid=0x7415 waiting on condition [0x00007f4d13dce000]
> 2020-06-04T11:34:52.0357656Z java.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-06-04T11:34:52.0358349Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0359574Z - parking to wait for <0x0000000080c55a08> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> 2020-06-04T11:34:52.0360584Z at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:52.0361681Z at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 2020-06-04T11:34:52.0362832Z at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> 2020-06-04T11:34:52.0363905Z at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> 2020-06-04T11:34:52.0365103Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> 2020-06-04T11:34:52.0366197Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0367350Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0368223Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0368669Z
> 2020-06-04T11:34:52.0369901Z "surefire-forkedjvm-ping-30s" #23 daemon prio=5
> os_prio=0 tid=0x00007f4e703f6000 nid=0x7412 waiting on condition
> [0x00007f4d343c2000]
> 2020-06-04T11:34:52.0370924Z java.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-06-04T11:34:52.0371592Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0372853Z - parking to wait for <0x0000000080c79c40> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2020-06-04T11:34:52.0374047Z at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-06-04T11:34:52.0375351Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> 2020-06-04T11:34:52.0376773Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> 2020-06-04T11:34:52.0378153Z at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> 2020-06-04T11:34:52.0379351Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> 2020-06-04T11:34:52.0380402Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> 2020-06-04T11:34:52.0381480Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-04T11:34:52.0382367Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0382797Z
> 2020-06-04T11:34:52.0383991Z "surefire-forkedjvm-command-thread" #22 daemon
> prio=5 os_prio=0 tid=0x00007f4e703c4000 nid=0x7411 runnable
> [0x00007f4d346cd000]
> 2020-06-04T11:34:52.0385010Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0385725Z at java.io.FileInputStream.readBytes(Native
> Method)
> 2020-06-04T11:34:52.0386500Z at
> java.io.FileInputStream.read(FileInputStream.java:255)
> 2020-06-04T11:34:52.0387475Z at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> 2020-06-04T11:34:52.0388394Z at
> java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> 2020-06-04T11:34:52.0389548Z - locked <0x0000000080c9fff8> (a
> java.io.BufferedInputStream)
> 2020-06-04T11:34:52.0390386Z at
> java.io.DataInputStream.readInt(DataInputStream.java:387)
> 2020-06-04T11:34:52.0391401Z at
> org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
> 2020-06-04T11:34:52.0392588Z at
> org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391)
> 2020-06-04T11:34:52.0393518Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-04T11:34:52.0393960Z
> 2020-06-04T11:34:52.0394775Z "Service Thread" #21 daemon prio=9 os_prio=0
> tid=0x00007f4e702d5000 nid=0x740f runnable [0x0000000000000000]
> 2020-06-04T11:34:52.0395757Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0396158Z
> 2020-06-04T11:34:52.0396945Z "C1 CompilerThread14" #20 daemon prio=9
> os_prio=0 tid=0x00007f4e702c8000 nid=0x740e waiting on condition
> [0x0000000000000000]
> 2020-06-04T11:34:52.0398088Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0398505Z
> 2020-06-04T11:34:52.0399271Z "C1 CompilerThread13" #19 daemon prio=9
> os_prio=0 tid=0x00007f4e702c5800 nid=0x740d waiting on condition
> [0x0000000000000000]
> 2020-06-04T11:34:52.0400201Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0400594Z
> 2020-06-04T11:34:52.0401379Z "C1 CompilerThread12" #18 daemon prio=9
> os_prio=0 tid=0x00007f4e702c3800 nid=0x740c waiting on condition
> [0x0000000000000000]
> 2020-06-04T11:34:52.0402290Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0402685Z
> 2020-06-04T11:34:52.0403469Z "C1 CompilerThread11" #17 daemon prio=9
> os_prio=0 tid=0x00007f4e702c1800 nid=0x740b waiting on condition
> [0x0000000000000000]
> 2020-06-04T11:34:52.0404393Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0404872Z
> 2020-06-04T11:34:52.0405666Z "C1 CompilerThread10" #16 daemon prio=9
> os_prio=0 tid=0x00007f4e702c0000 nid=0x740a waiting on condition
> [0x0000000000000000]
> 2020-06-04T11:34:52.0406601Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0406994Z
> 2020-06-04T11:34:52.0407850Z "C2 CompilerThread9" #15 daemon prio=9 os_prio=0
> tid=0x00007f4e702bd000 nid=0x7409 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0408779Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0409172Z
> 2020-06-04T11:34:52.0409935Z "C2 CompilerThread8" #14 daemon prio=9 os_prio=0
> tid=0x00007f4e702bb000 nid=0x7408 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0410862Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0411376Z
> 2020-06-04T11:34:52.0412166Z "C2 CompilerThread7" #13 daemon prio=9 os_prio=0
> tid=0x00007f4e702b9000 nid=0x7407 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0413076Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0413488Z
> 2020-06-04T11:34:52.0414247Z "C2 CompilerThread6" #12 daemon prio=9 os_prio=0
> tid=0x00007f4e702b7000 nid=0x7406 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0415279Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0415709Z
> 2020-06-04T11:34:52.0416490Z "C2 CompilerThread5" #11 daemon prio=9 os_prio=0
> tid=0x00007f4e702b5000 nid=0x7405 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0417456Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0417872Z
> 2020-06-04T11:34:52.0418634Z "C2 CompilerThread4" #10 daemon prio=9 os_prio=0
> tid=0x00007f4e702b3000 nid=0x7404 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0419554Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0419952Z
> 2020-06-04T11:34:52.0420729Z "C2 CompilerThread3" #9 daemon prio=9 os_prio=0
> tid=0x00007f4e702a9000 nid=0x7403 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0421635Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0422055Z
> 2020-06-04T11:34:52.0422815Z "C2 CompilerThread2" #8 daemon prio=9 os_prio=0
> tid=0x00007f4e702a6800 nid=0x7402 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0423747Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0424141Z
> 2020-06-04T11:34:52.0424985Z "C2 CompilerThread1" #7 daemon prio=9 os_prio=0
> tid=0x00007f4e702a4800 nid=0x7401 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0425921Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0426339Z
> 2020-06-04T11:34:52.0427094Z "C2 CompilerThread0" #6 daemon prio=9 os_prio=0
> tid=0x00007f4e702a2800 nid=0x7400 waiting on condition [0x0000000000000000]
> 2020-06-04T11:34:52.0428062Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0428457Z
> 2020-06-04T11:34:52.0429191Z "Signal Dispatcher" #5 daemon prio=9 os_prio=0
> tid=0x00007f4e702a0800 nid=0x73ff runnable [0x0000000000000000]
> 2020-06-04T11:34:52.0430050Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0430443Z
> 2020-06-04T11:34:52.0431290Z "Surrogate Locker Thread (Concurrent GC)" #4
> daemon prio=9 os_prio=0 tid=0x00007f4e7029f000 nid=0x73fe waiting on
> condition [0x0000000000000000]
> 2020-06-04T11:34:52.0432508Z java.lang.Thread.State: RUNNABLE
> 2020-06-04T11:34:52.0432969Z
> 2020-06-04T11:34:52.0433783Z "Finalizer" #3 daemon prio=8 os_prio=0
> tid=0x00007f4e7026e800 nid=0x73fd in Object.wait() [0x00007f4d7d0b2000]
> 2020-06-04T11:34:52.0434997Z java.lang.Thread.State: WAITING (on object
> monitor)
> 2020-06-04T11:34:52.0435827Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0437241Z - waiting on <0x0000000080c56688> (a
> java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:52.0438175Z at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> 2020-06-04T11:34:52.0439453Z - locked <0x0000000080c56688> (a
> java.lang.ref.ReferenceQueue$Lock)
> 2020-06-04T11:34:52.0440348Z at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> 2020-06-04T11:34:52.0441259Z at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
> 2020-06-04T11:34:52.0441825Z
> 2020-06-04T11:34:52.0442576Z "Reference Handler" #2 daemon prio=10 os_prio=0
> tid=0x00007f4e7026a000 nid=0x73fc in Object.wait() [0x00007f4d7d1b3000]
> 2020-06-04T11:34:52.0443567Z java.lang.Thread.State: WAITING (on object
> monitor)
> 2020-06-04T11:34:52.0444269Z at java.lang.Object.wait(Native Method)
> 2020-06-04T11:34:52.0445010Z at java.lang.Object.wait(Object.java:502)
> 2020-06-04T11:34:52.0445862Z at
> java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> 2020-06-04T11:34:52.0447007Z - locked <0x0000000080c56678> (a
> java.lang.ref.Reference$Lock)
> 2020-06-04T11:34:52.0447959Z at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> 2020-06-04T11:34:52.0448509Z
> 2020-06-04T11:34:52.0449218Z "main" #1 prio=5 os_prio=0
> tid=0x00007f4e7000b800 nid=0x73c4 waiting on condition [0x00007f4e768fd000]
> 2020-06-04T11:34:52.0450228Z java.lang.Thread.State: WAITING (parking)
> 2020-06-04T11:34:52.0450892Z at sun.misc.Unsafe.park(Native Method)
> 2020-06-04T11:34:52.0452052Z - parking to wait for <0x000000008757aa28> (a
> java.util.concurrent.CompletableFuture$Signaller)
> 2020-06-04T11:34:52.0453050Z at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2020-06-04T11:34:52.0454097Z at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2020-06-04T11:34:52.0455234Z at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> 2020-06-04T11:34:52.0456299Z at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2020-06-04T11:34:52.0457383Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2020-06-04T11:34:52.0458470Z at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:671)
> 2020-06-04T11:34:52.0459639Z at
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:81)
> 2020-06-04T11:34:52.0460933Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1685)
> 2020-06-04T11:34:52.0462322Z at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1667)
> 2020-06-04T11:34:52.0463724Z at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceITCase.executeAndVerify(CoordinatedSourceITCase.java:84)
> 2020-06-04T11:34:52.0465237Z at
> org.apache.flink.connector.base.source.reader.CoordinatedSourceITCase.testMultipleSources(CoordinatedSourceITCase.java:68)
> 2020-06-04T11:34:52.0466388Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-06-04T11:34:52.0467359Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-06-04T11:34:52.0468461Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-06-04T11:34:52.0469422Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-06-04T11:34:52.0470390Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-06-04T11:34:52.0471490Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-06-04T11:34:52.0472601Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-06-04T11:34:52.0473807Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-06-04T11:34:52.0474938Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-06-04T11:34:52.0475922Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-06-04T11:34:52.0476784Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-06-04T11:34:52.0477853Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-06-04T11:34:52.0478931Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-06-04T11:34:52.0479942Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-06-04T11:34:52.0480862Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-06-04T11:34:52.0481788Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-06-04T11:34:52.0482735Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-06-04T11:34:52.0483662Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-06-04T11:34:52.0484764Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-06-04T11:34:52.0485851Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-06-04T11:34:52.0486750Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-06-04T11:34:52.0487677Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-06-04T11:34:52.0488648Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-06-04T11:34:52.0489900Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-06-04T11:34:52.0491041Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-06-04T11:34:52.0492159Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-06-04T11:34:52.0493333Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-06-04T11:34:52.0494619Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2020-06-04T11:34:52.0495855Z at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> 2020-06-04T11:34:52.0496879Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> 2020-06-04T11:34:52.0497557Z
> 2020-06-04T11:34:52.0498119Z "VM Thread" os_prio=0 tid=0x00007f4e70260800
> nid=0x73fb runnable
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)