Someone with better knowledge of Flink runner will be able to assist you better.
Looks like all these threads are waiting for work. I think it will be more useful if you provide access to stacktraces of all the threads (post a an external link). Since there are no threads in 'KafkaUnboundedReader::consumerPollLoop()', it looks like none of the sources is actually active. On Fri, Nov 11, 2016 at 5:10 PM, amir bahmanyari <[email protected]> wrote: > Hi Raghu, > This is text version (of the threads; all look the same) of the image. > Hope makes some sense as to why it gets so slow... > Cheers+have a wonderful weekend. > > Name: Time Trigger for Source: Read(UnboundedKafkaSource) -> > AnonymousParDo -> AnonymousParDo (45/64) > State: TIMED_WAITING on java.util.concurrent.locks. > AbstractQueuedSynchronizer$ConditionObject@3f752f28 > Total blocked: 8,369 Total waited: 77,763 > > Stack trace: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take( > ScheduledThreadPoolExecutor.java:1093) > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take( > ScheduledThreadPoolExecutor.java:809) > java.util.concurrent.ThreadPoolExecutor.getTask( > ThreadPoolExecutor.java:1067) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1127) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > java.lang.Thread.run(Thread.java:745) > > > > ------------------------------ > *From:* Raghu Angadi <[email protected]> > *To:* [email protected]; amir bahmanyari <[email protected]> > > *Sent:* Thursday, November 10, 2016 6:20 PM > *Subject:* Re: TIMED_WAITING KafkaIO() Threads State > > Hi Amir, > > I suggest you post actual relevant thread stacks here. We can see those in > the the image, but text is better. > Essentially Flink UnboundedSource reader is doing a sleep().. > > On Thu, Nov 10, 2016 at 5:59 PM, amir bahmanyari <[email protected]> > wrote: > > I am checking the runtime state of the KafkaIO() threads using jconsole. > I see as many cores as I have specified on my server all right. > Is TIMED_WAITING the right state for them all to be in all the time? > I monitor them and that state never changes... > Is the "Total waited" time range reasonable? > Thanks+regards, > > [image: Inline image] > > > > >
