Hi, This is regarding the analytics for US Election 2016 Tweets. The ESB uses Twitter Connector to find tweets with some specific hashtags and sends the tweets as events to CEP via HTTP. The CEP version is 4.0.0.
The CEP receives the events via the Tomcat HTTP Connector (port 9763). As
mentioned in $subject, the CEP fails to accept requests as there are no
worker threads to handle the requests.
I have attached a thread dump and I analyzed it using the ThreadLogic
application [1]. All 250 HTTP worker threads (http-nio-9763-exec-*) are in
"PARKING" state and following is a part of stack trace in each thread.
"http-nio-9763-exec-102" #810 daemon prio=5 os_prio=0
tid=0x00007f7678010800 nid=0x710 waiting on condition [0x00007f75da48f000]
*java.lang.Thread.State: WAITING (parking)*
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005cc6bd6a8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
* at
org.wso2.carbon.event.input.adapter.http.HTTPEventAdapter$1.rejectedExecution(HTTPEventAdapter.java:99)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at
org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet.doPost(HTTPMessageServlet.java:177)*
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
The HTTPMessageServlet submits a HTTPRequestProcessor [2] to an executor
service and the executor service rejects it as the worker queue is full.
However in the RejectedExecutionHandler, the task is put back to same queue
[3]. Here the thread gets parked while waiting for some condition. This is
why the Tomcat HTTP connector can no longer process any requests.
Then I checked the thread pool used in [2] and found out that all 100
threads (pool-75-thread-*) in that pool are also in "PARKING" state.
Following is the stack trace.
"pool-75-thread-100" #758 prio=5 os_prio=0 tid=0x00007f7634017800 nid=0x6ba
runnable [0x00007f75dd8c4000]
* java.lang.Thread.State: TIMED_WAITING (parking)*
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at
com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
at
com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
at *com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)*
at
*org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:74)*
at
org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:99)
at
org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:49)
at
org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39)
at
org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:92)
at
org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:142)
at
org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27)
at
org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:256)
at
org.wso2.carbon.event.receiver.core.internal.EventReceiver.processMappedEvent(EventReceiver.java:200)
at
org.wso2.carbon.event.receiver.core.internal.EventReceiver$MappedEventSubscription.onEvent(EventReceiver.java:307)
at
org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(InputAdapterRuntime.java:109)
at
org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet$HTTPRequestProcessor.run(HTTPMessageServlet.java:210)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The call to RingBuffer.next method in SingleStreamEntryValve.send [4] puts
the thread into the PARKING state.
We need to figure out the reason for this. I really appreciate if you could
let us know what we should check further and how to avoid this kind of
behaviour with CEP.
I think it's better to include some metrics in CEP to see how the CEP
engine is behaving, especially some metrics related to Disruptor. Do we
have such metrics in the latest CEP release?
Thanks!
Best Regards,
[1] https://java.net/projects/threadlogic
[2]
https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPMessageServlet.java#L177
[3]
https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPEventAdapter.java#L99
[4]
https://github.com/wso2/siddhi/blob/v3.0.2/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/stream/input/SingleStreamEntryValve.java#L74
--
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware
about.me/chrishantha
Contact: +IsuruPereraWSO2 <https://www.google.com/+IsuruPereraWSO2/about>
CEP_Thread_Dump.txt.tar.gz
Description: GNU Zip compressed data
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
