[Adding Dev and Removing Architecture] On Fri, Mar 11, 2016 at 8:08 PM, Mohanadarshan Vivekanandalingam < [email protected]> wrote:
> Hi, >> >> > Hi Isuru, > > Please find my comments below.. > > > >> This is regarding the analytics for US Election 2016 Tweets. The ESB uses >> Twitter Connector to find tweets with some specific hashtags and sends the >> tweets as events to CEP via HTTP. The CEP version is 4.0.0. >> >> The CEP receives the events via the Tomcat HTTP Connector (port 9763). As >> mentioned in $subject, the CEP fails to accept requests as there are no >> worker threads to handle the requests. >> >> I have attached a thread dump and I analyzed it using the ThreadLogic >> application [1]. All 250 HTTP worker threads (http-nio-9763-exec-*) are in >> "PARKING" state and following is a part of stack trace in each thread. >> >> "http-nio-9763-exec-102" #810 daemon prio=5 os_prio=0 >> tid=0x00007f7678010800 nid=0x710 waiting on condition [0x00007f75da48f000] >> *java.lang.Thread.State: WAITING (parking)* >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00000005cc6bd6a8> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >> at >> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) >> >> >> >> >> * at >> org.wso2.carbon.event.input.adapter.http.HTTPEventAdapter$1.rejectedExecution(HTTPEventAdapter.java:99) >> at >> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) >> at >> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) >> at >> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) >> at >> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet.doPost(HTTPMessageServlet.java:177)* >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >> at >> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >> >> >> The HTTPMessageServlet submits a HTTPRequestProcessor [2] to an executor >> service and the executor service rejects it as the worker queue is full. >> However in the RejectedExecutionHandler, the task is put back to same >> queue [3]. Here the thread gets parked while waiting for some condition. >> This is why the Tomcat HTTP connector can no longer process any requests. >> >> > Yes, above implementation is done purposefully.. We had a requirement > where we need to block the HTTP requests when there is no thread (or space > in the queue) rather dropping the events. That is why above HTTP adapter is > implemented in such way.. Because of above implementation, there will not > be an event loss at receiver level. > > >> Then I checked the thread pool used in [2] and found out that all 100 >> threads (pool-75-thread-*) in that pool are also in "PARKING" state. >> Following is the stack trace. >> >> >> "pool-75-thread-100" #758 prio=5 os_prio=0 tid=0x00007f7634017800 >> nid=0x6ba runnable [0x00007f75dd8c4000] >> * java.lang.Thread.State: TIMED_WAITING (parking)* >> at sun.misc.Unsafe.park(Native Method) >> at >> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) >> at >> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136) >> at >> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105) >> at *com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)* >> at >> *org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:74)* >> at >> org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:99) >> at >> org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:49) >> at >> org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39) >> at >> org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:92) >> at >> org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:142) >> at >> org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27) >> at >> org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:256) >> at >> org.wso2.carbon.event.receiver.core.internal.EventReceiver.processMappedEvent(EventReceiver.java:200) >> at >> org.wso2.carbon.event.receiver.core.internal.EventReceiver$MappedEventSubscription.onEvent(EventReceiver.java:307) >> at >> org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(InputAdapterRuntime.java:109) >> at >> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet$HTTPRequestProcessor.run(HTTPMessageServlet.java:210) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> >> The call to RingBuffer.next method in SingleStreamEntryValve.send [4] >> puts the thread into the PARKING state. >> >> We need to figure out the reason for this. I really appreciate if you >> could let us know what we should check further and how to avoid this kind >> of behaviour with CEP. >> > > Now, let me explain about this issue.. In above usecase CEP is not running > with default functionalities or components.. Above CEP instance contains > few extensions (which is extended).. When analyzing the issue, found that a > specific extension is causing this behavior.. That extension consumes > considerable amount of processing time.. And it eventually become as the > reason for distributor to get full. Removing the extension solves the > issue.. > > It seems, above extension is a must for the usecase. Then, I have asked to > divide the execution plan in to few parts to reduce the load for the > distruptor (and to reduce the processing time of the single execution > plan). After this change, system is back to normal and haven't seen such > behavior.. > > Let's monitor the system over the weekend and see... > > >> >> I think it's better to include some metrics in CEP to see how the CEP >> engine is behaving, especially some metrics related to Disruptor. Do we >> have such metrics in the latest CEP release? >> >> > No, we don't have such metrics specifically for Distruptor.. But, I > believe we can use Siddhi metrics.. > > Thanks, > Mohan > > >> Thanks! >> >> Best Regards, >> >> [1] https://java.net/projects/threadlogic >> [2] >> https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPMessageServlet.java#L177 >> [3] >> https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPEventAdapter.java#L99 >> [4] >> https://github.com/wso2/siddhi/blob/v3.0.2/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/stream/input/SingleStreamEntryValve.java#L74 >> >> -- >> Isuru Perera >> Associate Technical Lead | WSO2, Inc. | http://wso2.com/ >> Lean . Enterprise . Middleware >> >> about.me/chrishantha >> Contact: +IsuruPereraWSO2 <https://www.google.com/+IsuruPereraWSO2/about> >> > > > > -- > *V. Mohanadarshan* > *Senior Software Engineer,* > *Data Technologies Team,* > *WSO2, Inc. http://wso2.com <http://wso2.com> * > *lean.enterprise.middleware.* > > email: [email protected] > phone:(+94) 771117673 > -- *V. Mohanadarshan* *Senior Software Engineer,* *Data Technologies Team,* *WSO2, Inc. http://wso2.com <http://wso2.com> * *lean.enterprise.middleware.* email: [email protected] phone:(+94) 771117673
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
