> > I think it's better to include some metrics in CEP to see how the CEP > engine is behaving, especially some metrics related to Disruptor. Do we > have such metrics in the latest CEP release? >
Currently we don't have queue sizes as metrics in Siddhi, but we do have, - throughput per stream - per query latency - per query memory consumption as metrics in Siddhi. I think as Mohan mentioned we can use those for identifying the hotspots. But it's included only in CEP 4.1.0 Thanks *,Sajith Ravindra* Senior Software Engineer WSO2 Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 77 2273550 blog: http://sajithr.blogspot.com/ <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> On Fri, Mar 11, 2016 at 7:39 AM, Mohanadarshan Vivekanandalingam < [email protected]> wrote: > [Adding Dev and Removing Architecture] > > On Fri, Mar 11, 2016 at 8:08 PM, Mohanadarshan Vivekanandalingam < > [email protected]> wrote: > >> Hi, >>> >>> >> Hi Isuru, >> >> Please find my comments below.. >> >> >> >>> This is regarding the analytics for US Election 2016 Tweets. The ESB >>> uses Twitter Connector to find tweets with some specific hashtags and sends >>> the tweets as events to CEP via HTTP. The CEP version is 4.0.0. >>> >>> The CEP receives the events via the Tomcat HTTP Connector (port 9763). >>> As mentioned in $subject, the CEP fails to accept requests as there are no >>> worker threads to handle the requests. >>> >>> I have attached a thread dump and I analyzed it using the ThreadLogic >>> application [1]. All 250 HTTP worker threads (http-nio-9763-exec-*) are in >>> "PARKING" state and following is a part of stack trace in each thread. >>> >>> "http-nio-9763-exec-102" #810 daemon prio=5 os_prio=0 >>> tid=0x00007f7678010800 nid=0x710 waiting on condition [0x00007f75da48f000] >>> *java.lang.Thread.State: WAITING (parking)* >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x00000005cc6bd6a8> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >>> at >>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) >>> >>> >>> >>> >>> * at >>> org.wso2.carbon.event.input.adapter.http.HTTPEventAdapter$1.rejectedExecution(HTTPEventAdapter.java:99) >>> at >>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) >>> at >>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) >>> at >>> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) >>> at >>> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet.doPost(HTTPMessageServlet.java:177)* >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >>> at >>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >>> >>> >>> The HTTPMessageServlet submits a HTTPRequestProcessor [2] to an >>> executor service and the executor service rejects it as the worker queue is >>> full. However in the RejectedExecutionHandler, the task is put back to >>> same queue [3]. Here the thread gets parked while waiting for some >>> condition. This is why the Tomcat HTTP connector can no longer process any >>> requests. >>> >>> >> Yes, above implementation is done purposefully.. We had a requirement >> where we need to block the HTTP requests when there is no thread (or space >> in the queue) rather dropping the events. That is why above HTTP adapter is >> implemented in such way.. Because of above implementation, there will not >> be an event loss at receiver level. >> >> >>> Then I checked the thread pool used in [2] and found out that all 100 >>> threads (pool-75-thread-*) in that pool are also in "PARKING" state. >>> Following is the stack trace. >>> >>> >>> "pool-75-thread-100" #758 prio=5 os_prio=0 tid=0x00007f7634017800 >>> nid=0x6ba runnable [0x00007f75dd8c4000] >>> * java.lang.Thread.State: TIMED_WAITING (parking)* >>> at sun.misc.Unsafe.park(Native Method) >>> at >>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) >>> at >>> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136) >>> at >>> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105) >>> at *com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)* >>> at >>> *org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:74)* >>> at >>> org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:99) >>> at >>> org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:49) >>> at >>> org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39) >>> at >>> org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:92) >>> at >>> org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:142) >>> at >>> org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27) >>> at >>> org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:256) >>> at >>> org.wso2.carbon.event.receiver.core.internal.EventReceiver.processMappedEvent(EventReceiver.java:200) >>> at >>> org.wso2.carbon.event.receiver.core.internal.EventReceiver$MappedEventSubscription.onEvent(EventReceiver.java:307) >>> at >>> org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(InputAdapterRuntime.java:109) >>> at >>> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet$HTTPRequestProcessor.run(HTTPMessageServlet.java:210) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> The call to RingBuffer.next method in SingleStreamEntryValve.send [4] >>> puts the thread into the PARKING state. >>> >>> We need to figure out the reason for this. I really appreciate if you >>> could let us know what we should check further and how to avoid this kind >>> of behaviour with CEP. >>> >> >> Now, let me explain about this issue.. In above usecase CEP is not >> running with default functionalities or components.. Above CEP instance >> contains few extensions (which is extended).. When analyzing the issue, >> found that a specific extension is causing this behavior.. That extension >> consumes considerable amount of processing time.. And it eventually become >> as the reason for distributor to get full. Removing the extension solves >> the issue.. >> >> It seems, above extension is a must for the usecase. Then, I have asked >> to divide the execution plan in to few parts to reduce the load for the >> distruptor (and to reduce the processing time of the single execution >> plan). After this change, system is back to normal and haven't seen such >> behavior.. >> >> Let's monitor the system over the weekend and see... >> >> >>> >>> I think it's better to include some metrics in CEP to see how the CEP >>> engine is behaving, especially some metrics related to Disruptor. Do we >>> have such metrics in the latest CEP release? >>> >>> >> No, we don't have such metrics specifically for Distruptor.. But, I >> believe we can use Siddhi metrics.. >> >> Thanks, >> Mohan >> >> >>> Thanks! >>> >>> Best Regards, >>> >>> [1] https://java.net/projects/threadlogic >>> [2] >>> https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPMessageServlet.java#L177 >>> [3] >>> https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPEventAdapter.java#L99 >>> [4] >>> https://github.com/wso2/siddhi/blob/v3.0.2/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/stream/input/SingleStreamEntryValve.java#L74 >>> >>> -- >>> Isuru Perera >>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/ >>> Lean . Enterprise . Middleware >>> >>> about.me/chrishantha >>> Contact: +IsuruPereraWSO2 >>> <https://www.google.com/+IsuruPereraWSO2/about> >>> >> >> >> >> -- >> *V. Mohanadarshan* >> *Senior Software Engineer,* >> *Data Technologies Team,* >> *WSO2, Inc. http://wso2.com <http://wso2.com> * >> *lean.enterprise.middleware.* >> >> email: [email protected] >> phone:(+94) 771117673 >> > > > > -- > *V. Mohanadarshan* > *Senior Software Engineer,* > *Data Technologies Team,* > *WSO2, Inc. http://wso2.com <http://wso2.com> * > *lean.enterprise.middleware.* > > email: [email protected] > phone:(+94) 771117673 > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
