Re: [Dev] CEP does not accept requests after receiving a high rate of events for some time

2016-03-11 Thread Sajith Ravindra 
>
> I think it's better to include some metrics in CEP to see how the CEP
> engine is behaving, especially some metrics related to Disruptor. Do we
> have such metrics in the latest CEP release?
>

Currently we don't have queue sizes as metrics in Siddhi, but we do have,
- throughput per stream
- per query latency
- per query memory consumption
as metrics in Siddhi.

I think as Mohan mentioned we can use those for identifying the hotspots.
But it's included only in CEP 4.1.0

Thanks
*,Sajith Ravindra*
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

mobile: +94 77 2273550
blog: http://sajithr.blogspot.com/


On Fri, Mar 11, 2016 at 7:39 AM, Mohanadarshan Vivekanandalingam <
mo...@wso2.com> wrote:

> [Adding Dev and Removing Architecture]
>
> On Fri, Mar 11, 2016 at 8:08 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> Hi,
>>>
>>>
>> Hi Isuru,
>>
>> Please find my comments below..
>>
>>
>>
>>> This is regarding the analytics for US Election 2016 Tweets. The ESB
>>> uses Twitter Connector to find tweets with some specific hashtags and sends
>>> the tweets as events to CEP via HTTP. The CEP version is 4.0.0.
>>>
>>> The CEP receives the events via the Tomcat HTTP Connector (port 9763).
>>> As mentioned in $subject, the CEP fails to accept requests as there are no
>>> worker threads to handle the requests.
>>>
>>> I have attached a thread dump and I analyzed it using the ThreadLogic
>>> application [1]. All 250 HTTP worker threads (http-nio-9763-exec-*) are in
>>> "PARKING" state and following is a part of stack trace in each thread.
>>>
>>> "http-nio-9763-exec-102" #810 daemon prio=5 os_prio=0
>>> tid=0x7f7678010800 nid=0x710 waiting on condition [0x7f75da48f000]
>>>*java.lang.Thread.State: WAITING (parking)*
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0005cc6bd6a8> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
>>>
>>>
>>>
>>>
>>> *at
>>> org.wso2.carbon.event.input.adapter.http.HTTPEventAdapter$1.rejectedExecution(HTTPEventAdapter.java:99)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>>> at
>>> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
>>> at
>>> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet.doPost(HTTPMessageServlet.java:177)*
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>
>>>
>>> The HTTPMessageServlet submits a HTTPRequestProcessor [2] to an
>>> executor service and the executor service rejects it as the worker queue is
>>> full. However in the RejectedExecutionHandler, the task is put back to
>>> same queue [3]. Here the thread gets parked while waiting for some
>>> condition. This is why the Tomcat HTTP connector can no longer process any
>>> requests.
>>>
>>>
>> Yes, above implementation is done purposefully.. We had a requirement
>> where we need to block the HTTP requests when there is no thread (or space
>> in the queue) rather dropping the events. That is why above HTTP adapter is
>> implemented in such way.. Because of above implementation, there will not
>> be an event loss at receiver level.
>>
>>
>>> Then I checked the thread pool used in [2] and found out that all 100
>>> threads (pool-75-thread-*) in that pool are also in "PARKING" state.
>>> Following is the stack trace.
>>>
>>>
>>> "pool-75-thread-100" #758 prio=5 os_prio=0 tid=0x7f7634017800
>>> nid=0x6ba runnable [0x7f75dd8c4000]
>>> *   java.lang.Thread.State: TIMED_WAITING (parking)*
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
>>> at
>>> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
>>> at
>>> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
>>> at *com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)*
>>> at
>>> *org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:74)*
>>> at
>>> org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:99)
>>> at
>>> org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:49)
>>> at
>>> org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputE

Re: [Dev] CEP does not accept requests after receiving a high rate of events for some time

2016-03-11 Thread Mohanadarshan Vivekanandalingam 
[Adding Dev and Removing Architecture]

On Fri, Mar 11, 2016 at 8:08 PM, Mohanadarshan Vivekanandalingam <
mo...@wso2.com> wrote:

> Hi,
>>
>>
> Hi Isuru,
>
> Please find my comments below..
>
>
>
>> This is regarding the analytics for US Election 2016 Tweets. The ESB uses
>> Twitter Connector to find tweets with some specific hashtags and sends the
>> tweets as events to CEP via HTTP. The CEP version is 4.0.0.
>>
>> The CEP receives the events via the Tomcat HTTP Connector (port 9763). As
>> mentioned in $subject, the CEP fails to accept requests as there are no
>> worker threads to handle the requests.
>>
>> I have attached a thread dump and I analyzed it using the ThreadLogic
>> application [1]. All 250 HTTP worker threads (http-nio-9763-exec-*) are in
>> "PARKING" state and following is a part of stack trace in each thread.
>>
>> "http-nio-9763-exec-102" #810 daemon prio=5 os_prio=0
>> tid=0x7f7678010800 nid=0x710 waiting on condition [0x7f75da48f000]
>>*java.lang.Thread.State: WAITING (parking)*
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0005cc6bd6a8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
>>
>>
>>
>>
>> *at
>> org.wso2.carbon.event.input.adapter.http.HTTPEventAdapter$1.rejectedExecution(HTTPEventAdapter.java:99)
>> at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>> at
>> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
>> at
>> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet.doPost(HTTPMessageServlet.java:177)*
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>> at
>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>
>>
>> The HTTPMessageServlet submits a HTTPRequestProcessor [2] to an executor
>> service and the executor service rejects it as the worker queue is full.
>> However in the RejectedExecutionHandler, the task is put back to same
>> queue [3]. Here the thread gets parked while waiting for some condition.
>> This is why the Tomcat HTTP connector can no longer process any requests.
>>
>>
> Yes, above implementation is done purposefully.. We had a requirement
> where we need to block the HTTP requests when there is no thread (or space
> in the queue) rather dropping the events. That is why above HTTP adapter is
> implemented in such way.. Because of above implementation, there will not
> be an event loss at receiver level.
>
>
>> Then I checked the thread pool used in [2] and found out that all 100
>> threads (pool-75-thread-*) in that pool are also in "PARKING" state.
>> Following is the stack trace.
>>
>>
>> "pool-75-thread-100" #758 prio=5 os_prio=0 tid=0x7f7634017800
>> nid=0x6ba runnable [0x7f75dd8c4000]
>> *   java.lang.Thread.State: TIMED_WAITING (parking)*
>> at sun.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
>> at
>> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
>> at
>> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
>> at *com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)*
>> at
>> *org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:74)*
>> at
>> org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:99)
>> at
>> org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:49)
>> at
>> org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39)
>> at
>> org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:92)
>> at
>> org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:142)
>> at
>> org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27)
>> at
>> org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:256)
>> at
>> org.wso2.carbon.event.receiver.core.internal.EventReceiver.processMappedEvent(EventReceiver.java:200)
>> at
>> org.wso2.carbon.event.receiver.core.internal.EventReceiver$MappedEventSubscription.onEvent(EventReceiver.java:307)
>> at
>> org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(In