Hi Ruwan, Suho Thanks for information provided will do the necessary tests as you mentioned.
thanks IsuruR On Mon, Apr 25, 2016 at 9:45 AM, Ruwan Abeykoon <[email protected]> wrote: > Hi Isuru, > I think the results are in favour of Disruptor model because, > 1. Latency and throughput seems to be stable in disruptor towards higher > load(at higher concurrency in the TPS and Latency graph) > 2. Disruptor based model uses less number of active threads. In > containerized environment, this is much more desirable as there will be > less number of context switches (due to threads) at high load and OS > can(usually) pin the active threads in the same core. > > I assume the above tests are done on single JVM. Can we run the same setup > with a load balancer fronted and simulate a handful of containers. That > will give us some realistic usage. The test server seems under utilized on > the above tests. I feel that almost same results on both tests are due to > the fact that the server is much more capable than the code demands. We > need to utilize around 80% of CPU and above 60% of physical RAM taken to > see any bottleneck. Further, we need to test One CPU bound disruptor, one > IO bound disruptor when doing this test agains 5 CPU disruptor case. > > Can you please share the code which the tests were done. > > > Cheers, > Ruwan > > On Sun, Apr 24, 2016 at 1:10 AM, Sriskandarajah Suhothayan <[email protected]> > wrote: > >> In CEP we have experienced latency issues chaining disrupter and don't go >> for PhasedBackOff strategy as there is a hit for calculating time and if >> you use spinning or yielding there you have to also correlate then with >> the number of cores on the system which will be a problem when it comes to >> deployment. >> >> Suho >> >> On Sat, Apr 23, 2016 at 7:18 PM, Isuru Ranawaka <[email protected]> wrote: >> >>> Hi Chanaka, >>> >>> I have run the test for header based routing scenario (Pure CPU) and >>> there is small improvement on Disruptor. We can get clear performance >>> difference if we send huge load such that requests are get queued at >>> ThreadPool and will has huge contention for internal queues but for >>> Disruptor it is locking free .when profiling this test I didn't see >>> contention in the ThreadPool level as well. Actually advantage we got >>> through Disruptor is due to cache optimization at disruptor level. Another >>> problem is we have active Netty thread pool as well. So there is no >>> probability that Disruptor threads are always get high priority. Due to >>> that also we cannot get maximum out of Disruptor thread model. I will run >>> long running test with significant amount of CPU bound work and see how it >>> behaves. >>> >>> @Suho, >>> >>> CPU bound Disruptor wait strategy is Sleep waiting and IO bound >>> Disruptor is Blocking Wait . Tested with PhasedBackOff with LiteLocking as >>> well not much difference in TPS but observed latency was varied in higher >>> amount for some requests. >>> >>> Number of Disruptors are configurable and I have used five CPU bound >>> Disruptors and one IO bound DIsruptor for testing which gives maximum Perf >>> according to testes we carried for Gateway. >>> >>> thanks >>> >>> >>> On Fri, Apr 22, 2016 at 8:20 PM, Sriskandarajah Suhothayan < >>> [email protected]> wrote: >>> >>>> Whats the waiting strategy that is being used in Disrupter? And is the >>>> maximum of Disrupter in all configuration is 2 ? >>>> >>>> Suho >>>> >>>> >>>> On Fri, Apr 22, 2016 at 8:01 PM, Chanaka Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi Isuru, >>>>> >>>>> Have we tested on the pure passthrough scenarios? According to the >>>>> results, we cannot see much performance difference in using disruptor over >>>>> thread pool. Shall we do some testing around passthrough scenarios and >>>>> verify? >>>>> >>>>> Cheers, >>>>> Chanaka >>>>> >>>>> On Fri, Apr 22, 2016 at 12:54 PM, Isuru Ranawaka <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We have working on decoupling engine thread model from transport >>>>>> thread model.Previously carbon transport was included with the Disruptor >>>>>> and according to the behaviour of MSF4J it is very hard to mapped custom >>>>>> logics written using MSF4J to Disruptor thread model in order to gain >>>>>> better performance. So we have moved Disruptor thread model to >>>>>> carbon-gateway and carbon-transport is kept with Netty thread pool. >>>>>> >>>>>> According to current implementation carbon-transport will dispatch >>>>>> events to registered message processor via Netty worker threads and there >>>>>> after it operates under engine level thread model. >>>>>> >>>>>> Following is the Thread model diagram for Integration Server. >>>>>> >>>>>> [image: gw_thread_model.png] >>>>>> >>>>>> Basically it works as follows >>>>>> >>>>>> - >>>>>> >>>>>> CPU bound mediators are working on CPU bound disruptor threads. >>>>>> - >>>>>> >>>>>> IO bound mediators are working on IO bound disruptor threads. >>>>>> - >>>>>> >>>>>> We assume custom mediators are written in bad way and may contain >>>>>> blocking calls so those are executed using IO bound mediator. >>>>>> - >>>>>> >>>>>> We have included ThreadPool implementation as well and can switch >>>>>> between ThreadPool based model or Disruptor based model. >>>>>> >>>>>> This is for we haven’t yet finalized exact thread model and we keep >>>>>> testing with different mediators how both are behaving according to >>>>>> different parameters like TPS, memory, startup time ,latency , .etc >>>>>> >>>>>> >>>>>> Following are some of the tests results we have conducted with both >>>>>> thread model implementations for two main scenarios. >>>>>> >>>>>> Machine Details >>>>>> >>>>>> Server :- 32 core machine with 64 GB memory >>>>>> >>>>>> Back End Service :- Netty based Echo Service which has TPS around >>>>>> 100000 >>>>>> >>>>>> Tested message size :- 4kb >>>>>> >>>>>> Server startup Time with Disruptor :- 1.34 s >>>>>> >>>>>> Server startup time without Disruptor :- 1.38 s >>>>>> >>>>>> Use case:- >>>>>> >>>>>> (CPU + IO) >>>>>> >>>>>> Header based routing with File Writing .One message path is writing >>>>>> message to file and other one sends messages to Echo service and respond >>>>>> back to client. >>>>>> >>>>>> TPS >>>>>> >>>>>> [image: image1.png] >>>>>> >>>>>> Latency >>>>>> >>>>>> [image: image2.png] >>>>>> >>>>>> Memory >>>>>> >>>>>> Disruptor >>>>>> >>>>>> [image: disruptor.png] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Thread Pool >>>>>> >>>>>> [image: threadpoolMemory.png] >>>>>> >>>>>> Use case:- >>>>>> >>>>>> (CPU ) >>>>>> >>>>>> Header based routing , send messages to Echo service and respond >>>>>> back to client. >>>>>> >>>>>> TPS >>>>>> >>>>>> [image: image3.png] >>>>>> >>>>>> Latency >>>>>> >>>>>> [image: image4.png] >>>>>> >>>>>> >>>>>> For test results please look in to [1] >>>>>> >>>>>> [1] >>>>>> https://docs.google.com/spreadsheets/d/1A2dxknP1xEJKBpl4ymbQD2Mt9kWywYCI-60j16JVLx0/edit#gid=0 >>>>>> >>>>>> >>>>>> >>>>>> Thanks >>>>>> IsuruR >>>>>> >>>>>> -- >>>>>> Best Regards >>>>>> Isuru Ranawaka >>>>>> M: +94714629880 >>>>>> Blog : http://isurur.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thank you and Best Regards, >>>>> Chanaka Fernando >>>>> Senior Technical Lead >>>>> WSO2, Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 773337238 >>>>> Blog : http://soatutorials.blogspot.com >>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>> Twitter:https://twitter.com/chanakaudaya >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *S. Suhothayan* >>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>> *WSO2 Inc. *http://wso2.com >>>> * <http://wso2.com/>* >>>> lean . enterprise . middleware >>>> >>>> >>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>>> http://lk.linkedin.com/in/suhothayan >>>> <http://lk.linkedin.com/in/suhothayan>* >>>> >>> >>> >>> >>> -- >>> Best Regards >>> Isuru Ranawaka >>> M: +94714629880 >>> Blog : http://isurur.blogspot.com/ >>> >> >> >> >> -- >> >> *S. Suhothayan* >> Technical Lead & Team Lead of WSO2 Complex Event Processor >> *WSO2 Inc. *http://wso2.com >> * <http://wso2.com/>* >> lean . enterprise . middleware >> >> >> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>* >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *Ruwan Abeykoon* > *Architect,* > *WSO2, Inc. http://wso2.com <http://wso2.com/> * > *lean.enterprise.middleware.* > > email: [email protected] > -- Best Regards Isuru Ranawaka M: +94714629880 Blog : http://isurur.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
