Hi Ranawaka, I think we should finalize the decision on the default thread model of the transport and the engine. How does the poc implementation with multiple disruptors (dedicated disruptors for IO and CPU bound work) perform?
Thanks, Kasun On Sun, Apr 24, 2016 at 9:19 PM, Isuru Ranawaka <[email protected]> wrote: > Hi Ruwan, Suho > > Thanks for information provided will do the necessary tests as you > mentioned. > > thanks > IsuruR > > On Mon, Apr 25, 2016 at 9:45 AM, Ruwan Abeykoon <[email protected]> wrote: > >> Hi Isuru, >> I think the results are in favour of Disruptor model because, >> 1. Latency and throughput seems to be stable in disruptor towards higher >> load(at higher concurrency in the TPS and Latency graph) >> 2. Disruptor based model uses less number of active threads. In >> containerized environment, this is much more desirable as there will be >> less number of context switches (due to threads) at high load and OS >> can(usually) pin the active threads in the same core. >> >> I assume the above tests are done on single JVM. Can we run the same >> setup with a load balancer fronted and simulate a handful of containers. >> That will give us some realistic usage. The test server seems under >> utilized on the above tests. I feel that almost same results on both tests >> are due to the fact that the server is much more capable than the code >> demands. We need to utilize around 80% of CPU and above 60% of physical RAM >> taken to see any bottleneck. Further, we need to test One CPU bound >> disruptor, one IO bound disruptor when doing this test agains 5 CPU >> disruptor case. >> >> Can you please share the code which the tests were done. >> >> >> Cheers, >> Ruwan >> >> On Sun, Apr 24, 2016 at 1:10 AM, Sriskandarajah Suhothayan <[email protected] >> > wrote: >> >>> In CEP we have experienced latency issues chaining disrupter and don't >>> go for PhasedBackOff strategy as there is a hit for calculating time >>> and if you use spinning or yielding there you have to also correlate >>> then with the number of cores on the system which will be a problem when it >>> comes to deployment. >>> >>> Suho >>> >>> On Sat, Apr 23, 2016 at 7:18 PM, Isuru Ranawaka <[email protected]> wrote: >>> >>>> Hi Chanaka, >>>> >>>> I have run the test for header based routing scenario (Pure CPU) and >>>> there is small improvement on Disruptor. We can get clear performance >>>> difference if we send huge load such that requests are get queued at >>>> ThreadPool and will has huge contention for internal queues but for >>>> Disruptor it is locking free .when profiling this test I didn't see >>>> contention in the ThreadPool level as well. Actually advantage we got >>>> through Disruptor is due to cache optimization at disruptor level. Another >>>> problem is we have active Netty thread pool as well. So there is no >>>> probability that Disruptor threads are always get high priority. Due to >>>> that also we cannot get maximum out of Disruptor thread model. I will run >>>> long running test with significant amount of CPU bound work and see how it >>>> behaves. >>>> >>>> @Suho, >>>> >>>> CPU bound Disruptor wait strategy is Sleep waiting and IO bound >>>> Disruptor is Blocking Wait . Tested with PhasedBackOff with LiteLocking as >>>> well not much difference in TPS but observed latency was varied in higher >>>> amount for some requests. >>>> >>>> Number of Disruptors are configurable and I have used five CPU bound >>>> Disruptors and one IO bound DIsruptor for testing which gives maximum Perf >>>> according to testes we carried for Gateway. >>>> >>>> thanks >>>> >>>> >>>> On Fri, Apr 22, 2016 at 8:20 PM, Sriskandarajah Suhothayan < >>>> [email protected]> wrote: >>>> >>>>> Whats the waiting strategy that is being used in Disrupter? And is the >>>>> maximum of Disrupter in all configuration is 2 ? >>>>> >>>>> Suho >>>>> >>>>> >>>>> On Fri, Apr 22, 2016 at 8:01 PM, Chanaka Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Isuru, >>>>>> >>>>>> Have we tested on the pure passthrough scenarios? According to the >>>>>> results, we cannot see much performance difference in using disruptor >>>>>> over >>>>>> thread pool. Shall we do some testing around passthrough scenarios and >>>>>> verify? >>>>>> >>>>>> Cheers, >>>>>> Chanaka >>>>>> >>>>>> On Fri, Apr 22, 2016 at 12:54 PM, Isuru Ranawaka <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> We have working on decoupling engine thread model from transport >>>>>>> thread model.Previously carbon transport was included with the Disruptor >>>>>>> and according to the behaviour of MSF4J it is very hard to mapped custom >>>>>>> logics written using MSF4J to Disruptor thread model in order to gain >>>>>>> better performance. So we have moved Disruptor thread model to >>>>>>> carbon-gateway and carbon-transport is kept with Netty thread pool. >>>>>>> >>>>>>> According to current implementation carbon-transport will dispatch >>>>>>> events to registered message processor via Netty worker threads and >>>>>>> there >>>>>>> after it operates under engine level thread model. >>>>>>> >>>>>>> Following is the Thread model diagram for Integration Server. >>>>>>> >>>>>>> [image: gw_thread_model.png] >>>>>>> >>>>>>> Basically it works as follows >>>>>>> >>>>>>> - >>>>>>> >>>>>>> CPU bound mediators are working on CPU bound disruptor threads. >>>>>>> - >>>>>>> >>>>>>> IO bound mediators are working on IO bound disruptor threads. >>>>>>> - >>>>>>> >>>>>>> We assume custom mediators are written in bad way and may >>>>>>> contain blocking calls so those are executed using IO bound mediator. >>>>>>> - >>>>>>> >>>>>>> We have included ThreadPool implementation as well and can >>>>>>> switch between ThreadPool based model or Disruptor based model. >>>>>>> >>>>>>> This is for we haven’t yet finalized exact thread model and we keep >>>>>>> testing with different mediators how both are behaving according to >>>>>>> different parameters like TPS, memory, startup time ,latency , .etc >>>>>>> >>>>>>> >>>>>>> Following are some of the tests results we have conducted with both >>>>>>> thread model implementations for two main scenarios. >>>>>>> >>>>>>> Machine Details >>>>>>> >>>>>>> Server :- 32 core machine with 64 GB memory >>>>>>> >>>>>>> Back End Service :- Netty based Echo Service which has TPS around >>>>>>> 100000 >>>>>>> >>>>>>> Tested message size :- 4kb >>>>>>> >>>>>>> Server startup Time with Disruptor :- 1.34 s >>>>>>> >>>>>>> Server startup time without Disruptor :- 1.38 s >>>>>>> >>>>>>> Use case:- >>>>>>> >>>>>>> (CPU + IO) >>>>>>> >>>>>>> Header based routing with File Writing .One message path is writing >>>>>>> message to file and other one sends messages to Echo service and respond >>>>>>> back to client. >>>>>>> >>>>>>> TPS >>>>>>> >>>>>>> [image: image1.png] >>>>>>> >>>>>>> Latency >>>>>>> >>>>>>> [image: image2.png] >>>>>>> >>>>>>> Memory >>>>>>> >>>>>>> Disruptor >>>>>>> >>>>>>> [image: disruptor.png] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thread Pool >>>>>>> >>>>>>> [image: threadpoolMemory.png] >>>>>>> >>>>>>> Use case:- >>>>>>> >>>>>>> (CPU ) >>>>>>> >>>>>>> Header based routing , send messages to Echo service and respond >>>>>>> back to client. >>>>>>> >>>>>>> TPS >>>>>>> >>>>>>> [image: image3.png] >>>>>>> >>>>>>> Latency >>>>>>> >>>>>>> [image: image4.png] >>>>>>> >>>>>>> >>>>>>> For test results please look in to [1] >>>>>>> >>>>>>> [1] >>>>>>> https://docs.google.com/spreadsheets/d/1A2dxknP1xEJKBpl4ymbQD2Mt9kWywYCI-60j16JVLx0/edit#gid=0 >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> IsuruR >>>>>>> >>>>>>> -- >>>>>>> Best Regards >>>>>>> Isuru Ranawaka >>>>>>> M: +94714629880 >>>>>>> Blog : http://isurur.blogspot.com/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thank you and Best Regards, >>>>>> Chanaka Fernando >>>>>> Senior Technical Lead >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 773337238 >>>>>> Blog : http://soatutorials.blogspot.com >>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *S. Suhothayan* >>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>>> *WSO2 Inc. *http://wso2.com >>>>> * <http://wso2.com/>* >>>>> lean . enterprise . middleware >>>>> >>>>> >>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>>>> http://lk.linkedin.com/in/suhothayan >>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards >>>> Isuru Ranawaka >>>> M: +94714629880 >>>> Blog : http://isurur.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> >>> *S. Suhothayan* >>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>> *WSO2 Inc. *http://wso2.com >>> * <http://wso2.com/>* >>> lean . enterprise . middleware >>> >>> >>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>* >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> >> *Ruwan Abeykoon* >> *Architect,* >> *WSO2, Inc. http://wso2.com <http://wso2.com/> * >> *lean.enterprise.middleware.* >> >> email: [email protected] >> > > > > -- > Best Regards > Isuru Ranawaka > M: +94714629880 > Blog : http://isurur.blogspot.com/ > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Kasun Indrasiri Software Architect WSO2, Inc.; http://wso2.com lean.enterprise.middleware cell: +94 77 556 5206 Blog : http://kasunpanorama.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
