On Tue, Jun 24, 2014 at 2:57 PM, Cagri Yalcin <[email protected]> wrote:
> Some JVM params: > -Xms128m -Xmx512m -XX:MaxPermSize=250m -XX:ReservedCodeCacheSize=64m -ea > -Dsun.io.useCanonCaches=false -Djava.net.preferIPv4Stack=true > -XX:+UseCodeCacheFlushing -XX:+UseConcMarkSweepGC > -XX:SoftRefLRUPolicyMSPerMB=50 > > What happens when you tweak xmx? > Based on YourKit, only CPU usage looks slightly worse than lock-based impl. > What does "worse" mean here? Whats the ratio of system time vs usertime, how many context switches do you have per second? > In terms of thread creation/occupation, we already optimized it via > parallelism-max > and verified the positive effect on YourKit. > > Increasing parallelism-max what difference did it make? > I do not think we have an obvious bottleneck from Akka details > perspective. However, despite trying a lot of custom confs, observing still > 2 times more response time comparing classical impls definitely is a > show-stopper.. > Till now, Akka outperforms very slightly only in terms of throughput. > > ps: Sync REST choice is mainly due to client limitation. > How does clients come in to this picture at all? (It's completely possible to have an async server and sync clients.) My educated quess is that your blocking is causing problems with thread wakeup lag and constant thread creation/destruction due to managed blocking. I.e. the cause is the choice of sync REST (given that I haven't seen the actual Akka code) Also, so far we have mainly talked about performance, what about the scalability? > > Thanks. > > Regards, > > Cagri > > > On Tuesday, June 24, 2014 2:12:28 PM UTC+2, √ wrote: > >> And if you do not block, what is your performance then? >> >> It sounds like you need to bring out a profiler to see what your >> bottleneck is (I've seen JMeter be the bottleneck as well). >> >> >> On Tue, Jun 24, 2014 at 1:50 PM, Cagri Yalcin <[email protected]> >> wrote: >> >>> And only blocking operation is on AkkaConnector (non-actor, just >>> concrete class) side (Await on aggregated futures OR Await on single future >>> for ResultCollectorActor response (where requests are sent via broadcast >>> router group to learning actors and result messages are kept in buffer and >>> aggregated in ResultCollectorActor). >>> >>> >>> On Tuesday, June 24, 2014 1:43:26 PM UTC+2, Cagri Yalcin wrote: >>>> >>>> Sync/Async REST terminology is based on https://jersey.java.net/doc >>>> umentation/latest/async.html >>>> Example akka + Async REST impl: https://github.com/pofal >>>> lon/jersey2-akka-java >>>> >>>> Note that currently we do not use this. Just sync Jersey REST on Java >>>> EE 7. >>>> >>>> OS: Windows 7 Enterprise >>>> >>>> JVM: >>>> java version "1.7.0_45" >>>> Java(TM) SE Runtime Environment (build 1.7.0_45-b18) >>>> Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) >>>> >>>> We are using 8 core machine and had already tried to set >>>> parallelism-max to 8. No significant change in mean response time. >>>> >>>> Regards, >>>> >>>> Cagri >>>> >>>> On Tuesday, June 24, 2014 12:00:17 PM UTC+2, √ wrote: >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Jun 24, 2014 at 11:55 AM, Cagri Yalcin <[email protected]> >>>>> wrote: >>>>> >>>>>> First of all, from your questions shall I assume the response time >>>>>> should not differ that much comparing to lock-based impls? We really need >>>>>> some best practices for Akka REST service actors, or >>>>>> at least some vision. >>>>>> >>>>> >>>>> "Assume nothing. Bench everything." >>>>> >>>>> >>>>>> >>>>>> JMeter was running on the same machine. For JVM, in all experiments >>>>>> we used default maven-t7-plugin settings + CMS garbage collector. >>>>>> >>>>> >>>>> What web framework are you using? What does "Sync REST" mean? >>>>> What _specific_ JVM settings, JVM version and operating system are you >>>>> using? >>>>> >>>>> >>>>>> Below is an Akka configuration that gives one of the best >>>>>> performances till now. >>>>>> >>>>>> default-dispatcher { >>>>>> >>>>>> fork-join-executor { >>>>>> parallelism-max = 100 >>>>>> >>>>> >>>>> This setting makes no sense (to me, unless you have more than 100 >>>>> cores on your machine) >>>>> >>>>> >>>>>> } >>>>>> throughput = 100 >>>>>> } >>>>>> >>>>>> learning-core-dispatcher { >>>>>> >>>>>> type = Dispatcher >>>>>> >>>>>> executor = "fork-join-executor" >>>>>> >>>>>> fork-join-executor { >>>>>> >>>>>> parallelism-min = 8 >>>>>> >>>>>> parallelism-factor = 3.0 >>>>>> >>>>>> parallelism-max = 40 >>>>>> >>>>> >>>>> Do you have > 40 cores on your machine? >>>>> >>>>> >>>>>> } >>>>>> >>>>>> throughput = 100 >>>>>> } >>>>>> >>>>>> default-mailbox { >>>>>> mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" >>>>>> } >>>>>> >>>>>> >>>>> Are you using _any_ blocking operations whatsoever? >>>>> >>>>> >>>>>> Regards, >>>>>> >>>>>> Cagri >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, June 24, 2014 11:33:10 AM UTC+2, √ wrote: >>>>>> >>>>>>> Also, what JVM parameters did you use? >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 24, 2014 at 11:32 AM, √iktor Ҡlang <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> What is your bottleneck, what config did you use and did you run >>>>>>>> JMeter on the same machine or on a different machine? >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jun 24, 2014 at 11:30 AM, Cagri Yalcin <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> We've implemented an online learning/recommendation system >>>>>>>>> adopting 2 approaches: >>>>>>>>> 1) Sync REST and learning components shielded with >>>>>>>>> ReentrantReadWriteLock's >>>>>>>>> 2) Sync REST and learning components deployed in Akka actors >>>>>>>>> >>>>>>>>> Throughout our stress tests using JMeter and several above 500K >>>>>>>>> real data, we analysed a huge set of configurations for Akka from >>>>>>>>> connector impl(component between akka actors and REST service), >>>>>>>>> actor architecture, dispatcher and mailbox settings perspectives. >>>>>>>>> Till now, best performance obtained via harnessing dedicated >>>>>>>>> distinct dispatchers with some (sub)optimized parameters for connector >>>>>>>>> (fully future-based and/or router+actor+future/based) >>>>>>>>> and core learning actors. In addition to this, using >>>>>>>>> SingleConsumerOnlyUnboundedMailbox as default mailbox type brings >>>>>>>>> some improvements. >>>>>>>>> >>>>>>>>> After extensive trials, comparing to ReentrantReadWriteLock-based >>>>>>>>> approach, what we observe is recommendation service response time is >>>>>>>>> always >>>>>>>>> around 2 times >>>>>>>>> more in Akka-based impls, although Akka-based impls might give >>>>>>>>> slightly better throughput in some cases. >>>>>>>>> >>>>>>>>> My questions: >>>>>>>>> >>>>>>>>> 1) Considering response time, is Akka latency is inevitable >>>>>>>>> comparing to classical lock-based impls on top of a REST service? Any >>>>>>>>> general suggestion/guideline to improve >>>>>>>>> response time in Akka REST service actors? >>>>>>>>> >>>>>>>>> 2) Shall we "really" expect a better throughput if we use Async >>>>>>>>> REST? >>>>>>>>> >>>>>>>>> Thanks a lot. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Cagri Yalcin >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>>>> urrent/additional/faq.html >>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>>>> p/akka-user >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Akka User List" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>> >>>>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Cheers, >>>>>>>> √ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Cheers, >>>>>>> √ >>>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>> urrent/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>> p/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> √ >>>>> >>>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Cheers, >> √ >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Cheers, √ -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
