On Wed, Jun 25, 2014 at 2:02 PM, Cagri Yalcin <[email protected]> wrote:
> Ok, there is a negative bias against the people asking questions here :)) > > Lol, not at all, we just like for things to be correct :) > Joking apart, we definitely check Async REST+Akka asap. However, we also > have freedom to have some internal plans on what to do next, right? > Frankly, I do not understand the purpose of your last questions. > > In the end, if we assume Sync REST + Akka is a bottleneck and verify this > via Async REST, this is a valuable info for now, as we generally assumed > blocking bottleneck of Akka occurs mainly if any blocking operation is > inside actors. > (As you remember, our awaiting connector distributing requests from REST > Handler is non-actor.) > But we still do not know what your current bottleneck is. If your blocking connector (sync rest) is the bottleneck, then there's little that Akka can do to make it faster, the risk is that Akka makes the problem worse since it puts additional pressure on the bottleneck. > > Thanks anyway for your informative help. > I'm still extremely curious as to what your bottleneck is. Have you benched the logic separately from the REST layer? > > +"Agreed, which is why it is important to measure the right thing, right > now you may as well be benchmarking the scheduling artifacts of doing > blocking—which is something that Akka strongly discourages." > -"Assume nothing. Trust people" > > Regards, > > Cagri > > > > On Wednesday, June 25, 2014 12:42:18 PM UTC+2, √ wrote: > >> >> >> >> On Wed, Jun 25, 2014 at 12:14 PM, Cagri Yalcin <[email protected]> >> wrote: >> >>> Hi Again, >>> >>> First of all, thanks for Sync REST warning, this was sth that we are >>> always suspicious.. We will verify Async REST+Akka performance as soon as >>> we have time for that. >>> >> >> Performance benchmarking wasn't important? >> >> >>> >>> The reason why I said "client limitation" is of course not technical: >>> for now we do not want to spend time for JMeter conf for Async REST client. >>> >> >> I'm sorry, but does this mean that making accurate performance testing is >> not important? If it isn't worth spending the time? >> >> >>> >>> Keeping in mind your suggestions, we are sailing for Akka persistance >>> now. >>> >>> Regards, >>> >>> ps: Personally, I am aware of scalability power of Akka. However, as you >>> might know, if you really wanna "sell" a new library to experienced, >>> "lock-based" coders, you cannot tolerate this much difference on response >>> time. >>> >> >> Agreed, which is why it is important to measure the right thing, right >> now you may as well be benchmarking the scheduling artifacts of doing >> blocking—which is something that Akka strongly discourages. >> >> >>> >>> >>> Cagri >>> >>> >>> On Tuesday, June 24, 2014 3:04:00 PM UTC+2, √ wrote: >>> >>>> >>>> >>>> >>>> On Tue, Jun 24, 2014 at 2:57 PM, Cagri Yalcin <[email protected]> >>>> wrote: >>>> >>>>> Some JVM params: >>>>> -Xms128m -Xmx512m -XX:MaxPermSize=250m -XX:ReservedCodeCacheSize=64m >>>>> -ea -Dsun.io.useCanonCaches=false -Djava.net.preferIPv4Stack=true >>>>> -XX:+UseCodeCacheFlushing -XX:+UseConcMarkSweepGC >>>>> -XX:SoftRefLRUPolicyMSPerMB=50 >>>>> >>>>> >>>> What happens when you tweak xmx? >>>> >>>> >>>>> Based on YourKit, only CPU usage looks slightly worse than lock-based >>>>> impl. >>>>> >>>> >>>> What does "worse" mean here? Whats the ratio of system time vs >>>> usertime, how many context switches do you have per second? >>>> >>>> >>>>> In terms of thread creation/occupation, we already optimized it via >>>>> parallelism-max >>>>> and verified the positive effect on YourKit. >>>>> >>>>> >>>> Increasing parallelism-max what difference did it make? >>>> >>>> >>>>> I do not think we have an obvious bottleneck from Akka details >>>>> perspective. However, despite trying a lot of custom confs, observing >>>>> still >>>>> 2 times more response time comparing classical impls definitely is a >>>>> show-stopper.. >>>>> Till now, Akka outperforms very slightly only in terms of throughput. >>>>> >>>>> ps: Sync REST choice is mainly due to client limitation. >>>>> >>>> >>>> How does clients come in to this picture at all? (It's completely >>>> possible to have an async server and sync clients.) >>>> >>>> My educated quess is that your blocking is causing problems with thread >>>> wakeup lag and constant thread creation/destruction due to managed >>>> blocking. I.e. the cause is the choice of sync REST (given that I haven't >>>> seen the actual Akka code) >>>> >>>> Also, so far we have mainly talked about performance, what about the >>>> scalability? >>>> >>>> >>>>> >>>>> Thanks. >>>>> >>>>> Regards, >>>>> >>>>> Cagri >>>>> >>>>> >>>>> On Tuesday, June 24, 2014 2:12:28 PM UTC+2, √ wrote: >>>>> >>>>>> And if you do not block, what is your performance then? >>>>>> >>>>>> It sounds like you need to bring out a profiler to see what your >>>>>> bottleneck is (I've seen JMeter be the bottleneck as well). >>>>>> >>>>>> >>>>>> On Tue, Jun 24, 2014 at 1:50 PM, Cagri Yalcin <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> And only blocking operation is on AkkaConnector (non-actor, just >>>>>>> concrete class) side (Await on aggregated futures OR Await on single >>>>>>> future >>>>>>> for ResultCollectorActor response (where requests are sent via broadcast >>>>>>> router group to learning actors and result messages are kept in buffer >>>>>>> and >>>>>>> aggregated in ResultCollectorActor). >>>>>>> >>>>>>> >>>>>>> On Tuesday, June 24, 2014 1:43:26 PM UTC+2, Cagri Yalcin wrote: >>>>>>>> >>>>>>>> Sync/Async REST terminology is based on https://jersey.java.net/doc >>>>>>>> umentation/latest/async.html >>>>>>>> Example akka + Async REST impl: https://github.com/pofal >>>>>>>> lon/jersey2-akka-java >>>>>>>> >>>>>>>> Note that currently we do not use this. Just sync Jersey REST on >>>>>>>> Java EE 7. >>>>>>>> >>>>>>>> OS: Windows 7 Enterprise >>>>>>>> >>>>>>>> JVM: >>>>>>>> java version "1.7.0_45" >>>>>>>> Java(TM) SE Runtime Environment (build 1.7.0_45-b18) >>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) >>>>>>>> >>>>>>>> We are using 8 core machine and had already tried to set >>>>>>>> parallelism-max to 8. No significant change in mean response time. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Cagri >>>>>>>> >>>>>>>> On Tuesday, June 24, 2014 12:00:17 PM UTC+2, √ wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 24, 2014 at 11:55 AM, Cagri Yalcin < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> First of all, from your questions shall I assume the response >>>>>>>>>> time should not differ that much comparing to lock-based impls? We >>>>>>>>>> really >>>>>>>>>> need some best practices for Akka REST service actors, or >>>>>>>>>> at least some vision. >>>>>>>>>> >>>>>>>>> >>>>>>>>> "Assume nothing. Bench everything." >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> JMeter was running on the same machine. For JVM, in all >>>>>>>>>> experiments we used default maven-t7-plugin settings + CMS garbage >>>>>>>>>> collector. >>>>>>>>>> >>>>>>>>> >>>>>>>>> What web framework are you using? What does "Sync REST" mean? >>>>>>>>> What _specific_ JVM settings, JVM version and operating system are >>>>>>>>> you using? >>>>>>>>> >>>>>>>>> >>>>>>>>>> Below is an Akka configuration that gives one of the best >>>>>>>>>> performances till now. >>>>>>>>>> >>>>>>>>>> default-dispatcher { >>>>>>>>>> >>>>>>>>>> fork-join-executor { >>>>>>>>>> parallelism-max = 100 >>>>>>>>>> >>>>>>>>> >>>>>>>>> This setting makes no sense (to me, unless you have more than 100 >>>>>>>>> cores on your machine) >>>>>>>>> >>>>>>>>> >>>>>>>>>> } >>>>>>>>>> throughput = 100 >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> learning-core-dispatcher { >>>>>>>>>> >>>>>>>>>> type = Dispatcher >>>>>>>>>> >>>>>>>>>> executor = "fork-join-executor" >>>>>>>>>> >>>>>>>>>> fork-join-executor { >>>>>>>>>> >>>>>>>>>> parallelism-min = 8 >>>>>>>>>> >>>>>>>>>> parallelism-factor = 3.0 >>>>>>>>>> >>>>>>>>>> parallelism-max = 40 >>>>>>>>>> >>>>>>>>> >>>>>>>>> Do you have > 40 cores on your machine? >>>>>>>>> >>>>>>>>> >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> throughput = 100 >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> default-mailbox { >>>>>>>>>> mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Are you using _any_ blocking operations whatsoever? >>>>>>>>> >>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Cagri >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tuesday, June 24, 2014 11:33:10 AM UTC+2, √ wrote: >>>>>>>>>> >>>>>>>>>>> Also, what JVM parameters did you use? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Jun 24, 2014 at 11:32 AM, √iktor Ҡlang < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> What is your bottleneck, what config did you use and did you >>>>>>>>>>>> run JMeter on the same machine or on a different machine? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jun 24, 2014 at 11:30 AM, Cagri Yalcin < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> We've implemented an online learning/recommendation system >>>>>>>>>>>>> adopting 2 approaches: >>>>>>>>>>>>> 1) Sync REST and learning components shielded with >>>>>>>>>>>>> ReentrantReadWriteLock's >>>>>>>>>>>>> 2) Sync REST and learning components deployed in Akka actors >>>>>>>>>>>>> >>>>>>>>>>>>> Throughout our stress tests using JMeter and several above >>>>>>>>>>>>> 500K real data, we analysed a huge set of configurations for Akka >>>>>>>>>>>>> from >>>>>>>>>>>>> connector impl(component between akka actors and REST >>>>>>>>>>>>> service), actor architecture, dispatcher and mailbox settings >>>>>>>>>>>>> perspectives. >>>>>>>>>>>>> Till now, best performance obtained via harnessing dedicated >>>>>>>>>>>>> distinct dispatchers with some (sub)optimized parameters for >>>>>>>>>>>>> connector >>>>>>>>>>>>> (fully future-based and/or router+actor+future/based) >>>>>>>>>>>>> and core learning actors. In addition to this, using >>>>>>>>>>>>> SingleConsumerOnlyUnboundedMailbox as default mailbox type >>>>>>>>>>>>> brings some improvements. >>>>>>>>>>>>> >>>>>>>>>>>>> After extensive trials, comparing to >>>>>>>>>>>>> ReentrantReadWriteLock-based approach, what we observe is >>>>>>>>>>>>> recommendation >>>>>>>>>>>>> service response time is always around 2 times >>>>>>>>>>>>> more in Akka-based impls, although Akka-based impls might give >>>>>>>>>>>>> slightly better throughput in some cases. >>>>>>>>>>>>> >>>>>>>>>>>>> My questions: >>>>>>>>>>>>> >>>>>>>>>>>>> 1) Considering response time, is Akka latency is inevitable >>>>>>>>>>>>> comparing to classical lock-based impls on top of a REST service? >>>>>>>>>>>>> Any >>>>>>>>>>>>> general suggestion/guideline to improve >>>>>>>>>>>>> response time in Akka REST service actors? >>>>>>>>>>>>> >>>>>>>>>>>>> 2) Shall we "really" expect a better throughput if we use >>>>>>>>>>>>> Async REST? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks a lot. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Cagri Yalcin >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>>>>>>>> urrent/additional/faq.html >>>>>>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>>>>>>>> p/akka-user >>>>>>>>>>>>> --- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "Akka User List" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>>>>> >>>>>>>>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Cheers, >>>>>>>>>>>> √ >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Cheers, >>>>>>>>>>> √ >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>>>>> urrent/additional/faq.html >>>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>>>>> p/akka-user >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Akka User List" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Cheers, >>>>>>>>> √ >>>>>>>>> >>>>>>>> -- >>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>> urrent/additional/faq.html >>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>> p/akka-user >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Akka User List" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Cheers, >>>>>> √ >>>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>> urrent/additional/faq.html >>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>> p/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> √ >>>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Cheers, >> √ >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Cheers, √ -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
