Hi, It's me again. After rewriting the full app to make it fully non-blocking (with Slick 3, a non-blocking Redis client, a non-blocking ElasticSearch client, etc.), everything runs much better... but not perfectly. The app still encounters starvation problems. I don't see anything strange in the logs just before a problem occurs. Response times are very short and CPU use very small. I don't notice the formation of the bottleneck that will made the app unusable. I guess that non-blocking code using Futures and work stealing by the fork/join framework are here not sufficient to avoid this kind of problems. I've also tried bigger instances, or a larger cluster, but I always end up with the same starvation problems...
So, I'm here to ask for help about what I should do to limit this kind of problem outside of the code (which is already, again, fully non-blocking). Here are my questions : - Can you confirm this situation is possible even with a fully non-blocking app ? I want to be sure I have done everything that needs to be done. - Would another type of dispatcher, like "thread-pool-executor", be preferable here to handle this kind of problem ? - What about the parameters *idle-timeout *or *backlog *? Maybe, without fixing the problems, they would at least make the situation more acceptable, with limiting the disaster, make it shorter and make the app recoverable ? Thank you very much for your help and sorry if some of my questions are naive or already answered somewhere... Manuel Le samedi 25 mars 2017 23:58:52 UTC+1, Manuel a écrit : > > Yes, but for me this solution would be more complex, I currently do not > have a specific actor for database IO. Upgrading to slick 3 seems more > logical to me ! > > No problem for the link, I think I've already read these docs ;) > > > > 2017-03-25 23:54 GMT+01:00 Konrad Malawski <[email protected]> > : > >> That's one way, yeah - slick provides a nice non blocking API and handles >> the blocking internally. >> The other way is to isolate the blocking in actors thanks to configuring >> custom dedicated threadpool based dispatchers - that's what we show in the >> docs. >> >> Again, sorry for no link, no fun to look for it a on a phone at friends >> place :) >> >> -- >> Konrad Malawski >> >> On 25 March 2017 at 23:52:06, Manuel Darcemont ( >> [email protected]) wrote: >> >>> Ok, that's what I thought, I only wanted to be sure about it. So I guess >>> integrating Slick 3 to benefit from it's non-blocking API would be a good >>> first step. >>> >>> Thank you ! >>> >>> 2017-03-25 23:25 GMT+01:00 Konrad Malawski < >>> [email protected]>: >>> >>>> Avoiding blocking (or dealing with it if you have to) in Akka is well >>>> documented, >>>> Please google for "akka http blocking" and you'll get to the right part >>>> of the docs. >>>> We also have docs search which will get you to the same, right docs. >>>> >>>> Not near a laptop to get you the link, please find it using that above >>>> query. >>>> >>>> -- >>>> Konrad Malawski >>>> >>>> On 25 March 2017 at 22:31:48, Manuel ([email protected]) >>>> wrote: >>>> >>>>> up vote >>>>> down votefavorite >>>>> <http://stackoverflow.com/questions/43022174/how-to-avoid-saturation-in-akka-http-latency-spikes#> >>>>> >>>>> >>>>> I have a akka-http (Scala) API server that serves data to a NodeJS >>>>> server. In moments after startup, everything works fine, everything is >>>>> fast. Latency is low. But suddenly, latency increases fastly. The API no >>>>> longer responds, and the website becomes unusable. >>>>> >>>>> [image: enter image description here] >>>>> <https://i.stack.imgur.com/uoy6o.png> >>>>> >>>>> The strange thing is that the traffic and the requests count remain >>>>> stable. Latency spikes seem decorrelated from them. >>>>> >>>>> I guess this saturation is achieved with the blocking of all the >>>>> threads in akka thread pool. Unfortunately, my Akka dispatcher is >>>>> blocking, >>>>> because I'm doing a lot of SQL queries (in MySQL) and because I'm not >>>>> using >>>>> a reactive library. I'm using Slick 2, which is, contrary to Slick 3, >>>>> blocking-only. >>>>> >>>>> So, my question is, how to avoid this sort of bottling ? How to keep a >>>>> latency proportional with the traffic ? Is there a way to evict the >>>>> requests that cause the saturation in order to prevent them to compromise >>>>> everything ? >>>>> >>>>> Thank you ! >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>> > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
