Hi, I was wondering if there were any strategies or published code for detecting and dealing with thread pool starvation in Akka apps, to save me from duplicating work?
I've seen https://groups.google.com/forum/#!topic/akka-user/JrDWOPeOFy8 but it doesn't directly address detecting this issue when it occurs accidentally. We have a high throughput Akka app, and we have occasionally encountered problems whereby a bug or a misused library has started blocking Actor threads. At high enough loads this will get to the point that all the worker threads in the thread pool are blocked and no Futures or callbacks get serviced until the next slow operation completes. This pauses everything, including logging and network i/o, and can be difficult to spot. For example, when this happened in the past we saw only odd network errors and very slow response times, but no logging relating to the thread starvation itself, and no obvious causes in the logs. I was thinking of adding a sentinel thread to periodically wake up and check on the queue size of the thread pool, and to log a big error with a stack dump if there were no free threads. Is there any published code which I can use to do this, or should I write my own? Is there a better strategy? I know that IntelliJ has a sentinel thread which watches the UI thread for long pauses and automatically logs a stack dump in that case (see "Automatic thread dumps" at https://intellij-support.jetbrains.com/entries/23348667-Getting-a-thread-dump-when-IDE-hangs-and-doesn-t-respond ) Is there anything similar that already exists for Akka? (If I do write my own, would it be useful for me to try to create a PR to add this to Akka itself? Should I get a design “signed off” somewhere first to maximise the chances of it being accepted, if so?) Thanks, Rich -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
