Hi Paul, Curious to know which monitoring tool you employed?
-Prakhyat M M On Saturday, 23 August 2014 19:32:05 UTC+5:30, Paul Cleary wrote: > > Thanks Ronald and Heiko. Turns out, the code that was doing the > monitoring \was the culprit. > > I removed my monitoring, and everything took off from there. I am > considering this issue closed, I appreciate your help. > > > On Saturday, August 23, 2014 8:09:25 AM UTC-4, rkuhn wrote: >> >> >> 23 aug 2014 kl. 02:37 skrev Paul Cleary <[email protected]>: >> >> I have an actor system that is setup using all of the akka defaults. >> >> Nothing in the system has a lot of latency, we are talking about sub >> millisecond (20 microsecond) latencies. >> >> All traffic flows through a single actor to downstream actors. This >> "main" actor just forwards the message onto children using a lookup. >> >> I am doing load testing, and have seen the same trend both on my local, >> as well as in a server environment. The "main" actor keeps up superbly up >> until a certain point. >> >> I get up to about 1500 requests per second in my load test, which >> generates a boat load (10s of thousands) of messages against my single >> actor. >> >> At that point, all of a sudden the mailbox size gets enormous on my main >> actor. >> >> None of the downstream actors are doing anything, so it isn't like some >> roque actor is chewing up threads. >> >> The main actor IS making progress, albeit very very very slowly. >> >> When I did a thread dump, almost all of the dispatcher threads were >> "waiting" (i.e., parked not doing anything). >> >> >> The interesting question (without which everything else is pure >> speculation) is: what were the active threads doing? >> >> Regards, >> >> Roland >> >> >> The mailbox size grows into the millions during the test. The main actor >> never really catches up, just putt putts along, it doesn't get faster. >> >> A few other notes: >> 1. I did not max out memory or hit stop the world GC at any point during >> the test. >> 2. The CPU is very low on the machine, under 50% >> 3. There are no hardcore errors on the main actor, he doesn't restart or >> anything like that >> 4. I am using Akka 2.2.3 >> >> So, my questions are: >> 1. If the threads aren't busy (they are waiting), why isn't my main actor >> making faster progress? Why are the other threads waiting seemingly not >> doing anything? >> >> 2. What might cause the system to seemingly "shut down"? >> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> *Dr. Roland Kuhn* >> *Akka Tech Lead* >> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >> twitter: @rolandkuhn >> <http://twitter.com/#!/rolandkuhn> >> >> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
