While this is sort of speculative, it looks like you have introduced a bottleneck. Maybe you can use a router for your main actor, resulting in parallelizing entry work (assuming you have multiple cores and enough threads)?
Heiko On 23 Aug 2014, at 02:37, Paul Cleary <[email protected]> wrote: > I have an actor system that is setup using all of the akka defaults. > > Nothing in the system has a lot of latency, we are talking about sub > millisecond (20 microsecond) latencies. > > All traffic flows through a single actor to downstream actors. This "main" > actor just forwards the message onto children using a lookup. > > I am doing load testing, and have seen the same trend both on my local, as > well as in a server environment. The "main" actor keeps up superbly up until > a certain point. > > I get up to about 1500 requests per second in my load test, which generates a > boat load (10s of thousands) of messages against my single actor. > > At that point, all of a sudden the mailbox size gets enormous on my main > actor. > > None of the downstream actors are doing anything, so it isn't like some roque > actor is chewing up threads. > > The main actor IS making progress, albeit very very very slowly. > > When I did a thread dump, almost all of the dispatcher threads were "waiting" > (i.e., parked not doing anything). > > The mailbox size grows into the millions during the test. The main actor > never really catches up, just putt putts along, it doesn't get faster. > > A few other notes: > 1. I did not max out memory or hit stop the world GC at any point during the > test. > 2. The CPU is very low on the machine, under 50% > 3. There are no hardcore errors on the main actor, he doesn't restart or > anything like that > 4. I am using Akka 2.2.3 > > So, my questions are: > 1. If the threads aren't busy (they are waiting), why isn't my main actor > making faster progress? Why are the other threads waiting seemingly not > doing anything? > > 2. What might cause the system to seemingly "shut down"? > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
