I should clarify my post. My messages aren't backing up in those three queues at the same time, the problem has moved each time I've changed the actor's queuing behavior. The first time I examined a heap dump all the objects were in the mailbox of the actor. So I changed the mailbox to use a bounded deque. The issue occurred again with the memory increasing so the next dump showed all the messages queuing up in the internalStash. So I changed the actor again to use 'persistAsync' instead of 'persist' and this time the dump shows all the messages queuing in the journalBatch.
So maybe my question should be: What is the message flow from the journalBatch into the Cassandra message table? Is the same dispatcher thread that processes messages used to move the messages to the Cassandra plug-in so if the 'persistAsync' or receiveCommand handler takes too much time processing the message we end up backing up messages in queue? As my other message said, just trying to figure out where to look next in the chain since this only happens under load. Regards, Richard On Sunday, January 15, 2017 at 6:56:42 PM UTC-8, Richard Ney wrote: > > Added to this based on my latest test runs. At this time I've based on > code changes witht he most recent being the change from 'persist' to > 'persistAsync' I've now observed my message backing up in three different > queues > > mailbox --> internalStash --> journalBatch > > The attached bitmap shows the chain. Is this a possible but with the > Cassandra journal plug in or maybe an issue with my custom serializers? Any > tips on how to chase this issue? > > -Richard > > On Friday, January 13, 2017 at 12:51:10 AM UTC-8, Richard Ney wrote: >> >> First a thank you to people answering questions. >> >> Against first the environment >> >> Akka 2.4.11 >> Scala 2.11.8 >> Cassandra 3.9 >> >> I've been chasing an issue for days where one of my persistent actors >> appears to stop processing message to the point where the JVM memory grows >> to 8GB before it gets an out of memory crash. Upon analyzing the dump I'm >> finding the internal stash is overflowing with billions of messages queued. >> I've done several 'jcmd <pid> Thread.print' commands to try and see if the >> threads are blocking in my code. The dumps are revealing nothing and I >> can't find a single one that's actually executing my code. I examined all >> the akka.actor.default-dispatcher threads which appear to be in a waiting >> state: >> >> "manhattan-akka.actor.default-dispatcher-43" #89 prio=5 os_prio=0 >> tid=0x00007fa4f0007800 nid=0x97 waiting on condition [0x00007fa5a435b000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00000005c0fbdf80> (a >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) >> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> And the Cassandra plug-in threads are also waiting: >> >> "manhattan-cassandra-plugin-default-dispatcher-49" #95 prio=5 os_prio=0 >> tid=0x00007fa504049000 nid=0x9d waiting on condition [0x00007fa52f5fa000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00000005c1b65640> (a >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) >> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> Trying to figure out if this a bug where a journal write is never >> returning which is all I can think of at this time. I've now set a limit on >> the internal stash but that will only prevent the OOM exception but my >> actor will still stop processing data. >> >> Any pearls of wisdom are appreciated. >> >> -Richard >> > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
