Hi Ryadh, On 3 April 2014 at 13:30:02, Ryadh khsib ([email protected]) wrote:
Hi Patrik, I did the upgrade to the version 2.3.0 and the joinSeedNodeProcess issue is fixed. For the current setup, 2 singletons and 1 router clusters are deployed inside the same Akka system. I presume this is causing the repeated calls the joinSeedNodeProcess actor. What do recommend for this setup? Should it be a unique cluster for every Akka system? We faced a new issue after the upgrade. The thread count on Tomcat increase due to 500 New I/O worker threads. Are you aware of any issue with the 2.3.0 that could cause this? It's very difficult to diagnose why you get so many Netty worker threads without having any information about the system you are running on. Do the machines have a large number of cores? Do you start multiple actor systems inside the same JVM? Have you done custom configuration of I/O dispatchers? B/ Many thanks! Ryadh The thread dump of I/O threads: Thread 10466: (state = IN_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=81 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - org.jboss.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame) - org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=409 (Compiled frame) - org.jboss.netty.channel.socket.nio.AbstractNioSelector.run() @bci=49, line=206 (Compiled frame) - org.jboss.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=90 (Interpreted frame) - org.jboss.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame) - org.jboss.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame) - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame) On Wednesday, April 2, 2014 11:23:40 AM UTC+1, Patrik Nordwall wrote: Hi Ryadh, I looked at the implementation and this is bug that can happen if you call joinSeedNodes repeatedly. I don't see a reason why you would need to do that, but it is supported usage and should be fixed. Ticket: https://www.assembla.com/spaces/akka/tickets/3973 Regards, Patrik On Sat, Mar 29, 2014 at 6:28 PM, Ryadh khsib <[email protected]> wrote: Hi Patrik, I have upgraded to the version 2.3.0 (the current version is the 2.2.0). The fix is going to production next week. Hopefully this will fix it and will stabilize the cluster setup in general. I will keep you informed ;) Thanks, Ryadh On Friday, March 28, 2014 7:55:43 PM UTC, Patrik Nordwall wrote: Ryadh, did that solve the problem? Otherwise I would like to see a ticket so that we can fix it. http://doc.akka.io/docs/akka/current/project/issue-tracking.html /Patrik On Mar 21, 2014, at 18:52, √iktor Ҡlang <[email protected]> wrote: Always upgrade to at least the latest bugfix release for the version of Akka you are using and see if that resolves your problem. Did that solve it? On Fri, Mar 21, 2014 at 6:37 PM, Ryadh khsib <[email protected]> wrote: Hi all, We are getting intermittently a "[actor name [joinSeedNodeProcess] is not unique!]" exception at the akka Cluster system start up. The production setup is a akka Singleton cluster deployed on 12 tomcats across 3 physical machines (4 nodes on every machine). Do you have any idea how we could fix this? Many thanks, Ryadh Exception trace: ERROR akka.cluster.ClusterCoreSupervisor 2014-03-21 08:59:28,092 [build=null] [System-akka.actor.default-dispatcher-8] - Cluster node [akka.tcp://System@host084:2551] crashed, [actor name [joinSeedNodeProcess] is not unique!] - shutting down... [exceptionClass=akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer, exceptionMethod=reserve, exceptionID=74a5d13f, ] akka.actor.InvalidActorNameException: actor name [joinSeedNodeProcess] is not unique! at akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.ActorCell.reserveChild(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.dungeon.Children$class.makeChild(Children.scala:186) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.dungeon.Children$class.actorOf(Children.scala:38) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.ActorCell.actorOf(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.cluster.ClusterCoreDaemon.joinSeedNodes(ClusterDaemon.scala:361) ~[akka-cluster_2.10-2.2.0.jar:na] at akka.cluster.ClusterCoreDaemon$$anonfun$tryingToJoin$1.applyOrElse(ClusterDaemon.scala:302) ~[akka-cluster_2.10-2.2.0.jar:na] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.ActorCell.invoke(ActorCell.scala:456) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.dispatch.Mailbox.run(Mailbox.scala:219) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10-2.2.0.jar:2.2.0] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.2.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.2.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.2.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.2.jar:na] ERROR akka.actor.OneForOneStrategy 2014-03-21 08:59:28,093 [build=null] [System-akka.actor.default-dispatcher-8] - actor name [joinSeedNodeProcess] is not unique! [exceptionClass=akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer, exceptionMethod=reserve, exceptionID=74a5d13f, ] akka.actor.InvalidActorNameException: actor name [joinSeedNodeProcess] is not unique! at akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.ActorCell.reserveChild(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.dungeon.Children$class.makeChild(Children.scala:186) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.dungeon.Children$class.actorOf(Children.scala:38) ~[akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.ActorCell.actorOf(ActorCell.scala:338) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.cluster.ClusterCoreDaemon.joinSeedNodes(ClusterDaemon.scala:361) ~[akka-cluster_2.10-2.2.0.jar:na] at akka.cluster.ClusterCoreDaemon$$anonfun$tryingToJoin$1.applyOrElse(ClusterDaemon.scala:302) ~[akka-cluster_2.10-2.2.0.jar:na] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.actor.ActorCell.invoke(ActorCell.scala:456) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.dispatch.Mailbox.run(Mailbox.scala:219) [akka-actor_2.10-2.2.0.jar:2.2.0] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10-2.2.0.jar:2.2.0] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.2.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.2.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.2.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.2.jar:na] -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ ——————— Viktor Klang Chief Architect - Typesafe Twitter: @viktorklang -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Typesafe - Reactive apps on the JVM Twitter: @patriknw JOIN US. REGISTER TODAY! Scala Days June 16th-18th, Berlin -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Björn Antonsson Typesafe – Reactive Apps on the JVM twitter: @bantonsson JOIN US. REGISTER TODAY! Scala Days June 16th-18th, Berlin -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
