Hi Everyone, I've been trying to set an Akka cluster with one master node and multiple workers. The workers are actor systems than run on Android emulators. As a start, I work with one worker (emulator). I verify that it successfully joins the cluster and start sending it messages, that are handled successfully. After some time (from 2-3 to 30-40 minutes), however, it disconnects from the cluster. Trying to figure out what causes the problem, I noticed that even if the worker is idle (no messages are sent), it disconnects from the cluster after some time.
In the Android logcat, the following message is displayed: [ClusterSystem-akka.remote.default-remote-dispatcher-5] > [akka.tcp://[email protected]:2553/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%4010.141.4.104%3A2551-0] > > Association with remote system [akka.tcp://[email protected]:2551] > has failed, address is now gated for [5000] ms. Reason is: []. and then: [ClusterSystem-cluster-dispatcher-15] [akka.tcp://[email protected]:2553/system/cluster/core/daemon] Cluster Node [akka.tcp://[email protected]:2553] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://[email protected]:2551, status = Up)] and eventually: [ClusterSystem-cluster-dispatcher-26] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2553] - Leader is auto-downing unreachable node [akka.tcp://[email protected]:2551] [ClusterSystem-cluster-dispatcher-26] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2553] - Marking unreachable node [akka.tcp://[email protected]:2551] as [Down] [ClusterSystem-cluster-dispatcher-27] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2553] - Leader is removing unreachable node [akka.tcp://[email protected]:2551] After I subscribed to AssociationErrorEvent, I was able to get more details: AssociationErrorEvent has occurred: AssociationError > [akka.tcp://[email protected]:2553] -> > [akka.tcp://[email protected]:2551]: Error [] [ > akka.remote.EndpointException: > at > com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770) > at > akka.remote.ContainerFormats$Selection$Builder.build(ContainerFormats.java:1513) > at > akka.remote.ContainerFormats$SelectionEnvelope$Builder.addPattern(ContainerFormats.java:931) > at > akka.remote.serialization.MessageContainerSerializer$$anonfun$serializeSelection$1.apply(MessageContainerSerializer.scala:45) > at > akka.remote.serialization.MessageContainerSerializer$$anonfun$serializeSelection$1.apply(MessageContainerSerializer.scala:43) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > akka.remote.serialization.MessageContainerSerializer.serializeSelection(MessageContainerSerializer.scala:43) > at > akka.remote.serialization.MessageContainerSerializer.toBinary(MessageContainerSerializer.scala:25) > at akka.remote.MessageSerializer$.serialize(MessageSerializer.scala:36) > at > akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:842) > at > akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:842) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at akka.remote.EndpointWriter.serializeMessage(Endpoint.scala:841) > at akka.remote.EndpointWriter.writeSend(Endpoint.scala:742) > at akka.remote.EndpointWriter$$anonfun$2.applyOrElse(Endpoint.scala:717) > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:410) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) > at akka.dispatch.Mailbox.run(Mailbox.scala:221) > at akka.dispatch.Mailbox.exec(Mailbox.scala:231) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: com.google.protobuf.UninitializedMessageException: Message > missing required fields: > ... 30 more > ] > akka.remote.EndpointException: > at > com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770) > at > akka.remote.ContainerFormats$Selection$Builder.build(ContainerFormats.java:1513) > at > akka.remote.ContainerFormats$SelectionEnvelope$Builder.addPattern(ContainerFormats.java:931) > at > akka.remote.serialization.MessageContainerSerializer$$anonfun$serializeSelection$1.apply(MessageContainerSerializer.scala:45) > at > akka.remote.serialization.MessageContainerSerializer$$anonfun$serializeSelection$1.apply(MessageContainerSerializer.scala:43) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > akka.remote.serialization.MessageContainerSerializer.serializeSelection(MessageContainerSerializer.scala:43) > at > akka.remote.serialization.MessageContainerSerializer.toBinary(MessageContainerSerializer.scala:25) > at akka.remote.MessageSerializer$.serialize(MessageSerializer.scala:36) > at > akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:842) At first I though that there's a serialization problem with one of the messages that are sent to or from the worker. However, the problem repeats itself even when there are no messages sent to the worker at all. If I restart the worker, it re-joins the cluster and everything is back to normal again (until the next disconnection event) - so the problem isn't permanent. I'm using Akka 2.3.9 on both master and worker. What could be causing the problem? Could it be Android related? Thanks. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
