One more thing to add to this, in case it is relevant. I see multiple of these messages in the log:
[akka://ClusterSystem/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator] stopped First, why is it stopping (or why does it stop, in general), and secondly, is it significant that the url prefix is akka://ClusterSystem/ rather than akka.tcp://[email protected]:2552/ And second, I assume it makes no difference that I’m using the same akka-persistence journal/snapshot store as I used when the app was binding to 127.0.0.1:2552. I get tons of log messages indicating that receiveRecover is not happy trying to recover shards associated with akka.tcp://[email protected]:2552/system/sharing/ExperimentInstance#-396422686 <akka.tcp://[email protected]:2552/system/sharing/ExperimentInstance#-396422686>. I’m assuming this is expected and that akka persistence should be able to deal with this case. It should fail to recover these and then carry on with new persistence events that are targeted to the new ClusterSystem on the new IP address. Examples of the rejected receiveRecover messages that I’m seeing are: [akka.tcp://[email protected]:2552/system/cassandra-journal] Starting message scan from 1 [DEBUG] [03/19/2016 02:09:02.712] [ClusterSystem-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:2552/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator] receiveRecover ShardRegionRegistered(Actor[akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance#-396422686]) — Eric > On Mar 18, 2016, at 6:54 PM, Eric Swenson <[email protected]> wrote: > > I’ve been unsuccessful in trying to get an akka-cluster application that > works fine with one instance to work when there are multiple members of the > clusters. A bit of background is in order: > > 1) the application is an akka-cluster-sharing application > 2) it runs in a docker container > 3) the cluster is comprised of multiple docker hosts, each running the akka > application > 4) the error I’m getting is this: > > [WARN] [03/19/2016 01:39:18.086] > [ClusterSystem-akka.actor.default-dispatcher-3] > [akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance > <akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance>] > Trying to register to coordinator at > [Some(ActorSelection[Anchor(akka://ClusterSystem/ <akka://ClusterSystem/>), > Path(/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator)])], > but no acknowledgement. Total [1] buffered messages. > [WARN] [03/19/2016 01:39:20.086] > [ClusterSystem-akka.actor.default-dispatcher-3] > [akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance > <akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance>] > Trying to register to coordinator at > [Some(ActorSelection[Anchor(akka://ClusterSystem/ <akka://ClusterSystem/>), > Path(/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator)])], > but no acknowledgement. Total [1] buffered messages. > > 5) the warning message is logged repeatedly and the cluster never initializes. > 6) I’ve set the following config parameters: > akka.remote.netty.tcp.hostname: to the actual host ip address (the one > accessible from all the other docker hosts) > akka.remote.netty.tcp.bind-hostname: to 0.0.0.0 (so that it binds on > the docker0 interface, on the ip address of the container) > akka.remote.netty.tcp.port: 2552 > akka.remote.netty.tcp.bind-port:2552 > 7) when I start the container, I map port 2552 on the host to port 2552 on > the container. > 8) from the host, I’m able to do a “telnet <ip-address-of-host> 2552” so I > should be taking to the akka-remoting handler. > 9) I’m setting the akka..cluster.seed-nodes to a list of one element: > akka.tcp://ClusterSystem@<ip-address-of-host>:2552 > <akka.tcp://ClusterSystem@<ip-address-of-host>:2552>. I’m doing that > because, as far as I know, this seed-node list is advertised to all the other > members of the cluster (currently just the one) and must be accessible to > them all. The Docker container ip address (seen by the docker container) is > on the private docker0 interface, which is not accessible from the outside > (from outside the host). > > Now, before I tried all this, I set seed-nodes to a list of a single > aka.tcp://[email protected]:2552 > <aka.tcp://[email protected]:2552> entry, and left > akka.remote.netty.tcp.hostname, bind-hostname, port, and bind-port to their > default values. In this configuration, the cluster comes up perfectly fine > and the application (with an akka-http frontend and a cluster sharing > backend) works perfectly. In fact, when I use two seed nodes of the same > form, but different ports on the same local 127.0.0.1 host, and two instances > of the app (each binding to the different ports), the app works fine too. > The cluster comes up find and each node finds the other. > > But in a real environment, the multiple instances will be on different nodes > (diffferent ip addresses), and in my case, as docker containers on different > docker hosts. So clearly, the seed nodes must have externally accessible ip > addresses. > > Can anyone shed any light on what might be going wrong? > > How might I debug this? Thank. — Eric > > > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
