Hi Eric, I have no experience with Docker at all, but it does feel wrong (unless very specific use-cases) to have separate journals and snapshot stores *per-node*. I think you might have an issue with Docker NAT. Have you read this part of the documentation: http://doc.akka.io/docs/akka/2.4.2/scala/remoting.html#Akka_behind_NAT_or_in_a_Docker_container
-Endre On Sat, Mar 19, 2016 at 3:27 AM, Eric Swenson <[email protected]> wrote: > Well, I may be able to answer my own question. It absolutely does matter > that the new remote cluster system is using the same akka-persistence > (cassandra keyspace) store as the old (local) one. When I changed the > cassandra keyspace to a new one, everything started working. > > So the question now is this: Do I have to have a separate akka-persistence > journal and snapshot store for every node in the cluster? This is very > inconvenient, as it means I have to make up keyspace names that are somehow > tied to each individual node. I guess I can add the ip address of the > Docker host to the keyspace name, but this isn’t terribly resilient. Why > does akka-persistence care? The journal should reflect events that apply to > all nodes. If a node goes down (getting a new address) and a new one takes > its place, it should be able to recover all the events from the old node. > There must be something else at play here. > > Help! > > — Eric > > On Mar 18, 2016, at 7:17 PM, Eric Swenson <[email protected]> wrote: > > One more thing to add to this, in case it is relevant. I see multiple of > these messages in the log: > > [ > akka://ClusterSystem/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator] > stopped > > > First, why is it stopping (or why does it stop, in general), and secondly, > is it significant that the url prefix is akka://ClusterSystem/ rather > than akka.tcp://[email protected]:2552/ > > And second, I assume it makes no difference that I’m using the same > akka-persistence journal/snapshot store as I used when the app was binding > to 127.0.0.1:2552. I get tons of log messages indicating that > receiveRecover is not happy trying to recover shards associated with > akka.tcp://[email protected]:2552/system/sharing/ExperimentInstance#-396422686. > I’m assuming this is expected and that akka persistence should be able to > deal with this case. It should fail to recover these and then carry on with > new persistence events that are targeted to the new ClusterSystem on the > new IP address. > > Examples of the rejected receiveRecover messages that I’m seeing are: > > [akka.tcp://[email protected]:2552/system/cassandra-journal] > Starting message scan from 1 > [DEBUG] [03/19/2016 02:09:02.712] > [ClusterSystem-akka.actor.default-dispatcher-18] [ > akka.tcp://[email protected]:2552/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator] > receiveRecover ShardRegionRegistered(Actor[ > akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance#-396422686 > ]) > > — Eric > > On Mar 18, 2016, at 6:54 PM, Eric Swenson <[email protected]> wrote: > > I’ve been unsuccessful in trying to get an akka-cluster application that > works fine with one instance to work when there are multiple members of the > clusters. A bit of background is in order: > > 1) the application is an akka-cluster-sharing application > 2) it runs in a docker container > 3) the cluster is comprised of multiple docker hosts, each running the > akka application > 4) the error I’m getting is this: > > [WARN] [03/19/2016 01:39:18.086] > [ClusterSystem-akka.actor.default-dispatcher-3] [ > akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance] > Trying to register to coordinator at [Some(ActorSelection[Anchor( > akka://ClusterSystem/), > Path(/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator)])], > but no acknowledgement. Total [1] buffered messages. > [WARN] [03/19/2016 01:39:20.086] > [ClusterSystem-akka.actor.default-dispatcher-3] [ > akka.tcp://[email protected]:2552/system/sharding/ExperimentInstance] > Trying to register to coordinator at [Some(ActorSelection[Anchor( > akka://ClusterSystem/), > Path(/system/sharding/ExperimentInstanceCoordinator/singleton/coordinator)])], > but no acknowledgement. Total [1] buffered messages. > > 5) the warning message is logged repeatedly and the cluster never > initializes. > 6) I’ve set the following config parameters: > akka.remote.netty.tcp.hostname: to the actual host ip address (the one > accessible from all the other docker hosts) > akka.remote.netty.tcp.bind-hostname: to 0.0.0.0 (so that it binds on the > docker0 interface, on the ip address of the container) > akka.remote.netty.tcp.port: 2552 > akka.remote.netty.tcp.bind-port:2552 > 7) when I start the container, I map port 2552 on the host to port 2552 on > the container. > 8) from the host, I’m able to do a “telnet <ip-address-of-host> 2552” so I > should be taking to the akka-remoting handler. > 9) I’m setting the akka..cluster.seed-nodes to a list of one element: > akka.tcp://ClusterSystem@<ip-address-of-host>:2552. I’m doing that > because, as far as I know, this seed-node list is advertised to all the > other members of the cluster (currently just the one) and must be > accessible to them all. The Docker container ip address (seen by the > docker container) is on the private docker0 interface, which is not > accessible from the outside (from outside the host). > > Now, before I tried all this, I set seed-nodes to a list of a single > aka.tcp://[email protected]:2552 entry, and left > akka.remote.netty.tcp.hostname, bind-hostname, port, and bind-port to their > default values. In this configuration, the cluster comes up perfectly fine > and the application (with an akka-http frontend and a cluster sharing > backend) works perfectly. In fact, when I use two seed nodes of the same > form, but different ports on the same local 127.0.0.1 host, and two > instances of the app (each binding to the different ports), the app works > fine too. The cluster comes up find and each node finds the other. > > But in a real environment, the multiple instances will be on different > nodes (diffferent ip addresses), and in my case, as docker containers on > different docker hosts. So clearly, the seed nodes must have externally > accessible ip addresses. > > Can anyone shed any light on what might be going wrong? > > How might I debug this? Thank. — Eric > > > > > > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Typesafe - Reactive apps on the JVM Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
