On Wed, Jun 24, 2015 at 2:32 PM, Anders Båtstrand <ander...@gmail.com> wrote:
> Sorry about the confusion, I am probably using some terminology wrong. I > will try again. > > This problem is happening on all my clusters under load, using Akka 2.3.11. > > I am using auto-down-after-unreachable, so nodes are downed (that is what > I called removed) automatically. > > When I start the cluster, all three members has all three members in the > set cluster.readView().members(). > > Under load, some members go to the set unreachableMembers(), and back > again. In some cases, they are too long unreachable, and are downed. Then > they are no longer part > of the set members(), and no longer part of the set unreachableMembers(). > That is why I called it "removed". > Thanks. Now I got it, and you are using the right terminology. > > After running the tests for a couple of hours, I see that TWO of the nodes > only have each other in members(), but the third node has all three in > members(). The third node in some cases also has a different leader. > > On all the nodes, the set unreachableMembers() is empty. > > What I don't understand is how the third node can have all three nodes in > the members() set, but the other nodes does not have it in theirs. This is > a stable state, I have to restart the third node to fix this. I would > expect that if one node is seeing another (has the node in members()), that > goes both ways. > Is the third node still operational? No OutOfMemoryErrors there? We have this issue https://github.com/akka/akka/issues/17479 with the cluster.readView (cluster.state is the public api, by the way). We should look at cluster logs (INFO level is enough), especially from the third node. Grep for log messages with "Cluster Node ". > > Hope this was more clear! I am trying to reproduce this in a more > controlled example, but I have not managed it so far. Our planned, > temporary solution is to run clusters of size one so far... :-( > > Anders > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
