Hi Idar, A lot of things have been fixed in akka since 2.2.4. Would it be possible to upgrade to 2.3.9?
>From the log that you pasted it seems like the akka protocol state actor for >the connection to node 2 has died on node 1, before your log starts. Is there >something unusual going on on the machines, like GC or something else that >would consume resources at the time of the failure? Are you monitoring other >health metrics like disk space and swap usage? B/ On 26 January 2015 at 12:31:19, Idar Borlaug ([email protected]) wrote: Hi I have a 2 node akka cluster, which looses connection once every two days. Often when the server i almost idle. We are still on akka 2.2.4, using oracle java 1.8_25. Its a virtual servers running on vmware esx. The application have been running fine for 3 years, but started with this a few weeks ago. Anyone have any ideas on were to start troubleshooting this? My clusterconfig: remote { log-sent-messages = on log-received-messages = on log-remote-lifecycle-events = off netty.tcp { port = 45000 } watch-failure-detector.threshold = 15 } cluster { log-info = on auto-down = on auto-join = on failure-detector { threshold = 15 min-std-deviation = 500 ms acceptable-heartbeat-pause = 20 s } heartbeat-request { grace-period = 20 s expected-response-after = 10 s time-to-live = 60 s } use-dispatcher = cluster-dispatcher } } cluster-dispatcher { type = "Dispatcher" executor = "fork-join-executor" fork-join-executor { parallelism-min = 2 parallelism-max = 4 } } logoutput: node1: 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621]) 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/ /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621]) 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621]) 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424]) 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424]) 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424]) 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(TakeOver(akka.remote.transport.AkkaProtocolHandle@40711a86),Actor[ak ka://svarut/system/endpointManager#1220933312],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424]) 2015-01-24 03:30:27,666 [svarut-akka.actor.default-dispatcher-31] ERROR a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://[email protected]:45000] - Marking node(s) as UNREACHABLE [Memb er(address = akka.tcp://[email protected]:45000, status = Up)] 2015-01-24 03:30:28,504 [svarut-akka.actor.default-dispatcher-17] INFO Cluster(akka://svarut) - Cluster Node [akka.tcp://[email protected]:45000] - Leader is marking unreachable node [akka.tcp://[email protected]:45000] as [Down] 2015-01-24 03:30:28,506 [svarut-akka.actor.default-dispatcher-25] INFO Cluster(akka://svarut) - Cluster Node [akka.tcp://[email protected]:45000] - Leader is removing unreachable node [akka.tcp://[email protected]:45000] Node2: 2015-01-24 03:30:02,040 [svarut-akka.actor.default-dispatcher-12] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903]) 2015-01-24 03:30:02,042 [svarut-akka.actor.default-dispatcher-21] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/ /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903]) 2015-01-24 03:30:28,636 [svarut-akka.actor.default-dispatcher-2] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Acto r[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980]) 2015-01-24 03:30:28,637 [svarut-akka.actor.default-dispatcher-18] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/ /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980]) 2015-01-24 03:30:41,457 [svarut-akka.actor.default-dispatcher-22] INFO Cluster(akka://svarut) - Cluster Node [akka.tcp://[email protected]:45000] - Trigger extra expected heartbeat fr om [akka.tcp://[email protected]:45000] 2015-01-24 03:31:06,471 [svarut-akka.actor.default-dispatcher-2] ERROR a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://[email protected]:45000] - Marking node(s) as UNREACHABLE [Membe r(address = akka.tcp://[email protected]:45000, status = Up)] 2015-01-24 03:31:07,445 [svarut-akka.actor.default-dispatcher-12] INFO Cluster(akka://svarut) - Cluster Node [akka.tcp://[email protected]:45000] - Leader is marking unreachable node [akka.tcp://[email protected]:45000] as [Down] 2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-12] INFO Cluster(akka://svarut) - Cluster Node [akka.tcp://[email protected]:45000] - Leader is removing unreachable node [akka.tcp://[email protected]:45000] 2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-2] INFO Remoting - Address [akka.tcp://[email protected]:45000] is now quarantined, all messages to this address will be delivered to dead letters. 2015-01-24 03:31:07,516 [svarut-akka.actor.default-dispatcher-12] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/ /svarut/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-1/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-0#- 1902872711]) 2015-01-24 03:31:07,517 [svarut-akka.actor.default-dispatcher-12] WARN n.k.e.DeadLetterLogger - Dead letter: DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/ /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-5#-2023131039]) -- Idar -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Björn Antonsson Typesafe – Reactive Apps on the JVM twitter: @bantonsson -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
