Hi
I have a 2 node akka cluster, which looses connection once every two days.
Often when the server i almost idle. We are still on akka 2.2.4, using
oracle java 1.8_25. Its a virtual servers running on vmware esx.
The application have been running fine for 3 years, but started with this a
few weeks ago.
Anyone have any ideas on were to start troubleshooting this?
My clusterconfig:
remote {
log-sent-messages = on
log-received-messages = on
log-remote-lifecycle-events = off
netty.tcp {
port = 45000
}
watch-failure-detector.threshold = 15
}
cluster {
log-info = on
auto-down = on
auto-join = on
failure-detector {
threshold = 15
min-std-deviation = 500 ms
acceptable-heartbeat-pause = 20 s
}
heartbeat-request {
grace-period = 20 s
expected-response-after = 10 s
time-to-live = 60 s
}
use-dispatcher = cluster-dispatcher
}
}
cluster-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 2
parallelism-max = 4
}
}
logoutput:
node1:
2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(TakeOver(akka.remote.transport.AkkaProtocolHandle@40711a86
),Actor[ak
ka://svarut/system/endpointManager#1220933312],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:27,666 [svarut-akka.actor.default-dispatcher-31] ERROR
a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://
[email protected]:45000] - Marking node(s) as UNREACHABLE [Memb
er(address = akka.tcp://[email protected]:45000, status = Up)]
2015-01-24 03:30:28,504 [svarut-akka.actor.default-dispatcher-17] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
[email protected]:45000] - Leader is marking unreachable node
[akka.tcp://[email protected]:45000] as [Down]
2015-01-24 03:30:28,506 [svarut-akka.actor.default-dispatcher-25] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
[email protected]:45000] - Leader is removing unreachable node
[akka.tcp://[email protected]:45000]
Node2:
2015-01-24 03:30:02,040 [svarut-akka.actor.default-dispatcher-12] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903])
2015-01-24 03:30:02,042 [svarut-akka.actor.default-dispatcher-21] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903])
2015-01-24 03:30:28,636 [svarut-akka.actor.default-dispatcher-2] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Acto
r[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980])
2015-01-24 03:30:28,637 [svarut-akka.actor.default-dispatcher-18] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980])
2015-01-24 03:30:41,457 [svarut-akka.actor.default-dispatcher-22] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
[email protected]:45000] - Trigger extra expected heartbeat fr
om [akka.tcp://[email protected]:45000]
2015-01-24 03:31:06,471 [svarut-akka.actor.default-dispatcher-2] ERROR
a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://
[email protected]:45000] - Marking node(s) as UNREACHABLE [Membe
r(address = akka.tcp://[email protected]:45000, status = Up)]
2015-01-24 03:31:07,445 [svarut-akka.actor.default-dispatcher-12] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
[email protected]:45000] - Leader is marking unreachable node
[akka.tcp://[email protected]:45000] as [Down]
2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-12] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
[email protected]:45000] - Leader is removing unreachable node
[akka.tcp://[email protected]:45000]
2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-2] INFO
Remoting - Address [akka.tcp://[email protected]:45000] is now
quarantined, all messages to this address will be
delivered to dead letters.
2015-01-24 03:31:07,516 [svarut-akka.actor.default-dispatcher-12] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-1/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-0#-
1902872711])
2015-01-24 03:31:07,517 [svarut-akka.actor.default-dispatcher-12] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-5#-2023131039])
--
Idar
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
