Hi.
First of all, thanks for your time reading this.
I'm working with an Akka cluster and I don't understand a peculiar effect
I'm observing. If I send 70 simultaneous messages (and for each message a
callback resend the message again) everything works fine. But if I try the
same with 100 simultaneous messages I'm getting messages like:
[INFO] [05/20/2014 23:38:07.469]
[object-server-cluster-akka.actor.default-dispatcher-5]
[Cluster(akka://object-server-cluster)] Cluster Node
[akka.tcp://[email protected]:62912] - Welcome from
[akka.tcp://[email protected]:12551]
[INFO] [05/20/2014 23:38:08.107]
[object-server-cluster-akka.actor.default-dispatcher-5]
[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1]
Message [akka.remote.transport.AssociationHandle$Disassociated] from
Actor[akka://object-server-cluster/deadLetters] to
Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650]
was not delivered. [1] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[WARN] [05/20/2014 23:38:08.113]
[object-server-cluster-akka.remote.default-remote-dispatcher-7]
[akka.tcp://[email protected]:62912/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-0]
Association with remote system
[akka.tcp://[email protected]:12551] has failed, address is
now gated for [5000] ms. Reason is: [Disassociated].
[INFO] [05/20/2014 23:38:08.119]
[object-server-cluster-akka.actor.default-dispatcher-4]
[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1]
Message
[akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from
Actor[akka://object-server-cluster/deadLetters] to
Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650]
was not delivered. [2] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:08.120]
[object-server-cluster-akka.actor.default-dispatcher-3]
[akka://object-server-cluster/deadLetters] Message
[dw.core.objectserver.providers.akka.client.CommandRequest] from
Actor[akka://object-server-cluster/user/pipelining-router#476466102] to
Actor[akka://object-server-cluster/deadLetters] was not delivered. [3] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
( I've attached the error log for your convenience and my config files ).
The processors are at backend side (process port 12551 and the log is
related to the frontend process.
I'm sure it is related to configuration but I couldn't cut the right wire
yet.
Many thanks for your help.
Regards,
Eduardo.
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
[INFO] [05/20/2014 23:38:07.469]
[object-server-cluster-akka.actor.default-dispatcher-5]
[Cluster(akka://object-server-cluster)] Cluster Node
[akka.tcp://[email protected]:62912] - Welcome from
[akka.tcp://[email protected]:12551]
[INFO] [05/20/2014 23:38:08.107]
[object-server-cluster-akka.actor.default-dispatcher-5]
[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1]
Message [akka.remote.transport.AssociationHandle$Disassociated] from
Actor[akka://object-server-cluster/deadLetters] to
Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650]
was not delivered. [1] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[WARN] [05/20/2014 23:38:08.113]
[object-server-cluster-akka.remote.default-remote-dispatcher-7]
[akka.tcp://[email protected]:62912/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-0]
Association with remote system
[akka.tcp://[email protected]:12551] has failed, address is now
gated for [5000] ms. Reason is: [Disassociated].
[INFO] [05/20/2014 23:38:08.119]
[object-server-cluster-akka.actor.default-dispatcher-4]
[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1]
Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying]
from Actor[akka://object-server-cluster/deadLetters] to
Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650]
was not delivered. [2] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:08.120]
[object-server-cluster-akka.actor.default-dispatcher-3]
[akka://object-server-cluster/deadLetters] Message
[dw.core.objectserver.providers.akka.client.CommandRequest] from
Actor[akka://object-server-cluster/user/pipelining-router#476466102] to
Actor[akka://object-server-cluster/deadLetters] was not delivered. [3] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:08.222]
[object-server-cluster-akka.actor.default-dispatcher-2]
[akka://object-server-cluster/deadLetters] Message [akka.cluster.GossipStatus]
from Actor[akka://object-server-cluster/system/cluster/core/daemon#1263987973]
to Actor[akka://object-server-cluster/deadLetters] was not delivered. [4] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:08.505]
[object-server-cluster-akka.actor.default-dispatcher-2]
[akka://object-server-cluster/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855]
to Actor[akka://object-server-cluster/deadLetters] was not delivered. [5] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:09.212]
[object-server-cluster-akka.actor.default-dispatcher-4]
[akka://object-server-cluster/deadLetters] Message [akka.cluster.GossipStatus]
from Actor[akka://object-server-cluster/system/cluster/core/daemon#1263987973]
to Actor[akka://object-server-cluster/deadLetters] was not delivered. [6] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:09.502]
[object-server-cluster-akka.actor.default-dispatcher-4]
[akka://object-server-cluster/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855]
to Actor[akka://object-server-cluster/deadLetters] was not delivered. [7] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[WARN] [05/20/2014 23:38:09.526]
[object-server-cluster-akka.remote.default-remote-dispatcher-7] [Remoting]
Tried to associate with unreachable remote address
[akka.tcp://[email protected]:62888]. Address is now gated for
5000 ms, all messages to this address will be delivered to dead letters.
Reason: Connection refused: no further information: /127.0.0.1:62888
[INFO] [05/20/2014 23:38:09.537]
[object-server-cluster-akka.actor.default-dispatcher-3]
[akka://object-server-cluster/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855]
to Actor[akka://object-server-cluster/deadLetters] was not delivered. [8] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:09.537]
[object-server-cluster-akka.actor.default-dispatcher-3]
[akka://object-server-cluster/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855]
to Actor[akka://object-server-cluster/deadLetters] was not delivered. [9] dead
letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[INFO] [05/20/2014 23:38:10.210]
[object-server-cluster-akka.actor.default-dispatcher-3]
[akka://object-server-cluster/deadLetters] Message [akka.cluster.GossipStatus]
from Actor[akka://object-server-cluster/system/cluster/core/daemon#1263987973]
to Actor[akka://object-server-cluster/deadLetters] was not delivered. [10] dead
letters encountered, no more dead letters will be logged. This logging can be
turned off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
include "object_server_common"
object-server-cluster {
pipeline-actor-flush-timeout = 1
pipeline-flush-timeout = 1
pipeline-host-entry-timeout = 3600000
pipeline-max-host-queue-size = 1000000
pipeline-max-host-queue-len = 1000
pipeline-actor-max-cache-size = 1000000
}
akka.actor.deployment {
/pipelining-router {
router = consistent-hashing-pool
nr-of-instances = 5
virtual-nodes-factor = 10
}
/worker-router {
router = round-robin-pool
nr-of-instances = 5
}
}
akka {
# loglevel = "DEBUG"
actor {
provider = "akka.cluster.ClusterActorRefProvider"
default-dispatcher {
throughput = 1024
fork-join-executor {
parallelism-max = 4
}
}
# default-mailbox {
# mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox"
# }
}
remote {
log-remote-lifecycle-events = off
netty.tcp {
hostname = "127.0.0.1"
port = 0
tcp-nodelay = on
write-buffer-high-water-mark = 40000000b
write-buffer-low-water-mark = 0b
send-buffer-size = 40000000b
receive-buffer-size = 40000000b
}
}
cluster {
seed-nodes = ["akka.tcp://[email protected]:12551"]
# seed-nodes = ["akka.tcp://[email protected]:12552"]
auto-down-unreachable-after = 10s
}
}
akka.cluster.min-nr-of-members = 2
akka.cluster.role {
frontend.min-nr-of-members = 1
backend.min-nr-of-members = 1
}
akka.contrib.cluster.sharding.role = "backend"
akka.actor.deployment {
/user/router {
router = round-robin-pool
nr-of-instances = 5
cluster {
enabled = on
use-role = backend
}
}
}
