Hi. First of all, thanks for your time reading this.
I'm working with an Akka cluster and I don't understand a peculiar effect I'm observing. If I send 70 simultaneous messages (and for each message a callback resend the message again) everything works fine. But if I try the same with 100 simultaneous messages I'm getting messages like: [INFO] [05/20/2014 23:38:07.469] [object-server-cluster-akka.actor.default-dispatcher-5] [Cluster(akka://object-server-cluster)] Cluster Node [akka.tcp://object-server-cluster@127.0.0.1:62912] - Welcome from [akka.tcp://object-server-cluster@127.0.0.1:12551] [INFO] [05/20/2014 23:38:08.107] [object-server-cluster-akka.actor.default-dispatcher-5] [akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://object-server-cluster/deadLetters] to Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [05/20/2014 23:38:08.113] [object-server-cluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://object-server-cluster@127.0.0.1:62912/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-0] Association with remote system [akka.tcp://object-server-cluster@127.0.0.1:12551] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. [INFO] [05/20/2014 23:38:08.119] [object-server-cluster-akka.actor.default-dispatcher-4] [akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://object-server-cluster/deadLetters] to Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:08.120] [object-server-cluster-akka.actor.default-dispatcher-3] [akka://object-server-cluster/deadLetters] Message [dw.core.objectserver.providers.akka.client.CommandRequest] from Actor[akka://object-server-cluster/user/pipelining-router#476466102] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. ( I've attached the error log for your convenience and my config files ). The processors are at backend side (process port 12551 and the log is related to the frontend process. I'm sure it is related to configuration but I couldn't cut the right wire yet. Many thanks for your help. Regards, Eduardo. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
[INFO] [05/20/2014 23:38:07.469] [object-server-cluster-akka.actor.default-dispatcher-5] [Cluster(akka://object-server-cluster)] Cluster Node [akka.tcp://object-server-cluster@127.0.0.1:62912] - Welcome from [akka.tcp://object-server-cluster@127.0.0.1:12551] [INFO] [05/20/2014 23:38:08.107] [object-server-cluster-akka.actor.default-dispatcher-5] [akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://object-server-cluster/deadLetters] to Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [05/20/2014 23:38:08.113] [object-server-cluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://object-server-cluster@127.0.0.1:62912/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-0] Association with remote system [akka.tcp://object-server-cluster@127.0.0.1:12551] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. [INFO] [05/20/2014 23:38:08.119] [object-server-cluster-akka.actor.default-dispatcher-4] [akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://object-server-cluster/deadLetters] to Actor[akka://object-server-cluster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fobject-server-cluster%40127.0.0.1%3A12551-1#-2009383650] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:08.120] [object-server-cluster-akka.actor.default-dispatcher-3] [akka://object-server-cluster/deadLetters] Message [dw.core.objectserver.providers.akka.client.CommandRequest] from Actor[akka://object-server-cluster/user/pipelining-router#476466102] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:08.222] [object-server-cluster-akka.actor.default-dispatcher-2] [akka://object-server-cluster/deadLetters] Message [akka.cluster.GossipStatus] from Actor[akka://object-server-cluster/system/cluster/core/daemon#1263987973] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:08.505] [object-server-cluster-akka.actor.default-dispatcher-2] [akka://object-server-cluster/deadLetters] Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:09.212] [object-server-cluster-akka.actor.default-dispatcher-4] [akka://object-server-cluster/deadLetters] Message [akka.cluster.GossipStatus] from Actor[akka://object-server-cluster/system/cluster/core/daemon#1263987973] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:09.502] [object-server-cluster-akka.actor.default-dispatcher-4] [akka://object-server-cluster/deadLetters] Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [05/20/2014 23:38:09.526] [object-server-cluster-akka.remote.default-remote-dispatcher-7] [Remoting] Tried to associate with unreachable remote address [akka.tcp://object-server-cluster@127.0.0.1:62888]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: no further information: /127.0.0.1:62888 [INFO] [05/20/2014 23:38:09.537] [object-server-cluster-akka.actor.default-dispatcher-3] [akka://object-server-cluster/deadLetters] Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:09.537] [object-server-cluster-akka.actor.default-dispatcher-3] [akka://object-server-cluster/deadLetters] Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://object-server-cluster/system/cluster/core/daemon/heartbeatSender#440189855] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [05/20/2014 23:38:10.210] [object-server-cluster-akka.actor.default-dispatcher-3] [akka://object-server-cluster/deadLetters] Message [akka.cluster.GossipStatus] from Actor[akka://object-server-cluster/system/cluster/core/daemon#1263987973] to Actor[akka://object-server-cluster/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
include "object_server_common" object-server-cluster { pipeline-actor-flush-timeout = 1 pipeline-flush-timeout = 1 pipeline-host-entry-timeout = 3600000 pipeline-max-host-queue-size = 1000000 pipeline-max-host-queue-len = 1000 pipeline-actor-max-cache-size = 1000000 } akka.actor.deployment { /pipelining-router { router = consistent-hashing-pool nr-of-instances = 5 virtual-nodes-factor = 10 } /worker-router { router = round-robin-pool nr-of-instances = 5 } }
akka { # loglevel = "DEBUG" actor { provider = "akka.cluster.ClusterActorRefProvider" default-dispatcher { throughput = 1024 fork-join-executor { parallelism-max = 4 } } # default-mailbox { # mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" # } } remote { log-remote-lifecycle-events = off netty.tcp { hostname = "127.0.0.1" port = 0 tcp-nodelay = on write-buffer-high-water-mark = 40000000b write-buffer-low-water-mark = 0b send-buffer-size = 40000000b receive-buffer-size = 40000000b } } cluster { seed-nodes = ["akka.tcp://object-server-cluster@127.0.0.1:12551"] # seed-nodes = ["akka.tcp://object-server-cluster@127.0.0.1:12552"] auto-down-unreachable-after = 10s } } akka.cluster.min-nr-of-members = 2 akka.cluster.role { frontend.min-nr-of-members = 1 backend.min-nr-of-members = 1 } akka.contrib.cluster.sharding.role = "backend" akka.actor.deployment { /user/router { router = round-robin-pool nr-of-instances = 5 cluster { enabled = on use-role = backend } } }