Hi
I am yet unable to get the distributed-workers example to work for the case
when the lead master is terminated.
To eliminate any effects of any local code changes, I did a git clone of
https://github.com/typesafehub/activator-akka-distributed-workers-java.git
and worked on the code in the master branch.
I invoked the masters, frontend and workers using activator runMain
command. Attached are the logs and the conf files
master_node1.log is the log from master started on port 2551. It has the
logs from when it was started until it was terminated.
master_node2.log is the log from master started on port 2552. It has the
logs before and after Master 1 was terminated. It shows that it is downing
master on 2551 and starting the singleton master.
frontend.log is the log from the frontend. It shows that after Master1 is
terminated, it identifies Master 2 as the singleton master and thereafter
it is stuck in "Work not accepted, retry after a while"
worker.log is the log from the worker. It shows that after Master1 is
terminated, the cluster client is connected to receptionist on 2552
git status shows that there are no local changes
share@myhost:~/activator-akka-distributed-workers-java$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Any pointers on getting this working will be greatly appreciated. I have
read other posts mentioning that this specific usecase works, wondering if
this is an environment specific problem.
Thanks,
Atul
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
share@myhost:~/activator-akka-distributed-workers-java$
~/activator-1.3.9-minimal/bin/activator "runMain worker.Main 0"
[info] Loading project definition from
/home/share/activator-akka-distributed-workers-java/project
[info] Set current project to akka-distributed-workers-java (in build
file:/home/share/activator-akka-distributed-workers-java/)
[info] Running worker.Main 0
[INFO] [04/12/2016 16:19:37.470] [run-main-0] [akka.remote.Remoting] Starting
remoting
[INFO] [04/12/2016 16:19:37.560] [run-main-0] [akka.remote.Remoting] Remoting
started; listening on addresses :[akka.tcp://[email protected]:55458]
[INFO] [04/12/2016 16:19:37.561] [run-main-0] [akka.remote.Remoting] Remoting
now listens on addresses: [akka.tcp://[email protected]:55458]
[INFO] [04/12/2016 16:19:37.685] [WorkerSystem-akka.actor.default-dispatcher-3]
[akka.tcp://[email protected]:55458/user/clusterClient] Connected to
[akka.tcp://[email protected]:2551/system/receptionist]
[INFO] [04/12/2016 16:19:37.813] [WorkerSystem-akka.actor.default-dispatcher-4]
[akka://WorkerSystem/user/worker] Got work: 1
[INFO] [04/12/2016 16:19:37.814] [WorkerSystem-akka.actor.default-dispatcher-2]
[akka://WorkerSystem/user/worker/exec] Produced result 1 * 1 = 1
[INFO] [04/12/2016 16:19:37.814] [WorkerSystem-akka.actor.default-dispatcher-4]
[akka://WorkerSystem/user/worker] Work is complete. Result 1 * 1 = 1.
[INFO] [04/12/2016 16:19:37.903] [WorkerSystem-akka.actor.default-dispatcher-3]
[akka://WorkerSystem/user/worker] Got work: 2
[INFO] [04/12/2016 16:19:37.904]
[WorkerSystem-akka.actor.default-dispatcher-15]
[akka://WorkerSystem/user/worker/exec] Produced result 2 * 2 = 4
[INFO] [04/12/2016 16:19:37.904]
[WorkerSystem-akka.actor.default-dispatcher-15]
[akka://WorkerSystem/user/worker] Work is complete. Result 2 * 2 = 4.
...
... Master 1 terminated ...
...
[INFO] [04/12/2016 16:19:48.315]
[WorkerSystem-akka.actor.default-dispatcher-16]
[akka://WorkerSystem/user/worker] Work is complete. Result 13 * 13 = 169.
[WARN] [04/12/2016 16:19:51.047]
[WorkerSystem-akka.remote.default-remote-dispatcher-14]
[akka.tcp://[email protected]:55458/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Disassociated]
[WARN] [04/12/2016 16:19:57.597]
[WorkerSystem-akka.remote.default-remote-dispatcher-6]
[akka.tcp://[email protected]:55458/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2551]] Caused by: [Connection refused:
/127.0.0.1:2551]
[INFO] [04/12/2016 16:19:57.598]
[WorkerSystem-akka.actor.default-dispatcher-15]
[akka://WorkerSystem/deadLetters] Message
[akka.cluster.pubsub.DistributedPubSubMediator$SendToAll] from
Actor[akka://WorkerSystem/user/worker#-1091758387] to
Actor[akka://WorkerSystem/deadLetters] was not delivered. [1] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [04/12/2016 16:20:03.602]
[WorkerSystem-akka.remote.default-remote-dispatcher-6]
[akka.tcp://[email protected]:55458/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2551]] Caused by: [Connection refused:
/127.0.0.1:2551]
[INFO] [04/12/2016 16:20:05.594]
[WorkerSystem-akka.actor.default-dispatcher-15]
[akka.tcp://[email protected]:55458/user/clusterClient] Lost contact with
[Actor[akka.tcp://[email protected]:2551/system/receptionist#247500857]],
restablishing connection
[INFO] [04/12/2016 16:20:05.595]
[WorkerSystem-akka.actor.default-dispatcher-15]
[akka://WorkerSystem/deadLetters] Message
[akka.cluster.client.ClusterReceptionist$Internal$GetContacts$] from
Actor[akka://WorkerSystem/user/clusterClient#527872476] to
Actor[akka://WorkerSystem/deadLetters] was not delivered. [2] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:20:05.604] [WorkerSystem-akka.actor.default-dispatcher-4]
[akka.tcp://[email protected]:55458/user/clusterClient] Connected to
[akka.tcp://[email protected]:2552/system/receptionist]
share@myhost:~/activator-akka-distributed-workers-java$
~/activator-1.3.9-minimal/bin/activator "runMain worker.Main 2552"
[info] Loading project definition from
/home/share/activator-akka-distributed-workers-java/project
[info] Set current project to akka-distributed-workers-java (in build
file:/home/share/activator-akka-distributed-workers-java/)
[info] Running worker.Main 2552
[INFO] [04/12/2016 16:18:00.998] [run-main-0] [akka.remote.Remoting] Starting
remoting
[INFO] [04/12/2016 16:18:01.104] [run-main-0] [akka.remote.Remoting] Remoting
started; listening on addresses :[akka.tcp://[email protected]:2552]
[INFO] [04/12/2016 16:18:01.106] [run-main-0] [akka.remote.Remoting] Remoting
now listens on addresses: [akka.tcp://[email protected]:2552]
[INFO] [04/12/2016 16:18:01.112] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Starting up...
[INFO] [04/12/2016 16:18:01.155] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Registered cluster JMX MBean
[akka:type=Cluster]
[INFO] [04/12/2016 16:18:01.155] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Started up successfully
[INFO] [04/12/2016 16:18:01.165]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Metrics will be retreived from
MBeans, and may be i
ncorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to
the classpath and the appropriate platform-specific native libary to
'java.library.path'. Reason: java.lang.ClassNotFoundException:
org.hyperic.sigar.Sigar
[INFO] [04/12/2016 16:18:01.167]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Metrics collection has started
successfully
[INFO] [04/12/2016 16:18:01.393]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Welcome from
[akka.tcp://[email protected]:2
551]
[INFO] [04/12/2016 16:18:02.196]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka.tcp://[email protected]:2552/user/master] ClusterSingletonManager
state change [Start -> Younger]
...
... Master 1 terminated ...
...
[WARN] [04/12/2016 16:19:51.046]
[ClusterSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Assoc
iation with remote system [akka.tcp://[email protected]:2551] has failed,
address is now gated for [5000] ms. Reason: [Disassociated]
[INFO] [04/12/2016 16:19:51.412]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/he
artbeatSender#1166801241] to Actor[akka://ClusterSystem/deadLetters] was not
delivered. [1] dead letters encountered. This logging can be turned off or
adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-le
tters-during-shutdown'.
[INFO] [04/12/2016 16:19:52.182]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka://ClusterSystem/deadLetters] Message [akka.cluster.GossipStatus] from
Actor[akka://ClusterSystem/system/cluster/core/daemon#-574089683] to Actor[a
kka://ClusterSystem/deadLetters] was not delivered. [2] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:52.182]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka://ClusterSystem/deadLetters] Message [akka.cluster.MetricsGossipEnvelope]
from Actor[akka://ClusterSystem/system/cluster/metrics#1854668607] to Ac
tor[akka://ClusterSystem/deadLetters] was not delivered. [3] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:52.201]
[ClusterSystem-akka.actor.default-dispatcher-20]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.pubsub.DistributedPubSubMediator$Internal$Status] from
Actor[akka://ClusterSystem/system/distri
butedPubSubMediator#1825547433] to Actor[akka://ClusterSystem/deadLetters] was
not delivered. [4] dead letters encountered. This logging can be turned off or
adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-d
ead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:52.411]
[ClusterSystem-akka.actor.default-dispatcher-20]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#1166801241]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [5] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:53.182]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka://ClusterSystem/deadLetters] Message [akka.cluster.GossipStatus] from
Actor[akka://ClusterSystem/system/cluster/core/daemon#-574089683] to
Actor[akka://ClusterSystem/deadLetters] was not delivered. [6] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:53.412]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#1166801241]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [7] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:54.412]
[ClusterSystem-akka.actor.default-dispatcher-21]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#1166801241]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [8] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [04/12/2016 16:19:55.182]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka.tcp://[email protected]:2552/system/cluster/core/daemon] Cluster
Node [akka.tcp://[email protected]:2552] - Marking node(s) as UNREACHABLE
[Member(address = akka.tcp://[email protected]:2551, status = Up)]
[INFO] [04/12/2016 16:19:55.411]
[ClusterSystem-akka.actor.default-dispatcher-22]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#1166801241]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [9] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [04/12/2016 16:19:56.213]
[ClusterSystem-akka.remote.default-remote-dispatcher-6]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2551]] Caused by: [Connection refused:
/127.0.0.1:2551]
[INFO] [04/12/2016 16:19:56.213]
[ClusterSystem-akka.actor.default-dispatcher-22]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.pubsub.DistributedPubSubMediator$Internal$Status] from
Actor[akka://ClusterSystem/system/distributedPubSubMediator#1825547433] to
Actor[akka://ClusterSystem/deadLetters] was not delivered. [10] dead letters
encountered, no more dead letters will be logged. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[WARN] [04/12/2016 16:20:01.415]
[ClusterSystem-akka.remote.default-remote-dispatcher-6]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2551]] Caused by: [Connection refused:
/127.0.0.1:2551]
[INFO] [04/12/2016 16:20:04.802]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Leader is auto-downing unreachable
node [akka.tcp://[email protected]:2551]
[INFO] [04/12/2016 16:20:04.803]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Marking unreachable node
[akka.tcp://[email protected]:2551] as [Down]
[INFO] [04/12/2016 16:20:06.191]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2552] - Leader is removing unreachable node
[akka.tcp://[email protected]:2551]
[INFO] [04/12/2016 16:20:06.191]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka.tcp://[email protected]:2552/user/master] Previous oldest removed
[akka.tcp://[email protected]:2551]
[INFO] [04/12/2016 16:20:06.193]
[ClusterSystem-akka.actor.default-dispatcher-22]
[akka.tcp://[email protected]:2552/user/master] Younger observed
OldestChanged: [None -> myself]
[WARN] [04/12/2016 16:20:06.193]
[ClusterSystem-akka.remote.default-remote-dispatcher-5] [akka.remote.Remoting]
Association to [akka.tcp://[email protected]:2551] having UID
[-1577153647] is irrecoverably failed. UID is now quarantined and all messages
to this UID will be delivered to dead letters. Remote actorsystem must be
restarted to recover from this situation.
[INFO] [04/12/2016 16:20:06.193]
[ClusterSystem-akka.actor.default-dispatcher-22]
[akka.tcp://[email protected]:2552/user/master] Singleton manager
starting singleton actor [akka://ClusterSystem/user/master/singleton]
[INFO] [04/12/2016 16:20:06.194]
[ClusterSystem-akka.actor.default-dispatcher-22]
[akka.tcp://[email protected]:2552/user/master] ClusterSingletonManager
state change [Younger -> Oldest]
[WARN] [04/12/2016 16:20:06.617]
[ClusterSystem-akka.remote.default-remote-dispatcher-6]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-4]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2551]] Caused by: [Connection refused:
/127.0.0.1:2551]
[ERROR] [04/12/2016 16:20:16.634]
[ClusterSystem-akka.actor.default-dispatcher-20]
[akka.tcp://[email protected]:2552/user/master/singleton] Persistence
failure when replaying events for persistenceId [master]. Last known sequence
number [0] (akka.pattern.CircuitBreaker$$anon$1)
[WARN] [04/12/2016 16:21:42.172]
[ClusterSystem-akka.remote.default-remote-dispatcher-6]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FWorkerSystem%40127.0.0.1%3A55458-3]
Association with remote system [akka.tcp://[email protected]:55458] has
failed, address is now gated for [5000] ms. Reason: [Disassociated]
[WARN] [04/12/2016 16:21:45.750]
[ClusterSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A3001-2]
Association with remote system [akka.tcp://[email protected]:3001] has
failed, address is now gated for [5000] ms. Reason: [Disassociated]
akka {
actor.provider = "akka.remote.RemoteActorRefProvider"
remote.netty.tcp.port=0
remote.netty.tcp.hostname=127.0.0.1
cluster.client {
initial-contacts = [
"akka.tcp://[email protected]:2551/system/receptionist",
"akka.tcp://[email protected]:2552/system/receptionist"
]
}
}
akka {
actor.provider = "akka.cluster.ClusterActorRefProvider"
remote.netty.tcp.port=0
remote.netty.tcp.hostname=127.0.0.1
cluster {
seed-nodes = [
"akka.tcp://[email protected]:2551",
"akka.tcp://[email protected]:2552"]
auto-down-unreachable-after = 10s
}
extensions = ["akka.cluster.client.ClusterClientReceptionist"]
persistence {
journal.plugin = "akka.persistence.journal.leveldb-shared"
journal.leveldb-shared.store {
# DO NOT USE 'native = off' IN PRODUCTION !!!
native = off
dir = "target/shared-journal"
}
snapshot-store.plugin = "akka.persistence.snapshot-store.local"
snapshot-store.local.dir = "target/snapshots"
}
}
share@myhost:~/activator-akka-distributed-workers-java$
~/activator-1.3.9-minimal/bin/activator "runMain worker.Main 3001"
[info] Loading project definition from
/home/share/activator-akka-distributed-workers-java/project
[info] Set current project to akka-distributed-workers-java (in build
file:/home/share/activator-akka-distributed-workers-java/)
[info] Running worker.Main 3001
[INFO] [04/12/2016 16:18:38.586] [run-main-0] [akka.remote.Remoting] Starting
remoting
[INFO] [04/12/2016 16:18:38.672] [run-main-0] [akka.remote.Remoting] Remoting
started; listening on addresses :[akka.tcp://[email protected]:3001]
[INFO] [04/12/2016 16:18:38.674] [run-main-0] [akka.remote.Remoting] Remoting
now listens on addresses: [akka.tcp://[email protected]:3001]
[INFO] [04/12/2016 16:18:38.683] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:3001] - Starting up...
[INFO] [04/12/2016 16:18:38.736] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:3001] - Registered cluster JMX MBean
[akka:type=Cluster]
[INFO] [04/12/2016 16:18:38.736] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:3001] - Started up successfully
[INFO] [04/12/2016 16:18:38.744]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:3001] - Metrics will be retreived from
MBeans, and may be i
ncorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to
the classpath and the appropriate platform-specific native libary to
'java.library.path'. Reason: java.lang.ClassNotFoundException:
org.hyperic.sigar.Sigar
[INFO] [04/12/2016 16:18:38.746]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:3001] - Metrics collection has started
successfully
[INFO] [04/12/2016 16:18:38.891]
[ClusterSystem-akka.actor.default-dispatcher-4]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:3001] - Welcome from
[akka.tcp://[email protected]:25
51]
[INFO] [04/12/2016 16:18:38.950]
[ClusterSystem-akka.actor.default-dispatcher-16]
[akka.tcp://[email protected]:3001/user/frontend/masterProxy] Singleton
identified at [akka.tcp://[email protected]:2551/user/master/singlet
on]
[INFO] [04/12/2016 16:18:43.783]
[ClusterSystem-akka.actor.default-dispatcher-23]
[akka://ClusterSystem/user/producer] Produced work: 1
[INFO] [04/12/2016 16:18:50.924]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka://ClusterSystem/user/producer] Produced work: 2
[INFO] [04/12/2016 16:18:54.044]
[ClusterSystem-akka.actor.default-dispatcher-22]
[akka://ClusterSystem/user/producer] Produced work: 3
[INFO] [04/12/2016 16:18:59.153]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka://ClusterSystem/user/producer] Produced work: 4
[INFO] [04/12/2016 16:19:04.283]
[ClusterSystem-akka.actor.default-dispatcher-22]
[akka://ClusterSystem/user/producer] Produced work: 5
[INFO] [04/12/2016 16:19:07.413]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka://ClusterSystem/user/producer] Produced work: 6
[INFO] [04/12/2016 16:19:10.484]
[ClusterSystem-akka.actor.default-dispatcher-23]
[akka://ClusterSystem/user/producer] Produced work: 7
[INFO] [04/12/2016 16:19:16.604]
[ClusterSystem-akka.actor.default-dispatcher-17]
[akka://ClusterSystem/user/producer] Produced work: 8
[INFO] [04/12/2016 16:19:20.724]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka://ClusterSystem/user/producer] Produced work: 9
[INFO] [04/12/2016 16:19:24.854]
[ClusterSystem-akka.actor.default-dispatcher-4]
[akka://ClusterSystem/user/producer] Produced work: 10
[INFO] [04/12/2016 16:19:31.984]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka://ClusterSystem/user/producer] Produced work: 11
[INFO] [04/12/2016 16:19:37.864]
[ClusterSystem-akka.actor.default-dispatcher-21]
[akka://ClusterSystem/user/consumer] Consumed result: 1 * 1 = 1
[INFO] [04/12/2016 16:19:37.945]
[ClusterSystem-akka.actor.default-dispatcher-21]
[akka://ClusterSystem/user/consumer] Consumed result: 2 * 2 = 4
[INFO] [04/12/2016 16:19:38.028]
[ClusterSystem-akka.actor.default-dispatcher-21]
[akka://ClusterSystem/user/consumer] Consumed result: 3 * 3 = 9
...
... Master 1 terminated ...
...
[WARN] [04/12/2016 16:19:51.045]
[ClusterSystem-akka.remote.default-remote-dispatcher-6]
[akka.tcp://[email protected]:3001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Assoc
iation with remote system [akka.tcp://[email protected]:2551] has failed,
address is now gated for [5000] ms. Reason: [Disassociated]
[INFO] [04/12/2016 16:19:51.764]
[ClusterSystem-akka.actor.default-dispatcher-15]
[akka://ClusterSystem/deadLetters] Message [akka.cluster.GossipStatus] from
Actor[akka://ClusterSystem/system/cluster/core/daemon#634265111] to
Actor[akka://ClusterSystem/deadLetters] was not delivered. [1] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:51.914]
[ClusterSystem-akka.actor.default-dispatcher-23]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#-33021440]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [2] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:52.914]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#-33021440]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [3] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:53.764]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka://ClusterSystem/deadLetters] Message [akka.cluster.MetricsGossipEnvelope]
from Actor[akka://ClusterSystem/system/cluster/metrics#-930176151] to
Actor[akka://ClusterSystem/deadLetters] was not delivered. [4] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:53.774]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.pubsub.DistributedPubSubMediator$Internal$Status] from
Actor[akka://ClusterSystem/system/distributedPubSubMediator#551963266] to
Actor[akka://ClusterSystem/deadLetters] was not delivered. [5] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:53.914]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#-33021440]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [6] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[WARN] [04/12/2016 16:19:54.770]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka.tcp://[email protected]:3001/system/cluster/core/daemon] Cluster
Node [akka.tcp://[email protected]:3001] - Marking node(s) as UNREACHABLE
[Member(address = akka.tcp://[email protected]:2551, status = Up)]
[INFO] [04/12/2016 16:19:54.774]
[ClusterSystem-akka.actor.default-dispatcher-4]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.pubsub.DistributedPubSubMediator$Internal$Status] from
Actor[akka://ClusterSystem/system/distributedPubSubMediator#551963266] to
Actor[akka://ClusterSystem/deadLetters] was not delivered. [7] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:54.914]
[ClusterSystem-akka.actor.default-dispatcher-15]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#-33021440]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [8] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:55.774]
[ClusterSystem-akka.actor.default-dispatcher-23]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.pubsub.DistributedPubSubMediator$Internal$Status] from
Actor[akka://ClusterSystem/system/distributedPubSubMediator#551963266] to
Actor[akka://ClusterSystem/deadLetters] was not delivered. [9] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 16:19:55.915]
[ClusterSystem-akka.actor.default-dispatcher-23]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/heartbeatSender#-33021440]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [10] dead
letters encountered, no more dead letters will be logged. This logging can be
turned off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
[WARN] [04/12/2016 16:19:56.925]
[ClusterSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://[email protected]:3001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2551]] Caused by: [Connection refused:
/127.0.0.1:2551]
[INFO] [04/12/2016 16:19:57.284]
[ClusterSystem-akka.actor.default-dispatcher-4]
[akka://ClusterSystem/user/producer] Produced work: 14
[INFO] [04/12/2016 16:20:02.306]
[ClusterSystem-akka.actor.default-dispatcher-15]
[akka://ClusterSystem/user/producer] Work not accepted, retry after a while
[WARN] [04/12/2016 16:20:02.916]
[ClusterSystem-akka.remote.default-remote-dispatcher-20]
[akka.tcp://[email protected]:3001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0]
Association with remote system [akka.tcp://[email protected]:2551] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2551]] Caused by: [Connection refused:
/127.0.0.1:2551]
[WARN] [04/12/2016 16:20:06.780]
[ClusterSystem-akka.remote.default-remote-dispatcher-5] [akka.remote.Remoting]
Association to [akka.tcp://[email protected]:2551] having UID
[-1577153647] is irrecoverably failed. UID is now quarantined and all messages
to this UID will be delivered to dead letters. Remote actorsystem must be
restarted to recover from this situation.
[INFO] [04/12/2016 16:20:06.782]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka.tcp://[email protected]:3001/user/frontend/masterProxy] Singleton
identified at [akka.tcp://[email protected]:2552/user/master/singleton]
[INFO] [04/12/2016 16:20:10.344]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka://ClusterSystem/user/producer] Work not accepted, retry after a while
[INFO] [04/12/2016 16:20:18.384]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka://ClusterSystem/user/producer] Work not accepted, retry after a while
[INFO] [04/12/2016 16:20:26.424]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka://ClusterSystem/user/producer] Work not accepted, retry after a while
[INFO] [04/12/2016 16:20:34.464]
[ClusterSystem-akka.actor.default-dispatcher-17]
[akka://ClusterSystem/user/producer] Work not accepted, retry after a while
share@myhost:~/activator-akka-distributed-workers-java$ java -version
java version "1.8.0_74"
Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)
[INFO] [04/12/2016 15:42:59.825] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Starting up...
[INFO] [04/12/2016 15:42:59.955] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Registered cluster JMX MBean
[akka:type=Cluster]
[INFO] [04/12/2016 15:42:59.955] [run-main-0]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Started up successfully
[INFO] [04/12/2016 15:42:59.966]
[ClusterSystem-akka.actor.default-dispatcher-4]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Metrics will be retreived from
MBeans, and may be incorrect on some platforms. To increase metric accuracy add
the 'sigar.jar' to the classpath and the appropriate platform-specific native
libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException:
org.hyperic.sigar.Sigar
[INFO] [04/12/2016 15:42:59.968]
[ClusterSystem-akka.actor.default-dispatcher-4]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Metrics collection has started
successfully
[WARN] [04/12/2016 15:43:00.076]
[ClusterSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://[email protected]:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2552-0]
Association with remote system [akka.tcp://[email protected]:2552] has
failed, address is now gated for [5000] ms. Reason: [Association failed with
[akka.tcp://[email protected]:2552]] Caused by: [Connection refused:
/127.0.0.1:2552]
[INFO] [04/12/2016 15:43:00.081]
[ClusterSystem-akka.actor.default-dispatcher-4]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.InternalClusterAction$InitJoin$] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1973555989]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [1] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 15:43:00.994]
[ClusterSystem-akka.actor.default-dispatcher-14]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.InternalClusterAction$InitJoin$] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1973555989]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [2] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 15:43:01.994]
[ClusterSystem-akka.actor.default-dispatcher-2]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.InternalClusterAction$InitJoin$] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1973555989]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [3] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 15:43:02.994]
[ClusterSystem-akka.actor.default-dispatcher-16]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.InternalClusterAction$InitJoin$] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1973555989]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [4] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 15:43:03.995]
[ClusterSystem-akka.actor.default-dispatcher-16]
[akka://ClusterSystem/deadLetters] Message
[akka.cluster.InternalClusterAction$InitJoin$] from
Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#1973555989]
to Actor[akka://ClusterSystem/deadLetters] was not delivered. [5] dead letters
encountered. This logging can be turned off or adjusted with configuration
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [04/12/2016 15:43:05.005]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Node
[akka.tcp://[email protected]:2551] is JOINING, roles [backend]
[INFO] [04/12/2016 15:43:05.989]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Leader is moving node
[akka.tcp://[email protected]:2551] to [Up]
[INFO] [04/12/2016 15:43:05.993]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka.tcp://[email protected]:2551/user/master] Singleton manager
starting singleton actor [akka://ClusterSystem/user/master/singleton]
[INFO] [04/12/2016 15:43:05.994]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka.tcp://[email protected]:2551/user/master] ClusterSingletonManager
state change [Start -> Oldest]
[INFO] [04/12/2016 16:18:01.343]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Node
[akka.tcp://[email protected]:2552] is JOINING, roles [backend]
[INFO] [04/12/2016 16:18:01.987]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Leader is moving node
[akka.tcp://[email protected]:2552] to [Up]
[INFO] [04/12/2016 16:18:38.880]
[ClusterSystem-akka.actor.default-dispatcher-15]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Node
[akka.tcp://[email protected]:3001] is JOINING, roles []
[INFO] [04/12/2016 16:18:39.973]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node
[akka.tcp://[email protected]:2551] - Leader is moving node
[akka.tcp://[email protected]:3001] to [Up]
...
[INFO] [04/12/2016 16:19:04.286]
[ClusterSystem-akka.actor.default-dispatcher-230]
[akka://ClusterSystem/user/master/singleton] Accepted work:
3af66b36-a286-42ab-9dea-6e4cb0c2339d
[INFO] [04/12/2016 16:19:07.414]
[ClusterSystem-akka.actor.default-dispatcher-18]
[akka://ClusterSystem/user/master/singleton] Accepted work:
1ed937b6-807b-4ce3-b43a-f21ab7e53943
[INFO] [04/12/2016 16:19:10.485]
[ClusterSystem-akka.actor.default-dispatcher-3]
[akka://ClusterSystem/user/master/singleton] Accepted work:
19640106-e897-4c98-9609-efe9640288b7
[INFO] [04/12/2016 16:19:16.605]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka://ClusterSystem/user/master/singleton] Accepted work:
fcc86083-59ae-4e3f-8ab5-c5f8125e105f
[INFO] [04/12/2016 16:19:20.726]
[ClusterSystem-akka.actor.default-dispatcher-17]
[akka://ClusterSystem/user/master/singleton] Accepted work:
561445bf-bfaa-46a1-9d04-6eabfc058479
[INFO] [04/12/2016 16:19:24.857]
[ClusterSystem-akka.actor.default-dispatcher-229]
[akka://ClusterSystem/user/master/singleton] Accepted work:
6a8cfa83-1627-43f0-94c0-5fdef1b508d8
[INFO] [04/12/2016 16:19:31.985]
[ClusterSystem-akka.actor.default-dispatcher-231]
[akka://ClusterSystem/user/master/singleton] Accepted work:
cb7353d9-f572-4abb-bc89-d7f6512f5193
[INFO] [04/12/2016 16:19:37.706]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka://ClusterSystem/user/master/singleton] Worker registered:
2c5d5265-ba54-49c8-b2c9-397188bf2f40
[INFO] [04/12/2016 16:19:37.811]
[ClusterSystem-akka.actor.default-dispatcher-17]
[akka://ClusterSystem/user/master/singleton] Giving worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40 some work
136ec536-88a5-47c1-9e8b-8ee7f0903647
[INFO] [04/12/2016 16:19:37.816]
[ClusterSystem-akka.actor.default-dispatcher-19]
[akka://ClusterSystem/user/master/singleton] Work
136ec536-88a5-47c1-9e8b-8ee7f0903647 is done by worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40
[INFO] [04/12/2016 16:19:37.903]
[ClusterSystem-akka.actor.default-dispatcher-17]
[akka://ClusterSystem/user/master/singleton] Giving worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40 some work
7f722010-90df-49ce-a1c2-a4122a886dcf
[INFO] [04/12/2016 16:19:37.905]
[ClusterSystem-akka.actor.default-dispatcher-229]
[akka://ClusterSystem/user/master/singleton] Work
7f722010-90df-49ce-a1c2-a4122a886dcf is done by worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40
[INFO] [04/12/2016 16:19:37.986]
[ClusterSystem-akka.actor.default-dispatcher-231]
[akka://ClusterSystem/user/master/singleton] Giving worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40 some work
15b896df-9a90-4a08-89ba-d4d3f0e7e25c
[INFO] [04/12/2016 16:19:37.988]
[ClusterSystem-akka.actor.default-dispatcher-229]
[akka://ClusterSystem/user/master/singleton] Work
15b896df-9a90-4a08-89ba-d4d3f0e7e25c is done by worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40
[INFO] [04/12/2016 16:19:38.069]
[ClusterSystem-akka.actor.default-dispatcher-231]
[akka://ClusterSystem/user/master/singleton] Giving worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40 some work
e36ea8eb-c4d0-4e47-90d2-04d7640a61da
[INFO] [04/12/2016 16:19:38.071]
[ClusterSystem-akka.actor.default-dispatcher-231]
[akka://ClusterSystem/user/master/singleton] Work
e36ea8eb-c4d0-4e47-90d2-04d7640a61da is done by worker
2c5d5265-ba54-49c8-b2c9-397188bf2f40
