Thank you for your response and time. I have updated to version 2.4.16 and
have Akka debug logging enabled. I will keep a further eye on this and
update as appropriate.
On Saturday, December 17, 2016 at 3:28:22 AM UTC-8, √ wrote:
>
> Hi!
>
> Update to most recent version and report back.
>
> --
> Cheers,
> √
>
> On Dec 17, 2016 08:20, "Jordan Messec" <jme...@gmail.com <javascript:>>
> wrote:
>
>> Hello, I am struggling with a problem I have spent days trying to
>> resolve. I was hoping someone here may have some input that could help me
>> look in the right direction.
>>
>> I am running a small cluster with 3 nodes. Two nodes reside on one
>> machine, while the third resides on a separate machine. This cluster is
>> formed between two applications. Call them Web and DataDig. DataDig and Web
>> co-reside on Machine1 and Web is duplicated on machine two.
>>
>> Both use Akka 2.4.4, with Web's dependencies being transitive through
>> Play 2.5.4
>>
>> My problem is that after sometime of running without issue, the nodes
>> start having trouble communicating with each other. Within 24 hours of
>> bringing the cluster members online, the logs start to display the
>> following:
>>
>> [WARN] [12/16/2016 21:07:32.645] [a.r.ReliableDeliverySupervisor] [akka.
>> tcp://application@host1:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%40host2%3A2552-588]
>>
>> Association with remote system [akka.tcp://host2:2552] has failed, address
>> is now gated for [5000] ms. Reason: [Disassociated]
>>
>> A service is used to monitor cluster health, at this time it starts to
>> report that cluster members are unreachable from each other.
>>
>> Obviously this starts to cause problems with cluster behavior, and also
>> results in messages stating the Leader can currently not perform its duties:
>>
>> [INFO] [12/16/2016 21:05:48.440] [a.c.Cluster(akka://application)]
>> [akka.cluster.Cluster(akka://application)] Cluster Node
>> [akka.tcp://application@host1:2552] - Leader can currently not perform its
>> duties, reachability status: [akka.tcp://application@host1:2552 ->
>> akka.tcp://application@host2:2552: Unreachable [Unreachable] (328),
>> akka.tcp://application@host1:37770 -> akka.tcp://application@host2:2552:
>> Unreachable [Unreachable] (1), akka.tcp://application@host2:2552 ->
>> akka.tcp://application@host1:2552: Reachable [Reachable] (616),
>> akka.tcp://application@host2:2552 -> akka.tcp://application@host1:37770:
>> Unreachable [Unreachable] (617)], member status:
>> [akka.tcp://application@host1:2552 Up seen=true,
>> akka.tcp://application@host1:37770 Up seen=false,
>> akka.tcp://application@host2:2552 Leaving seen=false]
>>
>>
>> I have turned on Akka debug logging but the only further messages around
>> the time of Disassociation I see are:
>>
>> [DEBUG] [12/16/2016 21:42:58.893] [application-akka.actor.default-
>> dispatcher-24]
>> [akka.tcp://application@host1:37770/system/cluster/core/daemon]
>> Cluster Node [akka.tcp://application@host1:37770] - Receiving gossip from
>> [UniqueAddress(akka.tcp://application@host2:2552,921200398)]
>>
>> and
>>
>> [DEBUG] [12/16/2016 21:06:03.310] [a.r.EndpointWriter]
>> [akka.tcp://application@host1:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%40host2%3A2552-588/endpointWriter]
>>
>> Drained buffer with maxWriteCount: 50, fullBackoffCount: 1,
>> smallBackoffCount: 0, noBackoffCount: 0 , adaptiveBackoff: 1000
>>
>>
>>
>> Here is the configuration being used for Web:
>>
>> akka {
>> actor {
>> provider = "akka.cluster.ClusterActorRefProvider"
>> }
>>
>> remote {
>> secure-cookie = "9C7BBB890AB2C39691FC7B2A34F616C1D87FCC5B"
>> require-cookie = on
>> netty.tcp {
>> hostname = "localhost"
>> hostname = *$*{?HOSTNAME}
>> port = 2552
>> }
>> log-remote-lifecycle-events = off
>> }
>>
>> cluster {
>> failure-detector.threshold = 10
>> pub-sub {
>> name = distributedPubSubMediator
>> routing-logic = round-robin
>> gossip-interval = 1s
>> removed-time-to-live = 60s
>> max-delta-elements = 3000
>> }
>>
>> roles = ["Web"]
>>
>> seed-nodes = "akka.tcp://application@host1:2552"
>>
>> }
>>
>> loglevel = "DEBUG"
>> log-dead-letters-during-shutdown = off
>> log-dead-letters = off
>>
>> extensions = ["akka.cluster.pubsub.DistributedPubSub"]
>> }
>>
>> with JAVA_OPTS="
>> ...
>>
>> -XX:HeapDumpPath=$HOME/log/ \
>> -XX:+UseG1GC \
>> -XX:MaxGCPauseMillis=300 \
>> -XX:G1HeapWastePercent=20 \
>> -XX:InitiatingHeapOccupancyPercent=75 \
>> -XX:ConcGCThreads=32 \
>> -XX:ParallelGCThreads=48 \
>> -XX:NewRatio=1 \
>> -verbose:gc \
>> -XX:+UseGCLogFileRotation \
>> -XX:NumberOfGCLogFiles=1 \
>> -XX:GCLogFileSize=512M \
>> -XX:+PrintGCDetails \
>> -XX:+PrintGCTimeStamps \
>> -Xloggc:$HOME/log/services_web_gc.log
>> "
>>
>> With *very* similar config for DataDig.
>>
>> These hosts are very powerful machines that are not running any other
>> resource heavy processes (in fact they're barely running anything else at
>> all). There are a few GC pauses that are longer than I would expect.
>>
>>
>> Any help is appreciated, and I can provide any further
>> context/information.
>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+...@googlegroups.com <javascript:>.
>> To post to this group, send email to akka...@googlegroups.com
>> <javascript:>.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
