ons 20 sep. 2017 kl. 02:05 skrev Joe Wong <[email protected]>: > Hi, > > We have a setup of an akka cluster with the following: seed node, a master > node, and worker nodes. Workers will broadcast a request work message that > the master node respond to. We deployed this setup in Google Kubernetes > environment. It usually works fine. workers sometimes run out of memory but > docker will restart the pod and they usually are able to reconnect to the > akka cluster. > > We recently had an incident where all the worker nodes were busy, which is > normal, and all of a sudden the seed node cpu shot up and stayed up. At > that point we saw network traffic for seed, master, and worker nodes drop > to 0 at the same time (Datadog graphs). We had to restart the components to > fix the problem. Once restarted they were fine and continued to do the job. > The seed node memory usage was close to the limit but docker did not > restart the pod. > > The seed node has very little logic, it's main purpose is to serve as an > entry point to form the akka cluster. There are 40+ workers and 1 master. > > So I have the following questions: > > 1. Correct me if I'm wrong but when a cluster is formed the seed node can > disappear and the cluster will continue to function. The impact in this > scenario is no new workers can join the existing cluster. >
That is correct. Seed are initial contact points when joining, but otherwise equal as other nodes. Any node can be a seed node at any time. 2. The DataDog graphs do indicate when the seed node had high cpu load it > seems to stop the worker's message from reaching the master. Is that what > is happening? Please note the workers are broadcasting, not sending > messages directly to the master. > How do you broadcast? Distributed PubSub? 3. With the number of workers and the frequency of broadcast messages is it > possible to be overloading the nodes in the cluster with messages? > I don’t think you said how often they broadcast. I think you need to collect more information to understand what is going on. /Patrik > Thanks > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
