Thanks. I tried your settings. akka.remote.transport-failure-detector.heartbeat-interval = 1 s akka.remote.transport-failure-detector.acceptable-hearbeat-pause = 3 s It seems that the frequence of the problem reduces.
Can I use the following workaround ? One single Actor----STATE----in websocket server watches a actor in business server. When STATE receives a Terminated message, it starts to send ping messages to the business server periodly via ActorSelection until it receives a response. When a websocket client connects to the websocket server, it asks the single actor whether it can send ConnectionOpened message to the business server. If the business server is unreachable, it closes the websocket connection and retries a later time. Does this approach avoid the possible race condition? On Thu, Nov 26, 2015 at 9:06 PM, Akka Team <[email protected]> wrote: > Hi, > > Can you try the settings > akka.remote.transport-failure-detector.heartbeat-interval = 1 s > akka.remote.transport-failure-detector.acceptable-hearbeat-pause = 3 s > > and see if the race still happens? This seems to be related to a test > failure that we get from time to time but so far has been not able to > reliably reproduce. If the frequency of the problem with the above settings > reduces then there is a good chance that what you see is the same what we > looked at before. > > -Endre > > On Thu, Nov 26, 2015 at 1:29 PM, jie tang <[email protected]> wrote: > >> Highly possible but not always. >> >> A typical flow: >> 1 A websocket client connects to websocket server A >> 2 The websocket server A creates a WebsocketActor for the client. The >> WebsocketActor sends a ConnectionOpened message to businessserver via >> ActorSelection. >> 3 If the WebsocketActor can not receive the ConnectionRegistered message >> in 10 seconds , it kills it itself and closes websocket connection between >> the websocket client. >> 4 Now the websocket client may repeat step 1 immediately. >> >> There are thousands of websocket clients connecting to websocket server >> A. So when the business server restarts, the websocket server A may sends >> hundreds of ConnectionOpened messages via ActorSelection at the same time. >> There may be a race condition? >> >> >> On Thu, Nov 26, 2015 at 8:16 PM, Akka Team <[email protected]> >> wrote: >> >>> Hi, >>> >>> On Thu, Nov 26, 2015 at 1:13 PM, jie tang <[email protected]> wrote: >>> >>>> Thanks for your reply. >>>> >>>> But the two tcp connections were both initiated by websocket server A. >>>> Is that valid? >>>> >>> >>> No, *that* is not valid. It might happen for a short transient period, >>> but not for long. This is likely a bug. Can you reproduce this reliably? >>> >>> -Endre >>> >>> >>>> >>>> And it seemed that websocket server A could never receive >>>> ConnectionRegistered messages after business server restarted. I waited for >>>> 10 minutes. At last I have to restart websocket server A. >>>> >>>> On Thu, Nov 26, 2015 at 6:45 PM, Akka Team <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>>> But everything is ok for websocket server A. There was only a tcp >>>>>> connection between it and businessserver. And it received >>>>>> ConnectionRegistered messages. >>>>>> >>>>>> >>>>>> What's wrong with websocket server B? Is it ok for two tcp >>>>>> connections between websocket server B and business server? What I should >>>>>> do to avoid it? >>>>>> >>>>> >>>>> It is normal to have two TCP connections between Actor Systems. By >>>>> default an ActorSystem tries to reuse incoming connections for outgoing >>>>> messages. However, in cases where the two systems open the connections >>>>> concurrently (for example you restart one of them, while the other tries >>>>> to >>>>> connect to it, so when it comes up they both try to connect) there might >>>>> be >>>>> two such connections alive. While there is a theoretical opportunity for >>>>> the two systems to reconcile and agree on closing one of them, it is not a >>>>> simple thing to do safely in practice so we didn't bother so far. >>>>> >>>>> You must also be prepared for temporary message losses between >>>>> reconnects since Akka does not guarantee delivery and does not buffer >>>>> messages forever. See >>>>> http://doc.akka.io/docs/akka/2.4.0/scala/remoting.html#Lifecycle_and_Failure_Recovery_Model >>>>> for details. >>>>> >>>>> -Endre >>>>> >>>>> >>>>>> >>>>>> >>>>>> The netstat's output: >>>>>> [root@dev-sig-server ~]# netstat -anp | grep 2554 >>>>>> tcp 0 0 10.170.187.126:2554 0.0.0.0:* >>>>>> LISTEN 4119/java >>>>>> tcp 0 0 10.170.187.126:2554 10.162.209.21:32833 >>>>>> ESTABLISHED 4119/java >>>>>> tcp 0 0 10.170.187.126:2554 10.162.198.161:33326 >>>>>> ESTABLISHED 4119/java >>>>>> tcp 0 0 10.170.187.126:2554 10.162.209.21:32843 >>>>>> ESTABLISHED 4119/java >>>>>> >>>>>> >>>>>> We use akka-remote_2.10-2.3.12.jar >>>>>> >>>>>> business server's application.conf: >>>>>> akka { >>>>>> actor { >>>>>> provider = "akka.remote.RemoteActorRefProvider" >>>>>> } >>>>>> remote { >>>>>> log-received-messages = on >>>>>> log-sent-messages = on >>>>>> >>>>>> enabled-transports = ["akka.remote.netty.tcp"] >>>>>> >>>>>> netty.tcp { >>>>>> hostname = 10.170.187.126 >>>>>> port = 2554 >>>>>> } >>>>>> } >>>>>> >>>>>> loggers = ["akka.event.slf4j.Slf4jLogger"] >>>>>> >>>>>> loglevel = "DEBUG" >>>>>> } >>>>>> The websocket server's application.conf is similar. >>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: >>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: >>>>>> https://groups.google.com/group/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Akka Team >>>>> Typesafe - Reactive apps on the JVM >>>>> Blog: letitcrash.com >>>>> Twitter: @akkateam >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Akka Team >>> Typesafe - Reactive apps on the JVM >>> Blog: letitcrash.com >>> Twitter: @akkateam >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Akka Team > Typesafe - Reactive apps on the JVM > Blog: letitcrash.com > Twitter: @akkateam > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
