Hi, Can you try the settings akka.remote.transport-failure-detector.heartbeat-interval = 1 s akka.remote.transport-failure-detector.acceptable-hearbeat-pause = 3 s
and see if the race still happens? This seems to be related to a test failure that we get from time to time but so far has been not able to reliably reproduce. If the frequency of the problem with the above settings reduces then there is a good chance that what you see is the same what we looked at before. -Endre On Thu, Nov 26, 2015 at 1:29 PM, jie tang <[email protected]> wrote: > Highly possible but not always. > > A typical flow: > 1 A websocket client connects to websocket server A > 2 The websocket server A creates a WebsocketActor for the client. The > WebsocketActor sends a ConnectionOpened message to businessserver via > ActorSelection. > 3 If the WebsocketActor can not receive the ConnectionRegistered message > in 10 seconds , it kills it itself and closes websocket connection between > the websocket client. > 4 Now the websocket client may repeat step 1 immediately. > > There are thousands of websocket clients connecting to websocket server > A. So when the business server restarts, the websocket server A may sends > hundreds of ConnectionOpened messages via ActorSelection at the same time. > There may be a race condition? > > > On Thu, Nov 26, 2015 at 8:16 PM, Akka Team <[email protected]> > wrote: > >> Hi, >> >> On Thu, Nov 26, 2015 at 1:13 PM, jie tang <[email protected]> wrote: >> >>> Thanks for your reply. >>> >>> But the two tcp connections were both initiated by websocket server A. >>> Is that valid? >>> >> >> No, *that* is not valid. It might happen for a short transient period, >> but not for long. This is likely a bug. Can you reproduce this reliably? >> >> -Endre >> >> >>> >>> And it seemed that websocket server A could never receive >>> ConnectionRegistered messages after business server restarted. I waited for >>> 10 minutes. At last I have to restart websocket server A. >>> >>> On Thu, Nov 26, 2015 at 6:45 PM, Akka Team <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> >>>> >>>>> But everything is ok for websocket server A. There was only a tcp >>>>> connection between it and businessserver. And it received >>>>> ConnectionRegistered messages. >>>>> >>>>> >>>>> What's wrong with websocket server B? Is it ok for two tcp connections >>>>> between websocket server B and business server? What I should do to avoid >>>>> it? >>>>> >>>> >>>> It is normal to have two TCP connections between Actor Systems. By >>>> default an ActorSystem tries to reuse incoming connections for outgoing >>>> messages. However, in cases where the two systems open the connections >>>> concurrently (for example you restart one of them, while the other tries to >>>> connect to it, so when it comes up they both try to connect) there might be >>>> two such connections alive. While there is a theoretical opportunity for >>>> the two systems to reconcile and agree on closing one of them, it is not a >>>> simple thing to do safely in practice so we didn't bother so far. >>>> >>>> You must also be prepared for temporary message losses between >>>> reconnects since Akka does not guarantee delivery and does not buffer >>>> messages forever. See >>>> http://doc.akka.io/docs/akka/2.4.0/scala/remoting.html#Lifecycle_and_Failure_Recovery_Model >>>> for details. >>>> >>>> -Endre >>>> >>>> >>>>> >>>>> >>>>> The netstat's output: >>>>> [root@dev-sig-server ~]# netstat -anp | grep 2554 >>>>> tcp 0 0 10.170.187.126:2554 0.0.0.0:* >>>>> LISTEN 4119/java >>>>> tcp 0 0 10.170.187.126:2554 10.162.209.21:32833 >>>>> ESTABLISHED 4119/java >>>>> tcp 0 0 10.170.187.126:2554 10.162.198.161:33326 >>>>> ESTABLISHED 4119/java >>>>> tcp 0 0 10.170.187.126:2554 10.162.209.21:32843 >>>>> ESTABLISHED 4119/java >>>>> >>>>> >>>>> We use akka-remote_2.10-2.3.12.jar >>>>> >>>>> business server's application.conf: >>>>> akka { >>>>> actor { >>>>> provider = "akka.remote.RemoteActorRefProvider" >>>>> } >>>>> remote { >>>>> log-received-messages = on >>>>> log-sent-messages = on >>>>> >>>>> enabled-transports = ["akka.remote.netty.tcp"] >>>>> >>>>> netty.tcp { >>>>> hostname = 10.170.187.126 >>>>> port = 2554 >>>>> } >>>>> } >>>>> >>>>> loggers = ["akka.event.slf4j.Slf4jLogger"] >>>>> >>>>> loglevel = "DEBUG" >>>>> } >>>>> The websocket server's application.conf is similar. >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Akka Team >>>> Typesafe - Reactive apps on the JVM >>>> Blog: letitcrash.com >>>> Twitter: @akkateam >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Akka Team >> Typesafe - Reactive apps on the JVM >> Blog: letitcrash.com >> Twitter: @akkateam >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Typesafe - Reactive apps on the JVM Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
