The problem is fixed by increasing the heartbeat interval to 1000s and and heartbeat pause to 6000s according to the suggestion in spark community.
https://github.com/apache/spark/pull/2784/files akka.remote.transport-failure-detector { heartbeat-interval = 1000s acceptable-heartbeat-pause = 6000s } Thanks On Sunday, August 23, 2015 at 10:40:17 AM UTC+8, √ wrote: > > As a general rule, it is quicker to try the latest release and see if that > fixes the problem. > > On Sat, Aug 22, 2015 at 9:00 AM, <[email protected] <javascript:>> > wrote: > >> I encounter a problem where the actor seems to stop its execution because >> of Association failure issue. Below is the error message >> >> [WARN] ... >> [akka.tcp://MySystem@ip:port/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FMySystem%40ip%3Aport-0] >> >> Association with remote system [akka.tcp://MySystem@ip:port] has failed, >> address is now gated for [5000] ms. Reason is: [Disassociated]. >> >> After searching on this forum and the internet, some seems to be related >> to my problem. >> >> >> http://qnalist.com/questions/5582655/periodic-disassociated-with-remote-system >> >> https://groups.google.com/forum/#!searchin/akka-user/ >> "Association$20with$20remote$20system"$20-cluster/akka-user/kZFlfxRHnas/P6fDdk87NUsJ >> >> So I change heartbeat-interval to 10s and acceptable-heartbeat-pause to >> 25s >> >> akka { >> [... other setting ] >> remote { >> [... other setting] >> transport-failure-detector { >> heartbeat-interval = 10s # default 4s >> acceptable-heartbeat-pause = 25s # default 10s >> } >> netty.tcp { >> maximum-frame-size = 200000b >> [...other setting] >> } >> >> } >> } >> >> In addition, I have some messages (most of messages are smaller than >> default) larger than 128000b, so change maximum-frame-size value as well. >> But the system still throws that warning and my actor seems to just stop >> its original execution after that. >> >> My environment is akka 2.3.6 (including remote and actor), scala 2.11.x. >> >> How can I fix this problem? >> >> Thanks >> >> >> >> >> >> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Cheers, > √ > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
