Hi Oren,
1. We changed our internal scheduler to send heartbeat directly to log > instead of sending heartbeat as message > 2. We change our first in line actor to use PinnedDispacher instead of the > default one. > It should be the other way around, the actors doing the blocking calls should be on the PinnedDispatcher and the rest can run on the default one. If the blocking actors share the dispatcher with clustering for example it will just die. > > And yet a...It happened again, and when do we didn't not see any heartbeat > logs anymore which indicate it is not dispatcher issue. > If you use a scheduled task that does not send messages to actors, but for example prints to stdout, and still stops, then there is a different issue lurking there. If this is the case, and you verified it, then this is either a bug in the scheduler, or amazon virtual machines mess up CPU timers which is then an Amazon issue. -Endre > > Any idea? > > > On Monday, October 27, 2014 12:38:27 PM UTC+2, Akka Team wrote: >> >> Hi Oren, >> >> >> >> On Mon, Oct 27, 2014 at 7:22 AM, Oren Razon <[email protected]> wrote: >> >>> Hi, >>> I'm using akka 2.3.6 on a single node on Amazon m1.large instance. >>> I'm using akka system as a cluster (see my configuration file below). >>> My akka system include several parallel identical topologies. >>> Each topology start with an actor that act as a MQTT client at front >>> which send arrived massages after processing them to some other actors >>> (some of them written in scala and some in java). >>> When starting up the system everything seem to work well. >>> But after a couple of hours (usually ~8-10 hours) it seem that the MQTT >>> actor (the first in front) is still getting it's messages and work as >>> expected, but when sending them to its followers nothing happened. I do not >>> see dead letters, or any logging (applicative \ akka) that indicate >>> something regarding the actors (all heartbeats are stopped). >>> When looking in all my logs at debug level it seem that there was no >>> error at all, and that this scenario happen to all my topologies at the >>> exact same moment. >>> >>> Trying to debug it, we added an internal scheduler into the second actor >>> in line (the one that get messages from the MQTT actor) which send a "Tick" >>> message to himself every minute and print it. >>> As long as the system work well we see these "tick" messages, but when >>> the system is "down" we do not see anymore "tick" messages. >>> >>> >> This suspiciously looks like the common case when a dispatcher becomes >> stalled because blocking calls are consuming up all the threads in the >> underlying thread-pool. Do you have any blocking calls, or long-running >> computations in any of the receive blocks of your actors? Do you use >> Await.result()? >> >> -Endre >> >> >> >>> >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Akka Team >> Typesafe - The software stack for applications that scale >> Blog: letitcrash.com >> Twitter: @akkateam >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
