just to close the thread. akka death watch was triggered by high GC pause, which is caused by memory leak in our code during Flink job restart.
noted that akka.ask.timeout wasn't related to akka death watch, which Flink has documented and linked. On Sat, Aug 26, 2017 at 10:58 AM, Steven Wu <stevenz...@gmail.com> wrote: > this is a stateless job. so we don't use RocksDB. > > yeah. network can also be a possibility. will keep it in the radar. > unfortunately, our metrics system don't have the tcp metrics when running > inside containers. > > On Fri, Aug 25, 2017 at 2:09 PM, Robert Metzger <rmetz...@apache.org> > wrote: > >> Hi, >> are you using the RocksDB state backend already? >> Maybe writing the state to disk would actually reduce the pressure on the >> GC (but of course it'll also reduce throughput a bit). >> >> Are there any known issues with the network? Maybe the network bursts on >> restart cause the timeouts? >> >> >> On Fri, Aug 25, 2017 at 6:17 PM, Steven Wu <stevenz...@gmail.com> wrote: >> >>> Bowen, >>> >>> Heap size is ~50G. CPU was actually pretty low (like <20%) when high GC >>> pause and akka timeout was happening. So maybe memory allocation and GC >>> wasn't really an issue. I also recently learned that JVM can pause for >>> writing to GC log for disk I/O. that is another lead I am pursuing. >>> >>> Thanks, >>> Steven >>> >>> On Wed, Aug 23, 2017 at 10:58 AM, Bowen Li <bowen...@offerupnow.com> >>> wrote: >>> >>>> Hi Steven, >>>> Yes, GC is a big overhead, it may cause your CPU utilization to >>>> reach 100%, and every process stopped working. We ran into this a while >>>> too. >>>> >>>> How much memory did you assign to TaskManager? How much the your >>>> CPU utilization when your taskmanager is considered 'killed'? >>>> >>>> Bowen >>>> >>>> >>>> >>>> On Wed, Aug 23, 2017 at 10:01 AM, Steven Wu <stevenz...@gmail.com> >>>> wrote: >>>> >>>>> Till, >>>>> >>>>> Once our job was restarted for some reason (e.g. taskmangaer container >>>>> got killed), it can stuck in continuous restart loop for hours. Right now, >>>>> I suspect it is caused by GC pause during restart, our job has very high >>>>> memory allocation in steady state. High GC pause then caused akka timeout, >>>>> which then caused jobmanager to think taksmanager containers are >>>>> unhealthy/dead and kill them. And the cycle repeats... >>>>> >>>>> But I hasn't been able to prove or disprove it yet. When I was asking >>>>> the question, I was still sifting through metrics and error logs. >>>>> >>>>> Thanks, >>>>> Steven >>>>> >>>>> >>>>> On Tue, Aug 22, 2017 at 1:21 AM, Till Rohrmann < >>>>> till.rohrm...@gmail.com> wrote: >>>>> >>>>>> Hi Steven, >>>>>> >>>>>> quick correction for Flink 1.2. Indeed the MetricFetcher does not >>>>>> pick up the right timeout value from the configuration. Instead it uses a >>>>>> hardcoded 10s timeout. This has only been changed recently and is already >>>>>> committed in the master. So with the next release 1.4 it will properly >>>>>> pick >>>>>> up the right timeout settings. >>>>>> >>>>>> Just out of curiosity, what's the instability issue you're observing? >>>>>> >>>>>> Cheers, >>>>>> Till >>>>>> >>>>>> On Fri, Aug 18, 2017 at 7:07 PM, Steven Wu <stevenz...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Till/Chesnay, thanks for the answers. Look like this is a >>>>>>> result/symptom of underline stability issue that I am trying to track >>>>>>> down. >>>>>>> >>>>>>> It is Flink 1.2. >>>>>>> >>>>>>> On Fri, Aug 18, 2017 at 12:24 AM, Chesnay Schepler < >>>>>>> ches...@apache.org> wrote: >>>>>>> >>>>>>>> The MetricFetcher always use the default akka timeout value. >>>>>>>> >>>>>>>> >>>>>>>> On 18.08.2017 09:07, Till Rohrmann wrote: >>>>>>>> >>>>>>>> Hi Steven, >>>>>>>> >>>>>>>> I thought that the MetricFetcher picks up the right timeout from >>>>>>>> the configuration. Which version of Flink are you using? >>>>>>>> >>>>>>>> The timeout is not a critical problem for the job health. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Till >>>>>>>> >>>>>>>> On Fri, Aug 18, 2017 at 7:22 AM, Steven Wu <stevenz...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> We have set akka.ask.timeout to 60 s in yaml file. I also >>>>>>>>> confirmed the setting in Flink UI. But I saw akka timeout of 10 s for >>>>>>>>> metric query service. two questions >>>>>>>>> 1) why doesn't metric query use the 60 s value configured in yaml >>>>>>>>> file? does it always use default 10 s value? >>>>>>>>> 2) could this cause heartbeat failure between task manager and job >>>>>>>>> manager? or is this jut non-critical failure that won't affect job >>>>>>>>> health? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Steven >>>>>>>>> >>>>>>>>> 2017-08-17 23:34:33,421 WARN >>>>>>>>> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher >>>>>>>>> - Fetching metrics failed. akka.pattern.AskTimeoutException: Ask >>>>>>>>> timed out on [Actor[akka.tcp://flink@1.2.3.4 >>>>>>>>> :39139/user/MetricQueryService_23cd9db754bb7d123d80e6b1c0be21d6]] >>>>>>>>> after [10000 ms] at akka.pattern.PromiseActorRef$$ >>>>>>>>> anonfun$1.apply$mcV$sp(AskSupport.scala:334) at >>>>>>>>> akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) at >>>>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) >>>>>>>>> at >>>>>>>>> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) >>>>>>>>> at >>>>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) >>>>>>>>> at >>>>>>>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:474) >>>>>>>>> at >>>>>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:425) >>>>>>>>> at >>>>>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:429) >>>>>>>>> at >>>>>>>>> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381) >>>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
