Re: Metrics for Task States

2019-11-25 Thread Kelly Smith
with a side-car container once we move to Kubernetes. Kelly From: Caizhi Weng Date: Monday, November 25, 2019 at 1:41 AM To: Kelly Smith Cc: Piper Piper , "user@flink.apache.org" Subject: Re: Metrics for Task States Hi Kelly, As far as I know Flink currently does not have such metrics

Re: Metrics for Task States

2019-11-25 Thread Caizhi Weng
-- > *From:* Piper Piper > *Sent:* Friday, November 22, 2019 9:10:41 PM > *To:* Kelly Smith > *Cc:* user@flink.apache.org > *Subject:* Re: Metrics for Task States > > I am trying to reason why this problem should occur (i.e. why Flink could > not reject the job

Re: Metrics for Task States

2019-11-24 Thread Kelly Smith
is fixed. Get Outlook for Android<https://aka.ms/ghei36> From: Piper Piper Sent: Friday, November 22, 2019 9:10:41 PM To: Kelly Smith Cc: user@flink.apache.org Subject: Re: Metrics for Task States I am trying to reason why this problem should occur (i.e. why

Re: Metrics for Task States

2019-11-22 Thread Piper Piper
er >> observability). >> >> >> >> Does that make sense? >> >> >> >> Thanks, >> >> Kelly >> >> >> >> *From: *Piper Piper >> *Date: *Thursday, November 21, 2019 at 12:59 PM >> *To: *Kelly Smith >> *Cc: *

Re: Metrics for Task States

2019-11-21 Thread Piper Piper
*Date: *Thursday, November 21, 2019 at 12:59 PM > *To: *Kelly Smith > *Cc: *"user@flink.apache.org" > *Subject: *Re: Metrics for Task States > > > > Hello Kelly, > > > > I thought that Flink scheduler only starts a job if all requested > containers/TM

Re: Metrics for Task States

2019-11-21 Thread Kelly Smith
in each state for better observability). Does that make sense? Thanks, Kelly From: Piper Piper Date: Thursday, November 21, 2019 at 12:59 PM To: Kelly Smith Cc: "user@flink.apache.org" Subject: Re: Metrics for Task States Hello Kelly, I thought that Flink scheduler only starts a

Re: Metrics for Task States

2019-11-21 Thread Piper Piper
Hello Kelly, I thought that Flink scheduler only starts a job if all requested containers/TMs are available and allotted to that job. How can I reproduce your issue on Flink with YARN? Thank you, Piper On Thu, Nov 21, 2019, 1:48 PM Kelly Smith wrote: > I’ve been running Flink in production

Metrics for Task States

2019-11-21 Thread Kelly Smith
I’ve been running Flink in production on EMR (YARN) for some time and have found the metrics system to be quite useful, but there is one specific case where I’m missing a signal for this scenario: * When a job has been submitted, but YARN does not have enough resources to provide