Re: [kubernetes-users] How to monitor/alert on container/pod death or restart

2018-08-08 Thread Rodrigo Campos
It really depends on the monitoring solution. Usually this metrics are exported and you can just predicate on them, in the language they provide. In my case, I'm using a hosted solution (signalfx) that gives you a daemon set and sends that metric to them. You can then predicate. We have alerts whe

Re: [kubernetes-users] How to monitor/alert on container/pod death or restart

2018-08-08 Thread Agrawal, Punit
Engineer, Lead New Product Development From: on behalf of Marcio Garcia Reply-To: "kubernetes-users@googlegroups.com" Date: Wednesday, August 8, 2018 at 2:16 PM To: "kubernetes-users@googlegroups.com" Subject: Re: [kubernetes-users] How to monitor/alert on container/pod deat

Re: [kubernetes-users] How to monitor/alert on container/pod death or restart

2018-08-08 Thread Marcio Garcia
David, In Datadog events you can see the killed pods. But, if you have containers that need to be killed because they don't die when receiving a stop, you'll see a lot of events like: KILLED, DESTROYED, and this is not necessarily an error, could be only a container being restarted, keep that

Re: [kubernetes-users] How to monitor/alert on container/pod death or restart

2018-08-08 Thread
Most of what you're asking for is available via the k8s API, if you watch it. On Wed, Aug 8, 2018 at 12:58 PM David Rosenstrauch wrote: > As we're getting ready to go to production with our k8s-based system, > we're trying to pin down exactly how we're going to do all the needed > monitoring/ale

Re: [kubernetes-users] How to monitor/alert on container/pod death or restart

2018-08-08 Thread David Rosenstrauch
Thanks for the response, Marcio. We've actually recently started using Datadog already. (At least in dev/qa.) But DD is a bit of a sea of metrics, and I'm not clear how we would accomplish one of the specific tasks I've mentioned - for example, alerting when k8s has killed a container or pod

Re: [kubernetes-users] How to monitor/alert on container/pod death or restart

2018-08-08 Thread Marcio Garcia
Hi David, You can use DataDog to achieve this. On 8/8/18, David Rosenstrauch wrote: > As we're getting ready to go to production with our k8s-based system, > we're trying to pin down exactly how we're going to do all the needed > monitoring/alerting for it. We can easily collect many of the