David, What we do is export the kubernetes cluster events to Cloud PubSub using Stackdriver Export and then we have SumoLogic setup to ingest logs from PubSub. Then we use the SumoLogic Scheduled Search Capabilities to send alerts based on certain events.
Punit Agrawal Site Reliability Engineer, Lead New Product Development From: <kubernetes-users@googlegroups.com> on behalf of Marcio Garcia <mar...@jauntvr.com> Reply-To: "kubernetes-users@googlegroups.com" <kubernetes-users@googlegroups.com> Date: Wednesday, August 8, 2018 at 2:16 PM To: "kubernetes-users@googlegroups.com" <kubernetes-users@googlegroups.com> Subject: Re: [kubernetes-users] How to monitor/alert on container/pod death or restart David, In Datadog events you can see the killed pods. But, if you have containers that need to be killed because they don't die when receiving a stop, you'll see a lot of events like: KILLED, DESTROYED, and this is not necessarily an error, could be only a container being restarted, keep that in mind. On Wed, Aug 8, 2018 at 6:06 PM David Rosenstrauch <dar...@darose.net<mailto:dar...@darose.net>> wrote: Thanks for the response, Marcio. We've actually recently started using Datadog already. (At least in dev/qa.) But DD is a bit of a sea of metrics, and I'm not clear how we would accomplish one of the specific tasks I've mentioned - for example, alerting when k8s has killed a container or pod. Any pointers on how I might go about setting up an alert like that? Thanks, DR On 08/08/2018 04:45 PM, Marcio Garcia wrote: > Hi David, > > You can use DataDog to achieve this. > > On 8/8/18, David Rosenstrauch <dar...@darose.net<mailto:dar...@darose.net>> > wrote: >> As we're getting ready to go to production with our k8s-based system, >> we're trying to pin down exactly how we're going to do all the needed >> monitoring/alerting for it. We can easily collect many of the metrics >> we need (using kube-state-metrics to feed into prometheus, and/or >> Datadog) and alert off of those. >> >> However, there's other important k8s-related info about our system that >> we need to be able to access, monitor, and alert on, most notably things >> like: >> >> * If a container crashes and is restarted by k8s >> How would would go about gaining access to those k8s-related events in >> an automated fashion, and setting up monitoring/alerting off of those? -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com<mailto:kubernetes-users%2bunsubscr...@googlegroups.com>. To post to this group, send email to kubernetes-users@googlegroups.com<mailto:kubernetes-users@googlegroups.com>. Visit this group at https://groups.google.com/group/kubernetes-users<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fgroup%2Fkubernetes-users&data=02%7C01%7Cpunagrawal%40ebay.com%7C14f4c2e2c8854cf652e808d5fd743787%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636693597941948105&sdata=BlRi1xZeCOiekVAoJxZM2rDxor0MxIMt1wmKjSpl08U%3D&reserved=0>. For more options, visit https://groups.google.com/d/optout<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&data=02%7C01%7Cpunagrawal%40ebay.com%7C14f4c2e2c8854cf652e808d5fd743787%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636693597941948105&sdata=sXzqGHblNRjinZHXFnds3KKYz5eFTaYo6bcaparLAJo%3D&reserved=0>. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com<mailto:kubernetes-users+unsubscr...@googlegroups.com>. To post to this group, send email to kubernetes-users@googlegroups.com<mailto:kubernetes-users@googlegroups.com>. Visit this group at https://groups.google.com/group/kubernetes-users<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fgroup%2Fkubernetes-users&data=02%7C01%7Cpunagrawal%40ebay.com%7C14f4c2e2c8854cf652e808d5fd743787%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636693597941958113&sdata=AgY3IhrrBHjNBAtaOEyqFVDvj4zq%2BQsvv%2BqxmRaJp8Y%3D&reserved=0>. For more options, visit https://groups.google.com/d/optout<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&data=02%7C01%7Cpunagrawal%40ebay.com%7C14f4c2e2c8854cf652e808d5fd743787%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636693597941968125&sdata=irdZ4KQYZhngy6SEGGQSE0liLdvUGNSNPtYWPYwjbgU%3D&reserved=0>. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
