On Tue, Nov 19, 2019 at 6:33 PM Mateus Caruccio <mateus.caruc...@getupcloud.com> wrote: > > You must disable cluster-monitoring-operator since it will try to reconcile > the whole monitoring stack. > > $ oc scale --replicas=0 deploy/cluster-monitoring-operator
You'd need to disable the cluster version operator too IIRC and this has a bigger impact. > > Muting alerts using inhibit rules may have an unexpected side-effect as noted > by [1]. The recommended approach is to send alerts for a "blackhole" receiver > (rationale and example in the link) > > [1] > https://medium.com/@wrossmann/suppressing-informational-alerts-with-prometheus-and-alertmanager-4237feab7ce9 What I've described should work because source and target labels won't match the same alerts. Agreed that blackholing the notification is also a good solution. > > -- > Mateus Caruccio / Master of Puppets > GetupCloud.com > We make the infrastructure invisible > Gartner Cool Vendor 2017 > > > Em ter., 19 de nov. de 2019 às 13:27, Tim Dudgeon <tdudgeon...@gmail.com> > escreveu: >> >> No joy with that approach. I tried editing the ConfigMap and the CRD but >> both got reset when the cluster-monitoring-operator was restarted. >> >> Looks like I'll have to live with silencing the alert. >> >> On 19/11/2019 07:56, Vladimir REMENAR wrote: >> >> Hi Tim, >> >> You need to stop cluster-monitoring-operator than and then edit configmap. >> If cluster-monitoring-operator is running while editing configmap it will >> always revert it to default. >> >> >> Uz pozdrav, >> Vladimir Remenar >> >> >> >> From: Tim Dudgeon <tdudgeon...@gmail.com> >> To: Simon Pasquier <spasq...@redhat.com> >> Cc: users <users@lists.openshift.redhat.com> >> Date: 18.11.2019 17:46 >> Subject: Re: Changing Prometheus rules >> Sent by: users-boun...@lists.openshift.redhat.com >> ________________________________ >> >> >> >> The KubeAPILatencyHigh alert fires several times a day for us (on 2 >> different OKD clusters). >> >> On 18/11/2019 15:17, Simon Pasquier wrote: >> > The Prometheus instances deployed by the cluster monitoring operator >> > are read-only and can't be customized. >> > https://docs.openshift.com/container-platform/3.11/install_config/prometheus_cluster_monitoring.html#alerting-rules_prometheus-cluster-monitoring >> > >> > Can you provide more details about which alerts are noisy? >> > >> > On Mon, Nov 18, 2019 at 2:43 PM Tim Dudgeon <tdudgeon...@gmail.com> wrote: >> >> What is the "right" way to edit Prometheus rules that are deployed by >> >> default on OKD 3.11? >> >> I have alerts that are annoyingly noisy, and want to silence them forever! >> >> >> >> I tried editing the definition of the PrometheusRule CRD and/or the >> >> prometheus-k8s-rulefiles-0 ConfigMap in the openshift-monitoring project >> >> but my changes keep getting reverted back to the original. >> >> >> >> _______________________________________________ >> >> users mailing list >> >> users@lists.openshift.redhat.com >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > _______________________________________________ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users _______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
