Re: [prometheus-users] Removing zombie rule groups metrics

2020-08-05 Thread Julien Pivotto
On 05 Aug 13:59, Peter S wrote: > Hi, > > After we remove some rule groups, their metrics still show up, like > prometheus_rule_group_last_duration_seconds > (exported by prometheus server's built-in exporter at the metrics endpoint) > > They won't vanish until we restart the prometheus processe

[prometheus-users] Removing zombie rule groups metrics

2020-08-05 Thread Peter S
Hi, After we remove some rule groups, their metrics still show up, like prometheus_rule_group_last_duration_seconds (exported by prometheus server's built-in exporter at the metrics endpoint) They won't vanish until we restart the prometheus processes (managed by systemd). I wonder if there is

Re: [prometheus-users] Label injection at exposition time (Python library)

2020-08-05 Thread Pau Freixes
[Seems that is doable, Ive ended up doing something like this ``` class MyRegistry(CollectorRegistry) def collect(self): """Yields metrics from the collectors in the registry.""" for metric in super().collect(): metric.samples = [ Sample(

[prometheus-users] Re: Helm Charts Migration to https://github.com/prometheus-community

2020-08-05 Thread Scott Rigby
In case anyone else runs across this, the convo has been moved to the prometheus-developers group, here: https://groups.google.com/g/prometheus-developers/c/Gh8GMscELao/m/2zZFPwdLBwAJ?pli=1 Thanks Naseem! On Tuesday, March 24, 2020 at 9:47:12 PM UTC-4 Naseem Ullah wrote: > To whom it may conc

[prometheus-users] Label injection at exposition time (Python library)

2020-08-05 Thread Pau Freixes
HI to all, Brian pointed to me to that link [1] which basically claims that in Prometheus - compared to other TimeSeries databases - there is no necessity of prefixing the names of the metrics. And TBH I like the idea of not having to prefix the metric name with for example the service name, I

Re: [prometheus-users] Getting Security Policy : frame-ancestors error , please suggest

2020-08-05 Thread sunil sagar
Hi Stuart , Thank you for help on this . One thing I forgot to mention . We have Thanos also in front of Prometheus , will it impact if enable SSL ? Thanks > On 6 Aug 2020, at 1:36 AM, Stuart Clark wrote: > > On 05/08/2020 17:54, sunils...@gmail.com wrote: >> Hi All, >> >> In our current

Re: [prometheus-users] Getting Security Policy : frame-ancestors error , please suggest

2020-08-05 Thread Stuart Clark
On 05/08/2020 17:54, sunils...@gmail.com wrote: Hi All, In our current setup Prometheus is non-ssl configured , which means I am accessing Prometheus as http://server-name:9090 During our recent Security scan below details were reported . Please advise , what step I can take for this . Mes

[prometheus-users] Getting Security Policy : frame-ancestors error , please suggest

2020-08-05 Thread sunils...@gmail.com
Hi All, In our current setup Prometheus is non-ssl configured , which means I am accessing Prometheus as http://server-name:9090 During our recent Security scan below details were reported . Please advise , what step I can take for this . Message : X-Frame-Options or Content-Security-Policy

Re: [prometheus-users] Retrieve Swarm informations with dockerswarm_sd_configs using "tasks" role

2020-08-05 Thread Tom Kun
Sure Config prometheus.yml cadvisor: #image: docker-registry.ju.globaz.ch:5000/cadvisor:0.30.2-globaz image: google/cadvisor ports: - 8080:8080 volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro

Re: [prometheus-users] Retrieve Swarm informations with dockerswarm_sd_configs using "tasks" role

2020-08-05 Thread Julien Pivotto
On 05 Aug 07:14, Tom Kun wrote: > The Prometheus does not seems to take the labels define in my Docker > compose service... > > x-common-labels: &label-monitoring > com.docker.swarm.prometheus-job: monitoring > > cadvisor: > #image: docker-registry.ju.globaz.ch:5000/cadvisor:0.30.2-glob

Re: [prometheus-users] Retrieve Swarm informations with dockerswarm_sd_configs using "tasks" role

2020-08-05 Thread Tom Kun
The Prometheus does not seems to take the labels define in my Docker compose service... x-common-labels: &label-monitoring com.docker.swarm.prometheus-job: monitoring cadvisor: #image: docker-registry.ju.globaz.ch:5000/cadvisor:0.30.2-globaz image: google/cadvisor ports: -

Re: [prometheus-users] How can I Send Individual Alert for Different Servers on Different usage Criteria

2020-08-05 Thread Pachha Gopi
Hi @Christian I am facing an issue that my alaert manager is not sending the alerts on regular intervals.how can i configure my alert manager to send alerts to my slack? On Wednesday, August 5, 2020 at 6:11:38 PM UTC+5:30, Christian Hoffmann wrote: > > Hi, > > On 8/5/20 2:40 PM, Pachha Gopi wr

Re: [prometheus-users] How can I Send Individual Alert for Different Servers on Different usage Criteria

2020-08-05 Thread Pachha Gopi
Hi thanks for reply yes i had gone through article and it was very userfful On Wednesday, August 5, 2020 at 6:11:38 PM UTC+5:30, Christian Hoffmann wrote: > > Hi, > > On 8/5/20 2:40 PM, Pachha Gopi wrote: > > I am Using Prometheus for my Production Servers,my question is is there > > any way

Re: [prometheus-users] Retrieve Swarm informations with dockerswarm_sd_configs using "tasks" role

2020-08-05 Thread Tom Kun
On Wednesday, 5 August 2020 14:29:16 UTC+2, Julien Pivotto wrote: > > On 05 Aug 05:24, Tom Kun wrote: > > Hi folks, > > > > I'm trying to retrieve metrics from different Swarm clusters into a > > Prometheus container which is deployed in an other Swarm cluster > dedicated > > to the monitor

Re: [prometheus-users] How can I Send Individual Alert for Different Servers on Different usage Criteria

2020-08-05 Thread Christian Hoffmann
Hi, On 8/5/20 2:40 PM, Pachha Gopi wrote: > I am Using Prometheus for my Production Servers,my question is is there > any way that we can configure different alerts for individual server. > for example I have 3 Servers ,Server1 cpu usage is 20% ,Server 2 cpu > usage is 30% and Server 3 cpu usage i

[prometheus-users] How can I Send Individual Alert for Different Servers on Different usage Criteria

2020-08-05 Thread Pachha Gopi
Hi All, I am Using Prometheus for my Production Servers,my question is is there any way that we can configure different alerts for individual server. for example I have 3 Servers ,Server1 cpu usage is 20% ,Server 2 cpu usage is 30% and Server 3 cpu usage is 90% .Now I need to get alert if serv

Re: [prometheus-users] Retrieve Swarm informations with dockerswarm_sd_configs using "tasks" role

2020-08-05 Thread Julien Pivotto
On 05 Aug 05:24, Tom Kun wrote: > Hi folks, > > I'm trying to retrieve metrics from different Swarm clusters into a > Prometheus container which is deployed in an other Swarm cluster dedicated > to the monitoring part of the entire infrastructure. > > I have actually setup the http through the

Re: [prometheus-users] Alertmanager: resolved message received immediately after warning message even resolve_timeout is 5m

2020-08-05 Thread leiwa...@gmail.com
Yes,my AlertManager is used together with Prometheus. In prometheus.yml: alerting: alertmanagers: - static_configs: - targets: ["x.x.x.x:9093"] For all the warning message i received, then immediately a resolved message received. This is not i want. Thanks, Lei 在2020年8月5日星期三 UTC

Re: [prometheus-users] Transmission of (Database) Table Data to Prometheus

2020-08-05 Thread Christian Hoffmann
On 8/3/20 8:34 AM, 'Píer Bauer' via Prometheus Users wrote: > Due to the fact that my query (in real world) contains several thousand > rows of output, I would like to pursue a generic approach to avoid > setting a separate PowerShell variable for each table cell data... > > > But currently I don

Re: [prometheus-users] Alertmanager: resolved message received immediately after warning message even resolve_timeout is 5m

2020-08-05 Thread Christian Hoffmann
On 8/5/20 10:21 AM, leiwa...@gmail.com wrote: > rules.yml > groups: > - name: network-delay >   rules: >   - alert: "network delay" >     expr: probe_duration_seconds * 1000 > 3000 >     for: 1s >     labels: >       severity: warning >       team: ops >     annotations: >       description: "{{$la

Re: [prometheus-users] Prometheus memory issue

2020-08-05 Thread Christian Hoffmann
Hi, On 8/4/20 12:24 PM, Vinod M V wrote: > >           I am facing Memory usage with Prometheus service and > Maintaining 30 days of data from Node exporter, Process exporter and JMX > exporter for 95 servers in Prometheus Database.  > >          Grafana and Prometheus are running on the same no

Re: [prometheus-users] Able to specify bind port, but not address

2020-08-05 Thread Christian Hoffmann
Hi, On 8/4/20 3:23 PM, jumble wrote: > Latest prometheus, on RHEL8. > > Observed behavior: bound to |127.0.0.1:9090| This sounds unexpected. Are you using the official binaries from prometheus.io / github? Can you share the exact logs from your experiments? Is it possible that you've got multi

Re: [prometheus-users] blackbox dns probe failed

2020-08-05 Thread Christian Hoffmann
Hi, On 8/4/20 10:54 AM, e huang wrote: > ts=2020-08-04T05:41:58.646Z caller=main.go:169 > module=dns_eboss.enmonster.com target=10.208.100.9 level=debug > msg="Error while sending a DNS query" err="read udp4 10.208.100. > 10:36709->10.208.100.9:53: i/o timeout" > ts=2020-08-04T05:41:58.646Z caller

Re: [prometheus-users] How to prevent sending resolve notification after resolve_timeout?

2020-08-05 Thread Christian Hoffmann
Hi, On 8/4/20 2:21 PM, shiqi chai wrote: > Hey guys,I have a problem with configuration of resolve_timeout. As > it means, a notication of resolved will be send after the timeout. > But actually the issue still be firing, it disturb the correct > resolved notification How can I prevent it? Not su

Re: [prometheus-users] Absence rules and target down

2020-08-05 Thread Federico Buti
Hi again. Well, thinking about it, that makes sense. I guess I'll revisit our setup in light of this aspect. Thanks a ton! --- Federico Buti On Wed, 5 Aug 2020 at 11:02, Brian Brazil wrote: > On Wed, 5 Aug 2020 at 09:56, Federico Buti wrote: > >> Hi. >> >> Thanks for the reply Brian. >> So o

Re: [prometheus-users] Absence rules and target down

2020-08-05 Thread Brian Brazil
On Wed, 5 Aug 2020 at 09:56, Federico Buti wrote: > Hi. > > Thanks for the reply Brian. > So one should not alert on absence of a metric? Never ever? Just on the > upness of the targets? > Generally you should alert on the absence of up, as that indicates something has either gone wrong with ser

Re: [prometheus-users] Absence rules and target down

2020-08-05 Thread Federico Buti
Hi. Thanks for the reply Brian. So one should not alert on absence of a metric? Never ever? Just on the upness of the targets? --- Federico Buti On Wed, 5 Aug 2020 at 10:46, Brian Brazil wrote: > On Wed, 5 Aug 2020 at 09:32, Federico Buti wrote: > >> Hi all. >> >> A few months ago we introdu

Re: [prometheus-users] Absence rules and target down

2020-08-05 Thread Brian Brazil
On Wed, 5 Aug 2020 at 09:32, Federico Buti wrote: > Hi all. > > A few months ago we introduced target down rules to keep track of targets > that were missing. The rules are relatively simple being something like e.g. > > alert: target_down_slower_scraping_jobs > expr: up{job=~"monitoring-script

Re: [prometheus-users] How to Optimize Prometheus High memory?

2020-08-05 Thread Stuart Clark
On 05/08/2020 09:37, Liu Chang wrote: [prometheus]# ./bin/prometheus --version prometheus, version 2.19.2 ./bin/prometheus --config.file=./conf/prometheus.yml --web.listen-address="0.0.0.0:8089" --storage.tsdb.retention.size=200GB *Killed process 203738 (prometheus) total-vm:241826816kB, an

[prometheus-users] Re: Prometheus memory issue

2020-08-05 Thread Liu Chang
Mark On Tuesday, August 4, 2020 at 6:24:38 PM UTC+8, Vinod M V wrote: > > Hi All , > > I am facing Memory usage with Prometheus service and Maintaining > 30 days of data from Node exporter, Process exporter and JMX exporter for > 95 servers in Prometheus Database. > > Grafana

[prometheus-users] How to Optimize Prometheus High memory?

2020-08-05 Thread Liu Chang
[prometheus]# ./bin/prometheus --version prometheus, version 2.19.2 ./bin/prometheus --config.file=./conf/prometheus.yml --web.listen-address="0.0.0.0:8089" --storage.tsdb.retention.size=200GB *Killed process 203738 (prometheus) total-vm:241826816kB, anon-rss:185585992kB, file-rss:0kB* We h

[prometheus-users] Absence rules and target down

2020-08-05 Thread Federico Buti
Hi all. A few months ago we introduced target down rules to keep track of targets that were missing. The rules are relatively simple being something like e.g. alert: target_down_slower_scraping_jobs expr: up{job=~"monitoring-scripts-5m|monitoring-scripts-hourly"} == 0 for: 13m labels:

[prometheus-users] Alertmanager: resolved message received immediately after warning message even resolve_timeout is 5m

2020-08-05 Thread leiwa...@gmail.com
rules.yml groups: - name: network-delay rules: - alert: "network delay" expr: probe_duration_seconds * 1000 > 3000 for: 1s labels: severity: warning team: ops annotations: description: "{{$labels.instance}} : {{ $value }}" alertmanager resolve_timeout is 5m S

Re: [prometheus-users] scrape_interval is not working properly.

2020-08-05 Thread Ben Kochie
You didn't provide any evidence of why you think it's not working. You need to include more information. On Tue, Aug 4, 2020, 17:25 Byungkwon Choi wrote: > Hello, > > I want to collect the HTTP requests per second every second. > To do so, I'm using Prometheus and Prometheus Adapter. > > I set t