Re: [prometheus-users] How to check exporter down time in 1 week

[sayf eddine Hammemi]

rate() was an example, dont use it in your case

On Tue, Jul 7, 2020 at 6:21 AM Jack Chew  wrote:

> I try rate(up{idc="azure"}[5m] offset 1w) and then all value is 0. I need
> check the node within 1 week down time(up==0).
>
> 在 2020年7月6日星期一 UTC+8下午3:23:56，sayf eddine Hammemi写道：
>>
>> Hello, you can use offset to go back in time given you have enabled data
>> retention for at least one week
>> Exp: rate(http_requests_total[5m] offset 1w) (change it accordingly to
>> your metrics)
>>
>> On Mon, Jul 6, 2020 at 9:18 AM Jack Chew  wrote:
>>
>>> Hey guys,
>>>
>>>
>>>How i check up == 0 in 1week accourd  by instacne.
>>>
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/6127ffce-02ad-48ed-b70e-b5d27daf1647o%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/42464747-6a2d-47bb-a174-3b81840c7278o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiDNCUW%3Da34aVB55vQdNvvOSyNY9wOwicDUDORoUDaE7w%40mail.gmail.com.

Re: [prometheus-users] How to check exporter down time in 1 week

[Jack Chew]

I try rate(up{idc="azure"}[5m] offset 1w) and then all value is 0. I need 
check the node within 1 week down time(up==0).

在 2020年7月6日星期一 UTC+8下午3:23:56，sayf eddine Hammemi写道：
>
> Hello, you can use offset to go back in time given you have enabled data 
> retention for at least one week
> Exp: rate(http_requests_total[5m] offset 1w) (change it accordingly to 
> your metrics)
>
> On Mon, Jul 6, 2020 at 9:18 AM Jack Chew > 
> wrote:
>
>> Hey guys,
>>
>>
>>How i check up == 0 in 1week accourd  by instacne.
>>
>>
>> Thanks
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to promethe...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/6127ffce-02ad-48ed-b70e-b5d27daf1647o%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/42464747-6a2d-47bb-a174-3b81840c7278o%40googlegroups.com.

[prometheus-users] Where and How to declare the ---storage.tsdb.retention.time=90d flag

[Anirudh Pasalapudi]

Hello,


I deployed prometheus using automation and I used a shell file to run the 
configuration file of prometheus. The contents of the file are mentioned 
below. However currently the default storage retention period is 15d. I 
want this to be changed to 90d. I tried to declare the flag 
--storage.tsdb.retention.time=90d in the prometheus.yml file and also in 
the below mentioned bash file. When I do that application is not coming up. 
If I have to add that flag in the below file, where exactly I need to 
declare that flag and how can I do that. Can anyone please help.


#!/bin/bash

prometheus_binary=/apps/prometheus/prometheus-2.19.0.linux-amd64/prometheus
prometheus_pid=/var/run/prometheus.pid

prometheus_flags=--config.file=/apps/prometheus/prometheus-2.19.0.linux-amd64/prometheus.yml

case "$1" in
start)
if [ -f "$prometheus_binary" ]; then
$prometheus_binary $prometheus_flags> /dev/null 2>&1 &
echo $! >$prometheus_pid;
echo $prometheus_flags
fi
;;
stop)
PID=$(cat $prometheus_pid);
kill -9 $PID;
rm -rf $prometheus_pid
;;
restart)
  if [ -f $prometheus_pid ]; then
PID=$(cat $prometheus_pid);
  kill -9 $PID;
  $prometheus_binary $prometheus_flags > /dev/null 2>&1 &
  echo $! >$prometheus_pid
  else
$prometheus_binary -i $prometheus_flags > /dev/null 2>&1 &
echo $! >$prometheus_pid
  fi
exit 0
  ;;
*)
echo "Usage: $0 {start|stop|restart}"
  exit 1
esac
exit 0

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a1f92e94-08e6-46e7-9f0f-53fa7aaf6ff4o%40googlegroups.com.

[prometheus-users] Re: No joy from Prometheus snapshot

[Mike Spreitzer]

Interestingly, the Prometheus server running against the new snapshot 
logged some compactions after about a minute.

...
level=info ts=2020-07-07T01:57:08.694Z caller=main.go:646 msg="Server is 
ready to receive web requests."
level=info ts=2020-07-07T01:58:18.185Z caller=compact.go:441 component=tsdb 
msg="compact blocks" count=3 mint=159401520 maxt=159403680 ulid=
01ECKFX8HKYEP8Y2YPX2WB0PC3 sources="[01ECHNMH8MRPZYHM8KXKBE60ZA 
01ECHWG8GCAZWMW1JDZB9ERQ0N 01ECJ3BZRM3SP7XQAZRJ867C30]" duration=
9.494006165s
level=info ts=2020-07-07T01:58:27.860Z caller=compact.go:441 component=tsdb 
msg="compact blocks" count=3 mint=159403680 maxt=159405840 ulid=
01ECKFXHV47D51X0GREQM6QZE3 sources="[01ECJA7Q18RT0VW9S5HKSVDP0J 
01ECJH3E8ENEWVK8E41G5XWM71 01ECJQZ5GB6TGV21QMHCWZ3JFY]" duration=9.64782793s
level=info ts=2020-07-07T01:58:37.619Z caller=compact.go:441 component=tsdb 
msg="compact blocks" count=3 mint=159405840 maxt=159408000 ulid=
01ECKFXV9PSG91VCTAMAVKZ1YR sources="[01ECJYTWRJZG6YSGH3XBTJK6CX 
01ECK5PQETDE8VDRSR6NZ6FGXQ 01ECKCJEPAJ19AJ7DXJY6K6P6B]" duration=
9.725054346s
level=info ts=2020-07-07T01:58:53.953Z caller=compact.go:441 component=tsdb 
msg="compact blocks" count=3 mint=159401520 maxt=159408000 ulid=
01ECKFY4TFCY2J3KQ30VSZFRKS sources="[01ECKFX8HKYEP8Y2YPX2WB0PC3 
01ECKFXHV47D51X0GREQM6QZE3 01ECKFXV9PSG91VCTAMAVKZ1YR]" duration=
16.305728942s

This server logged finding 10 blocks when it started (see my previous 
email); those compactions compacted the first 9 blocks into one.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/67ceeb43-6e7c-43b4-96a1-519d7405c6dao%40googlegroups.com.

Re: [prometheus-users] JMX exporter problem, can't run process

[Wesley]

what's the log and error message?

Jacek 1974 wrote:

Anybody?


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6e1ae167-1c31-49d9-0f34-cda03ca2e885%40freenetMail.de.

[prometheus-users] JMX exporter problem, can't run process

[Jacek 1974]

Anybody?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c055ca5b-8dfe-4441-9396-639cc0697650o%40googlegroups.com.

[prometheus-users] Re: No joy from Prometheus snapshot

[Mike Spreitzer]

I suspected the relative directory and then noticed the double equal in the 
prometheus command line.  So I erased the old snapshot and tried again.

sysop@r26data0:/var/lib/prometheus/snapshots$ ls
sysop@r26data0:/var/lib/prometheus/snapshots$ curl -X POST http:
//localhost:30909/api/v1/admin/tsdb/snapshot
{"status":"success","data":{"name":"20200707T015331Z-b7fbfbcafd915bb"}}
sysop@r26data0:/var/lib/prometheus/snapshots$ 
sysop@r26data0:/var/lib/prometheus/snapshots$ ls -la
total 12
drwxr-xr-x  3 nobody nogroup 4096 Jul  7 01:53 .
drwxr-xr-x 14 nobody   65533 4096 Jul  7 00:59 ..
drwxr-xr-x 12 nobody nogroup 4096 Jul  7 01:53 20200707T015331Z-
b7fbfbcafd915bb

Next I run the server again.  This time it logs messages about finding data 
blocks.

sysop@r26data0:/var/lib/prometheus/snapshots/20200707T015331Z-b7fbfbcafd915bb$ 
sudo -u nobody ~/prometheus --storage.tsdb.path=$PWD 
 --web.enable-admin-api --config.file=$HOME/prom-config/config.yaml
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:302 msg="No time or 
size retention was set so using the default time retention" duration=15d
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:337 msg="Starting 
Prometheus" version="(version=2.19.1, branch=HEAD, 
revision=eba3fdcbf0d378b66600281903e3aab515732b39)"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:338 
build_context="(go=go1.14.4, 
user=root@62700b3d0ef9, date=20200618-16:35:26)"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:339 host_details="(Linux 
4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 
r26data0 (none))"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:340 
fd_limits="(soft=65535, 
hard=65535)"
level=info ts=2020-07-07T01:57:08.662Z caller=main.go:341 
vm_limits="(soft=unlimited, 
hard=unlimited)"
level=info ts=2020-07-07T01:57:08.664Z caller=main.go:678 msg="Starting 
TSDB ..."
level=info ts=2020-07-07T01:57:08.664Z caller=web.go:524 component=web 
msg="Start 
listening for connections" address=0.0.0.0:9090
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159401520 maxt=159402240 ulid=
01ECHNMH8MRPZYHM8KXKBE60ZA
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159402240 maxt=159402960 ulid=
01ECHWG8GCAZWMW1JDZB9ERQ0N
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159402960 maxt=159403680 ulid=
01ECJ3BZRM3SP7XQAZRJ867C30
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159403680 maxt=159404400 ulid=
01ECJA7Q18RT0VW9S5HKSVDP0J
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159404400 maxt=159405120 ulid=
01ECJH3E8ENEWVK8E41G5XWM71
level=info ts=2020-07-07T01:57:08.664Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159405120 maxt=159405840 ulid=
01ECJQZ5GB6TGV21QMHCWZ3JFY
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159405840 maxt=159406560 ulid=
01ECJYTWRJZG6YSGH3XBTJK6CX
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159406560 maxt=159407280 ulid=
01ECK5PQETDE8VDRSR6NZ6FGXQ
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159407280 maxt=159408000 ulid=
01ECKCJEPAJ19AJ7DXJY6K6P6B
level=info ts=2020-07-07T01:57:08.665Z caller=repair.go:59 component=tsdb 
msg="Found healthy block" mint=159408000 maxt=1594086829277 ulid=
01ECKFMSZ5XDWD5296R3C8ZXZ6
level=info ts=2020-07-07T01:57:08.688Z caller=head.go:645 component=tsdb msg
="Replaying WAL and on-disk memory mappable chunks if any, this may take a 
while"
level=info ts=2020-07-07T01:57:08.688Z caller=head.go:706 component=tsdb msg
="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2020-07-07T01:57:08.688Z caller=head.go:709 component=tsdb msg
="WAL replay completed" duration=365.54µs
level=info ts=2020-07-07T01:57:08.690Z caller=main.go:694 fs_type=
EXT4_SUPER_MAGIC
level=info ts=2020-07-07T01:57:08.690Z caller=main.go:695 msg="TSDB started"
level=info ts=2020-07-07T01:57:08.690Z caller=main.go:799 msg="Loading 
configuration file" filename=/home/sysop/prom-config/config.yaml
level=info ts=2020-07-07T01:57:08.694Z caller=main.go:827 msg="Completed 
loading of configuration file" filename=/home/sysop/prom-config/config.yaml
level=info ts=2020-07-07T01:57:08.694Z caller=main.go:646 msg="Server is 
ready to receive web requests."

The last data block ends about 5 minutes ago.

sysop@r26data0:~$ date --date @1594086829
Tue Jul  7 01:53:49 UTC 2020

But still no data.

sysop@r26data0:~$ curl http://localhost:9090/api/v1/metadata
{"status":"success","data":{}}sysop@r26data0:~$ 

The web UI shows the following for "/status".

Runtime 

[prometheus-users] prometheus remote_read doesn't work if external_labels added after metric wrote to influxdb

[艾客]

my question:* thanos can't query prometheus old data which have to read 
from influxdb by  remote_read*

env:
prometheus version=2.6.0
thanos, version 0.12.2

config:

scrape_configs:
  - job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'


params:
  'match[]':
- '{__name__=~".+"}'


static_configs:
  - targets:
- '10.10.44.20:9090'


*Things come from this*
I made a new prometheus cluster with thanos ,but i don't want to delete my 
historical data which stored in influxdb. So i added thanos-sidecar for my 
old prometheus ,also with external_labels configs. but it doesn't work. 
I found the reason was *prometheus query influxdb with the label 
external_labels ,if i removed external_labels, it work, but if to,the 
thanos query will not deduplicat for this prometheus.*

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/03c6af50-c511-4d62-9670-04f80e3c29efo%40googlegroups.com.

[prometheus-users] aws_elb_un_healthy_host_count_average incorrect value

['Roman Vogman' via Prometheus Users]

Currently, we have a total of 10 unhealthy hosts under a specific ELB, 
hosts are in different AZs.

aws_elb_un_healthy_host_count_average metric returns 10 unhealthy hosts for 
each AZ:

aws_elb_un_healthy_host_count_average{load_balancer_name="foo",availability_zone="us-west-2c",}
 
10.0 
aws_elb_un_healthy_host_count_average{load_balancer_name="foo",availability_zone="us-west-2b",}
 
10.0 
aws_elb_un_healthy_host_count_average{load_balancer_name="foo",availability_zone="us-west-2a",}
 
10.0

therefore, sum by (load_balancer_name) 
(aws_elb_un_healthy_host_count_average{load_balancer_name="foo"}) returns 
30 unhealthy hosts, which is incorrect (should be 10)

On the other hand, aws_elb_healthy_host_count_average returns just one 
value, without availability_zone label, and therefore returns a correct 
number.


Using cloudwatch_exporter-0.8.0 image 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/27564926-cc9a-4b56-9fcc-014a1dfdb58en%40googlegroups.com.

[prometheus-users] No datapoint after upgrading to kubernetes 1.16.8 (aks)

[dinor geler]

Hello all , i was redirected from github to here , appreciate the help on 
the subject below :

What did you do? upgraded my cluster to 1.16.8

What did you expect to see? to see graphs of the rules i have.

What did you see instead? Under which circumstances? am not able to see 
some metrics like :
sum by(pod_name) 
(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
 
* 1000
i get no datapoints found ,with 1.15.11 cluster ,it is working.

aks on azure 1.16.8

   - 
   
   System information:
   
   insert output of uname -srm here
   - 
   
   Prometheus version:
   
   insert output of prom/prometheus:v2.19.2prom/prometheus:v2.19.2 here
   - 
   
   Alertmanager version:
   
   insert output of alertmanager --version here (if relevant to the issue)
   - 
   
   Prometheus configuration file:
   
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this 
file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  alerting_rules.yml: |
{}
  alerts: |
{}
  prometheus.yml: |
global:
  evaluation_interval: 1m
  scrape_interval: 1m
  scrape_timeout: 10s
remote_write:
- url: http://prom-mdm-converter/receive

rule_files:
- /etc/config/recording_rules.yml
- /etc/config/alerting_rules.yml
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
- localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-apiservers
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
regex: default;kubernetes;https
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_service_name
- __meta_kubernetes_endpoint_port_name
  scheme: https
  tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-nodes
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
regex: __meta_kubernetes_node_label_(.+)
  - replacement: kubernetes.default.svc:443
target_label: __address__
  - regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
  scheme: https
  tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  job_name: kubernetes-nodes-cadvisor
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - action: labelmap
regex: __meta_kubernetes_node_label_(.+)
  - replacement: kubernetes.default.svc:443
target_label: __address__
  - regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
  scheme: https
  tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- job_name: kubernetes-service-endpoints
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
  - action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
  - action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
  - action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
  - action: labelmap
regex: __meta_kubernetes_service_label_(.+)
  - action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
  - action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_name
  - action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: kubernetes_node
- job_name: kubernetes-service-endpoints-slow
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
  - action: replace

[prometheus-users] Making TCP Checks Critical based on Response Time.

[yagyans...@gmail.com]

Hi. On some of my services I have setup some TCP Port Checks using 
blackbox. Now, I want those checks to be critical based on the Response 
Time of those TCP Checks. Is it possible to do so?

My TCP job.
  - job_name: 'blackbox_Service-TCPChecks'
scrape_timeout: 10s
metrics_path: /probe
params:
  module: [tcp_connect]
file_sd_configs:
  - files:
  - /etc/blackbox/HTTP_TCP-Targets/TCP_Targets.yml

This is how I am defining targets.
- targets:
 - x.x.x.x:80
  labels:
 checkname: 'myname'
 cluster: 'C1'
 node: 'N1'

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2c31e6f3-e06c-479f-a960-ff95a3d12bdan%40googlegroups.com.

[prometheus-users] mysqld-exporter mysql_heartbeat_lag_seconds and server_id label - HOW?

[Stefan Szebinski]

mysqld_exporter contains some useful code for a check on heartbeat lagging

groups:
- name: example.rules
  rules:
  - record: mysql_heartbeat_lag_seconds
expr: mysql_heartbeat_now_timestamp_seconds - 
mysql_heartbeat_stored_timestamp_seconds
   ...
  - alert: MySQLReplicationLag
expr: (mysql_heartbeat_lag_seconds > 30) and ON(instance) (
predict_linear(mysql_heartbeat_lag_seconds[5m],
  60 * 2) > 0)


Now, in my case the master server_id may change due to the way we operate 
our MySQL cluster, and hence, we may get the following metrics

{instance="batchdb001.mo-staging99-nonprod.dus1.cloud",job=
"prometheus-mysqld-exporter",server_id="2001500"} 0.5187849998474121
{instance="batchdb001.mo-staging99-nonprod.dus1.cloud",job=
"prometheus-mysqld-exporter",server_id="3212"}1594051555.519615


As you can see, for one instance there's multiple metrics only one of which 
is the right one as it refers to the correct server_id. In principle, it's 
easy to determine the correct one as there's also a 
metric mysql_slave_status_master_server_id which returns the correct 
server_id:

mysql_slave_status_master_server_id{instance=
"batchdb001.mo-staging99-nonprod.dus1.cloud",job=
"prometheus-mysqld-exporter",master_host="dbmaster001",master_uuid=
"005e9c3d-baea-11ea-ab06-027e6d15fde3"}. 2001500

so for the alert definition I would have to take into account the server_id:

- alert: MySQLReplicationLag
expr: (mysql_heartbeat_lag_seconds{server_id="2001500"} > 30) and ON(
instance) ...

but how to do this in my case, where server_id has to be compared with a 
metrics value (mysql_slave_status_master_server_id)?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/cca8ab4b-eae3-4c54-be79-ef1137e6a052o%40googlegroups.com.

[prometheus-users] Alertmanager webhook timeout

[Murali Krishna Kanagala]

Hello everyone,

Can someone help  me on how can
I change the webhook timeout settings for a webhook?
Also in an alert route if AM is waiting for a webhook response does it
holds of the rest of the pipeline until that

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKimyZuZzGuD2zc8Ygnz3D-_TktBWvrcQ2LWbQq4EO3DyADvxg%40mail.gmail.com.

Re: [prometheus-users] How Protected Prometheus (OpenID Auth enaled) can be use as target in federation Prometheus scrape config

[Aliaksandr Valialkin]

Prometheus supports basic auth
 and/or mutual
TLS  for scraping
targets - see `basic_auth` and `tls_config` sections in
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
for
details.

On Mon, Jul 6, 2020 at 1:44 PM chandan kashayp 
wrote:

> What would be the other auth methods suitable in my case. I didn't find
> any docs which relate like how federate Prometheus can access slave targets
> if it have some auth is involved.
>
> On Monday, 6 July 2020 16:05:42 UTC+5:30, Stuart Clark wrote:
>>
>> On 2020-07-06 11:17, chandan kashayp wrote:
>> > Hello Guys,
>> >
>> > I am stuck at point by doing integration of openid auth enabled
>> > prometheus to federation. Let me explain in detail about the
>> > configuration and blocker.
>> >
>> > My slave Prometheus is openid auth enabled. Whenever, We try to access
>> > the prometheus it ask for login authentication and get us IN if
>> > authorization get success. Post successful login, prometheus dashboard
>> > and its graph can be see.
>> >
>> > Federation prometheus is running at different place. Federation
>> > prometheus scrape_configs looks like below
>> >
>> > scrape_configs:
>> >
>> > *
>> >
>> > job_name: 'federate'
>> > scrape_interval: 15s
>> >
>> > honor_labels: true
>> > metrics_path: '/federate'
>> >
>> > params:
>> > 'match[]':
>> > - '{job="prometheus"}'
>> > - '{name=~"job:.*"}'
>> >
>> > static_configs:
>> >
>> > * targets:
>> >
>> > * 'prometheus-slave.xyz.com:443' (my slave prometheus
>> endpoint)
>> >
>> > Issue : The target status is DOWN and Status is "server returned HTTP
>> > status 403 Forbidden".
>> >
>> > I know the error is coming because of federation Prometheus has not
>> > getting credential to access slave Prometheus. But, I am not getting
>> > anything at federation prometheus where credentials related
>> > configuration will be done which allow federation Prometheus to access
>> > auth protected slave prometheus.
>> >
>> > Suggestion & help need !!
>> > #FederationPrometheus
>> >
>>
>> I don't believe Prometheus supports OIDC authenticaiton, so you would
>> need to allow other authentication or whitelisting methods for your
>> federation. OIDC is really best suited for people, with other forms
>> better for machines.
>>
>> --
>> Stuart Clark
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/cb2e02f3-c6af-47c6-9029-d1c5f0b66c9do%40googlegroups.com
> 
> .
>


-- 
Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPbKnmDkaRFaue6EvZyEYDboQxExRXdFDdBuJpkA%3DH2-JzUPvw%40mail.gmail.com.

Re: [prometheus-users] Prometheus query using variables from grafana templates having spaces doesn't work

[sayf eddine Hammemi]

Hello, did you try strict equality instead of regex?

On Mon, Jul 6, 2020 at 5:01 PM Debashish Ghosh 
wrote:

> Hi,
>I have created a grafana template varaible to extract names of
> organizations from Prometheus db using query
>
> label_values(custom_message_volume_endpoint_organization_total,Organization).
>
> This yields organization names that have spaces in some cases .. For
> example
> OrgA
> OrgB product1
> OrgC product2
>
> I use the following query to extract the timeseries corresponding to an
> organization..
>
> custom_latency_endpoint_organization_total{job="MyJob",Organization=~"$Organization"}
>
> It only works when I select the variable value OrgA but doesn't work for
> the other 2 . Apparently any organization name having space is discarded.
> Is there a workaround to get around this ?
>
> Thanks
> Debashish
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c0fe6136-8ce4-4309-8516-0826cc3f12fbo%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhUo_RZAxqBqK2jg6X5%2BNkfrFYQtKnv-NdXakkA4KrjGQ%40mail.gmail.com.

[prometheus-users] Prometheus query using variables from grafana templates having spaces doesn't work

[Debashish Ghosh]

Hi,
   I have created a grafana template varaible to extract names of 
organizations from Prometheus db using query 
label_values(custom_message_volume_endpoint_organization_total,Organization).

This yields organization names that have spaces in some cases .. For example
OrgA
OrgB product1
OrgC product2

I use the following query to extract the timeseries corresponding to an 
organization..
custom_latency_endpoint_organization_total{job="MyJob",Organization=~"$Organization"}

It only works when I select the variable value OrgA but doesn't work for 
the other 2 . Apparently any organization name having space is discarded.
Is there a workaround to get around this ?

Thanks
Debashish

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c0fe6136-8ce4-4309-8516-0826cc3f12fbo%40googlegroups.com.

Re: [prometheus-users] Apache exporter

[sayf eddine Hammemi]

Given the impossibility of multiple processes listening on the same port on
the same server, you need to add a proxy service that will collect the
metrics from all the servers and expose it in one endpoint. You need also
to worry about is the unicity of metrics because all apache will be
exposing the same metric name you need a way to inject a unique label per
apache server before pushing them into the tsdb.  tools like
https://github.com/rebuy-de/exporter-merger will help but I don't see how
you can add a unique label with them.
The real XY problem here is why would want to run many apache servers in
one machine that is counter-intuitive.

On Mon, Jul 6, 2020 at 4:35 PM Yasmine Mbarek  wrote:

> I have more than one apache server on the same machine working on
> different ports , is there any way to monitor all of them with only one
> service ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/69a40259-2f21-41d5-8fe5-80f12dbe5db2o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiJJv0TuBhH4_mpp6gqcPOaosUhg7BeL91x89_pOxG5iA%40mail.gmail.com.

[prometheus-users] Apache exporter

[Yasmine Mbarek]

I have more than one apache server on the same machine working on different 
ports , is there any way to monitor all of them with only one service ?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/69a40259-2f21-41d5-8fe5-80f12dbe5db2o%40googlegroups.com.

Re: [prometheus-users] Re: Prometheus as Windows Service

[Eivind Gussiås Løkseth]

I found this thread after first finding the GitHub issue where the question 
was asked initially. I would like to add a comment to that issue to link 
directly to this tread, but unfortunately the issue is locked and only 
participants can add more comments to it. So maybe you could, @thebitt? 
This is the link to add to issue #3881 
: 
https://groups.google.com/d/msg/prometheus-users/NjtTA7vsVJ8/Nq7tOHtDAAAJ

tirsdag 21. mai 2019 12.48.26 UTC+2 skrev thebitt...@gmail.com følgende:
>
> Thanks! I tried to understand what that parameter means.
> Everything is fine now
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/549b9eb2-0481-435e-b52a-a5e7664299fbo%40googlegroups.com.

Re: [prometheus-users] How Protected Prometheus (OpenID Auth enaled) can be use as target in federation Prometheus scrape config

[chandan kashayp]

What would be the other auth methods suitable in my case. I didn't find any 
docs which relate like how federate Prometheus can access slave targets if 
it have some auth is involved.

On Monday, 6 July 2020 16:05:42 UTC+5:30, Stuart Clark wrote:
>
> On 2020-07-06 11:17, chandan kashayp wrote: 
> > Hello Guys, 
> > 
> > I am stuck at point by doing integration of openid auth enabled 
> > prometheus to federation. Let me explain in detail about the 
> > configuration and blocker. 
> > 
> > My slave Prometheus is openid auth enabled. Whenever, We try to access 
> > the prometheus it ask for login authentication and get us IN if 
> > authorization get success. Post successful login, prometheus dashboard 
> > and its graph can be see. 
> > 
> > Federation prometheus is running at different place. Federation 
> > prometheus scrape_configs looks like below 
> > 
> > scrape_configs: 
> > 
> > * 
> > 
> > job_name: 'federate' 
> > scrape_interval: 15s 
> > 
> > honor_labels: true 
> > metrics_path: '/federate' 
> > 
> > params: 
> > 'match[]': 
> > - '{job="prometheus"}' 
> > - '{name=~"job:.*"}' 
> > 
> > static_configs: 
> > 
> > * targets: 
> > 
> > * 'prometheus-slave.xyz.com:443' (my slave prometheus endpoint) 
> > 
> > Issue : The target status is DOWN and Status is "server returned HTTP 
> > status 403 Forbidden". 
> > 
> > I know the error is coming because of federation Prometheus has not 
> > getting credential to access slave Prometheus. But, I am not getting 
> > anything at federation prometheus where credentials related 
> > configuration will be done which allow federation Prometheus to access 
> > auth protected slave prometheus. 
> > 
> > Suggestion & help need !! 
> > #FederationPrometheus 
> > 
>
> I don't believe Prometheus supports OIDC authenticaiton, so you would 
> need to allow other authentication or whitelisting methods for your 
> federation. OIDC is really best suited for people, with other forms 
> better for machines. 
>
> -- 
> Stuart Clark 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/cb2e02f3-c6af-47c6-9029-d1c5f0b66c9do%40googlegroups.com.

Re: [prometheus-users] How Protected Prometheus (OpenID Auth enaled) can be use as target in federation Prometheus scrape config

[Stuart Clark]

On 2020-07-06 11:17, chandan kashayp wrote:

Hello Guys,

I am stuck at point by doing integration of openid auth enabled
prometheus to federation. Let me explain in detail about the
configuration and blocker.

My slave Prometheus is openid auth enabled. Whenever, We try to access
the prometheus it ask for login authentication and get us IN if
authorization get success. Post successful login, prometheus dashboard
and its graph can be see.

Federation prometheus is running at different place. Federation
prometheus scrape_configs looks like below

scrape_configs:

*

job_name: 'federate'
scrape_interval: 15s

honor_labels: true
metrics_path: '/federate'

params:
'match[]':
- '{job="prometheus"}'
- '{name=~"job:.*"}'

static_configs:

* targets:

* 'prometheus-slave.xyz.com:443' (my slave prometheus endpoint)

Issue : The target status is DOWN and Status is "server returned HTTP
status 403 Forbidden".

I know the error is coming because of federation Prometheus has not
getting credential to access slave Prometheus. But, I am not getting
anything at federation prometheus where credentials related
configuration will be done which allow federation Prometheus to access
auth protected slave prometheus.

Suggestion & help need !!
#FederationPrometheus



I don't believe Prometheus supports OIDC authenticaiton, so you would 
need to allow other authentication or whitelisting methods for your 
federation. OIDC is really best suited for people, with other forms 
better for machines.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/973cc5a17a30e12db041276295615bdc%40Jahingo.com.

[prometheus-users] How Protected Prometheus (OpenID Auth enaled) can be use as target in federation Prometheus scrape config

[chandan kashayp]

Hello Guys,

I am stuck at point by doing integration of openid auth enabled prometheus 
to federation. Let me explain in detail about the configuration and blocker.

My slave Prometheus is openid auth enabled. Whenever, We try to access the 
prometheus it ask for login authentication and get us IN if authorization 
get success. Post successful login, prometheus dashboard and its graph can 
be see.

Federation prometheus is running at different place. Federation prometheus 
scrape_configs looks like below

scrape_configs:

   - 
   
   job_name: 'federate'
   scrape_interval: 15s
   
   honor_labels: true
   metrics_path: '/federate'
   
   params:
   'match[]':
   - '{job="prometheus"}'
   - '{name=~"job:.*"}'
   
   static_configs:
   - targets:
 - 'prometheus-slave.xyz.com:443' (my slave prometheus endpoint)
  
Issue : The target status is DOWN and Status is "server returned HTTP 
status 403 Forbidden".

I know the error is coming because of federation Prometheus has not getting 
credential to access slave Prometheus. But, I am not getting anything at 
federation prometheus where credentials related configuration will be done 
which allow federation Prometheus to access auth protected slave prometheus.

Suggestion & help need !!
#FederationPrometheus

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/475aed3a-5640-464c-beb0-66996cc838f3o%40googlegroups.com.

[prometheus-users] JMX exporter problem, can't run process

[Jacek 1974]

Hi, pls help I don't know where is The problem.

I have installed ActiveMQ in docker, something like: 
https://hub.docker.com/r/webcenter/activemq/, AMQ: -p 8161:8161 -p 
61616:61616 -p 61613:61613 \

according to: https://github.com/prometheus/jmx_exporter I want to run jmx 
exporter on my AMQ server. I tried a lot of variants:


*1. I found activemq.jar (Docker) 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/bin/activemq.jar*

*2. activemq.yml 
= 
https://github.com/prometheus/jmx_exporter/blob/master/example_configs/activemq.yml
 
*

*3. cmd:*
java 
-javaagent:/opt/scripts/jmx_prometheus_javaagent-0.13.0.jar=9110:/opt/scripts/activemq.yml
 
-jar 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/bin/activemq.jar

gets:

 java 
-javaagent:/opt/scripts/jmx_prometheus_javaagent-0.13.0.jar=9110:/opt/scripts/activemq.yml
 
-jar 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/bin/activemq.jar
Java Runtime: Oracle Corporation xx 
/usr/lib/jvm/java-
  Heap sizes: current=XXX free=XXX max=XXX
JVM args: 
-javaagent:/opt/scripts/jmx_prometheus_javaagent-0.13.0.jar=9110:/opt/scripts/activemq.yml
Extensions classpath:
  
[/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/lib,/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/lib/camel,/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/lib/optional,/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/lib/web,/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/lib/extra]
ACTIVEMQ_HOME: 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq
ACTIVEMQ_BASE: 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq
ACTIVEMQ_CONF: 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/conf
ACTIVEMQ_DATA: 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/data
Usage: Main [--extdir ] [task] [task-options] [task data]

Tasks:
browse   - Display selected messages in a specified 
destination.
bstat- Performs a predefined query that displays 
useful statistics regarding the specified broker
consumer - Receives messages from the broker
create   - Creates a runnable broker instance in the 
specified path.
decrypt  - Decrypts given text
dstat- Performs a predefined query that displays 
useful tabular statistics regarding the specified destination type
encrypt  - Encrypts given text
export   - Exports a stopped brokers data files to an 
archive file
list - Lists all available brokers in the specified 
JMX context
producer - Sends messages to the broker
purge- Delete selected destination's messages that 
matches the message selector
query- Display selected broker component's 
attributes and statistics.
start- Creates and starts a broker using a 
configuration file, or a broker URI.
stop - Stops a running broker specified by the 
broker name.

Task Options (Options specific to each task):
--extdir   - Add the jar files in the directory to the classpath.
--version   - Display the version information.
-h,-?,--help- Display this help information. To display task 
specific help, use Main [task] -h,-?,--help

Task Data:
- Information needed by each specific task.

JMX system property options:
-Dactivemq.jmx.url= (default is: 
'service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi')
-Dactivemq.jmx.user=
-Dactivemq.jmx.password=



*4. the same but with --version at the end* 

gets:
java 
-javaagent:/opt/scripts/jmx_prometheus_javaagent-0.13.0.jar=9110:/opt/scripts/activemq.yml
 
-jar 
/opt/docker/overlay2/a378910e4d6ca8e33b23502c106124644296b281e99343f14470dda79a326ff5/merged/opt/activemq/bin/activemq.jar
 
--version
Java Runtime: Oracle Corporation x /usr/lib/jvm/j.
  Heap sizes: current=XXX free=XXX max=XXX
JVM args: 
-javaagent:/opt/scripts/jmx_prometheus_javaagent-0.13.0.jar=9110:/opt/scripts/activemq.yml
Extensions classpath:
  

Re: [prometheus-users] Query with diivision

[Альберт Александров]

I suspected there is something wrong with labels. 

Thanks for your answer! That worked

понедельник, 6 июля 2020 г., 11:19:12 UTC+3 пользователь Aliaksandr 
Valialkin написал:
>
> Try the following query:
>
> (rules_job_count{cluster="loco-prod", status="failed"} + ignoring(status) 
> rules_job_count{cluster="loco-prod", status="cancelled"}) / 
> ignoring(status) rules_job_count{cluster="loco-prod", status="finished"}
>
> It instructs Prometheus to ignore the `status` label when performing the 
> addition and division operations. See more details about this at 
> https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching
>
>
> On Mon, Jul 6, 2020 at 10:48 AM Альберт Александров  > wrote:
>
>>
>> Hi all!
>>
>>
>> Have such metrics:
>>
>>
>> [image: photo_2020-07-06_10-30-12.jpg]
>>
>> I would like to query:
>>
>> (rules_job_count{cluster="loco-prod", status="failed"} + 
>>> rules_job_count{cluster="loco-prod", status="cancelled"}) / 
>>> rules_job_count{cluster="loco-prod", status="finished"}
>>
>>
>> But this didn't work. At the same time this query works:
>>
>> rules_job_count{cluster="loco-prod", status="failed"} + 
>>> rules_job_count{cluster="loco-prod", status="failed"} 
>>
>>
>> Could you say please how to make the first query work?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to promethe...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/ac417564-df12-4627-8c09-2538c759a7c7o%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Best Regards,
>
> Aliaksandr Valialkin, CTO VictoriaMetrics
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c55aed6d-c578-4168-a7c3-80295e7be160o%40googlegroups.com.

Re: [prometheus-users] Re: prometheus not scrapping targets when timestamp field is present

[Aliaksandr Valialkin]

Prometheus doesn't support storing historical data and samples with
out-of-order timestamps. If you need writing such data, then take a look at
other Prometheus-inspired solutions. See, for example,
https://victoriametrics.github.io/#backfilling .

On Mon, Jul 6, 2020 at 10:31 AM Venkata Bhagavatula 
wrote:

> Hi All,
> Can any one respond to my queries? Also we observed the following:
> 1. If for eg timestamp(epoch) in the scrape is 12:00:00, then prometheus
> is not scrapping the targets
> 2. If for eg timestamp(epoch) in the scrape is 12:00:01, then prometheus
> is scrapping the targets.
>
> Thanks & regards,
> Chalapathi
>
> On Thu, Jul 2, 2020 at 3:03 PM Venkata Bhagavatula <
> venkat.cha...@gmail.com> wrote:
>
>> Hi,
>>
>> We are using prometheus version 2.11.1, In our application, the scrape
>> target has timestamp field.  when timestamp field is present, then
>> prometheus is not scrapping any metrics.
>> Following is the output of the curl request for scrape target:
>>
>>- *cmd: curl  http://:24231/metrics*
>>
>> meas_gauge{id="Filtered",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Rejected",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="ReprocessedIn",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Created",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Duplicated",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Stored",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 336.0 159368040
>> meas_gauge{id="Retrieved",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 354.0 159368040
>> meas_gauge{id="ReducedInMerging",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>>
>>
>>
>>- I checked that time is in sync between the prometheus node and the
>>target node.
>>- Following is the epoch time on the prometheus node:
>>
>> *cmd: date +'%s%3N'*
>> *1593681793979*
>>
>>
>>- Epoch difference between the prometheus node and the time stamp
>>present in the sample is more than an hour.
>>
>> difference = ( 1593681793979 -  159368040) / 1000 = 1393sec = 23min
>>
>> Scrape_interval is configured as 300s
>> honor_timestamps is set to true.
>>
>> Can you let us know why prometheus is not able to scrape the targets? Is
>> it due to the timestamp difference between prometheus and target?
>> How much difference will prometheus tolerate?
>>
>> Thanks n Regards,
>> Chalapathi
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CABXnQPuB5iWDhDw06OLOepmz5_XgC2a%3DC9uuVaDKcczR9B-%2BAA%40mail.gmail.com
> 
> .
>


-- 
Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPbKnmCdo2txis5J8vVRGbMGB4X1N%2BjQiUr9aujrf1sqdXmjPw%40mail.gmail.com.

Re: [prometheus-users] Query with diivision

[Aliaksandr Valialkin]

Try the following query:

(rules_job_count{cluster="loco-prod", status="failed"} + ignoring(status)
rules_job_count{cluster="loco-prod", status="cancelled"}) /
ignoring(status) rules_job_count{cluster="loco-prod", status="finished"}

It instructs Prometheus to ignore the `status` label when performing the
addition and division operations. See more details about this at
https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching


On Mon, Jul 6, 2020 at 10:48 AM Альберт Александров 
wrote:

>
> Hi all!
>
>
> Have such metrics:
>
>
> [image: photo_2020-07-06_10-30-12.jpg]
>
> I would like to query:
>
> (rules_job_count{cluster="loco-prod", status="failed"} +
>> rules_job_count{cluster="loco-prod", status="cancelled"}) /
>> rules_job_count{cluster="loco-prod", status="finished"}
>
>
> But this didn't work. At the same time this query works:
>
> rules_job_count{cluster="loco-prod", status="failed"} +
>> rules_job_count{cluster="loco-prod", status="failed"}
>
>
> Could you say please how to make the first query work?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/ac417564-df12-4627-8c09-2538c759a7c7o%40googlegroups.com
> 
> .
>


-- 
Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPbKnmBXZoVtuxBXJjJ3GDf7PvVEszx%3DF3kqo_rwxa7yNKNhtA%40mail.gmail.com.

[prometheus-users] Query with diivision

[Альберт Александров]

Hi all!


Have such metrics:


[image: photo_2020-07-06_10-30-12.jpg]

I would like to query:

(rules_job_count{cluster="loco-prod", status="failed"} + 
> rules_job_count{cluster="loco-prod", status="cancelled"}) / 
> rules_job_count{cluster="loco-prod", status="finished"}


But this didn't work. At the same time this query works:

rules_job_count{cluster="loco-prod", status="failed"} + 
> rules_job_count{cluster="loco-prod", status="failed"} 


Could you say please how to make the first query work?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ac417564-df12-4627-8c09-2538c759a7c7o%40googlegroups.com.

Re: [prometheus-users] SNMP-Exporter: How to handle Not Accessible Index

[Ben Kochie]

Can you share the MIB?

On Sat, Jul 4, 2020 at 9:27 PM 0815Profil Ohneprofil 
wrote:

> Hi,
>
> I have a Broadcast Video router which has „Not Accessible Indices“
> So I can‘t perform a lookup...
>
> The Indices are just Id-counters.
> Such as frame-id or slot-Id ...
> Multi-Index with no chance to read the Index data from the device.
>
> I‘m looking for a possibility to set the labelnames without a lookup ...
> without Access to an Index oid ...
>
> Any idea?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c156f718-21ff-4244-bfe6-3059eb898156o%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrwPAPHD0ziof10%3Dw%3DgOFFmptWrMhafK-h2%2B9o6c%3D7VXA%40mail.gmail.com.

[prometheus-users] Re: prometheus not scrapping targets when timestamp field is present

[Venkata Bhagavatula]

Hi All,
Can any one respond to my queries? Also we observed the following:
1. If for eg timestamp(epoch) in the scrape is 12:00:00, then prometheus is
not scrapping the targets
2. If for eg timestamp(epoch) in the scrape is 12:00:01, then prometheus is
scrapping the targets.

Thanks & regards,
Chalapathi

On Thu, Jul 2, 2020 at 3:03 PM Venkata Bhagavatula 
wrote:

> Hi,
>
> We are using prometheus version 2.11.1, In our application, the scrape
> target has timestamp field.  when timestamp field is present, then
> prometheus is not scrapping any metrics.
> Following is the output of the curl request for scrape target:
>
>- *cmd: curl  http://:24231/metrics*
>
> meas_gauge{id="Filtered",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
> 0.0 159368040
> meas_gauge{id="Rejected",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
> 0.0 159368040
> meas_gauge{id="ReprocessedIn",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
> 0.0 159368040
> meas_gauge{id="Created",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0
> 159368040
> meas_gauge{id="Duplicated",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
> 0.0 159368040
> meas_gauge{id="Stored",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
> 336.0 159368040
> meas_gauge{id="Retrieved",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
> 354.0 159368040
> meas_gauge{id="ReducedInMerging",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
> 0.0 159368040
>
>
>
>- I checked that time is in sync between the prometheus node and the
>target node.
>- Following is the epoch time on the prometheus node:
>
> *cmd: date +'%s%3N'*
> *1593681793979*
>
>
>- Epoch difference between the prometheus node and the time stamp
>present in the sample is more than an hour.
>
> difference = ( 1593681793979 -  159368040) / 1000 = 1393sec = 23min
>
> Scrape_interval is configured as 300s
> honor_timestamps is set to true.
>
> Can you let us know why prometheus is not able to scrape the targets? Is
> it due to the timestamp difference between prometheus and target?
> How much difference will prometheus tolerate?
>
> Thanks n Regards,
> Chalapathi
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABXnQPuB5iWDhDw06OLOepmz5_XgC2a%3DC9uuVaDKcczR9B-%2BAA%40mail.gmail.com.

Re: [prometheus-users] How to check exporter down time in 1 week

[sayf eddine Hammemi]

Hello, you can use offset to go back in time given you have enabled data
retention for at least one week
Exp: rate(http_requests_total[5m] offset 1w) (change it accordingly to your
metrics)

On Mon, Jul 6, 2020 at 9:18 AM Jack Chew  wrote:

> Hey guys,
>
>
>How i check up == 0 in 1week accourd  by instacne.
>
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/6127ffce-02ad-48ed-b70e-b5d27daf1647o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjgscxSMFPZ2bJcpJhrg6nEZm7eB4mOpoiQbXK06cWF4w%40mail.gmail.com.

[prometheus-users] How to check exporter down time in 1 week

[Jack Chew]

Hey guys,


   How i check up == 0 in 1week accourd  by instacne.


Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6127ffce-02ad-48ed-b70e-b5d27daf1647o%40googlegroups.com.