[prometheus-users] [Remote read] cannot get the customize metrics both in another prometheus or InfluxDB

2020-08-26 Thread 毛毛
Hi Everyone, My Prometheus cannot get the customize metrics both in another Prometheus or InfluxDB I have three servers : Prometheus_A Prometheus_B InfluxDB Prometheus_A was set the remote write to InfluxDB, and I can see all metrics was written in the InfluxDB > SHOW MEASUREMENTS name: meas

[prometheus-users] How to Debug - Error on ingesting out-of-order samples" num_dropped=xx

2020-08-26 Thread dineshnithy...@gmail.com
Hi Team, we are currently facing issues where metrics were getting dropped throwing exception as "Error on ingesting out-of-order samples" - And this happens only for specific job configs and need pointers on On what circumstances does it happens and any alternative solutions to it ?

[prometheus-users] Any Metric/Alert When Consul dns disconnects from prometheus

2020-08-26 Thread dineshnithy...@gmail.com
Hi Team, we predominantly use multiple consul_sd_configs and have a service regex to scrape metrics across 1000's of target nodes Issues: Sometimes we observe that consul dns malfunctions and due to which prometheus drops connectivity to consul server and while this happens all the metric ing

Re: [prometheus-users] Correlating different alerts to produce single alert

2020-08-26 Thread Christian Hoffmann
Hi, On 8/27/20 2:19 AM, radhamani...@gmail.com wrote: > If pods in two different namespaces go down,then we need to send a alert > as an appA is down.. > Can I simply write expr as Kubepoddown_in_namespaceA and > Kubepoddown_in_namespaceB ,and send alert message as "AppA is down"? Yes, that's what

Re: [prometheus-users] go_memstats_alloc_bytes

2020-08-26 Thread Christian Hoffmann
Hi, On 8/26/20 1:27 PM, deepak...@gmail.com wrote: > How can i pull the metrics related to a application from the server > using Prometheus? You would usually instrument the application itself (by using one of the language clients such as client_python, client_golang, etc.). If this isn't possibl

Re: [prometheus-users] /metrics endpoint not showing metrics scraped from applications

2020-08-26 Thread Christian Hoffmann
Hi, On 8/27/20 3:16 AM, 'Rounak Salim' via Prometheus Users wrote: > The /metrics endpoint only shows me the metrics for the Prometheus > server and none of the metrics scraped by Prometheus from other > applications are shown. > > How can I get all my metrics to be shown in the /metrics endpoint

Re: [prometheus-users] Issues with prometheus rules once thanos is enabled

2020-08-26 Thread Rodrigo Martinez
sorry meant servicemonitor On Wednesday, August 26, 2020 at 9:35:04 PM UTC-5 Rodrigo Martinez wrote: > I did as well just add > relabelings: > - action: labeldrop > regex: (pod|service|endpoint|namespace) > > since looks like up to date prometheusrule uses this for > kube-state-me

Re: [prometheus-users] Issues with prometheus rules once thanos is enabled

2020-08-26 Thread Rodrigo Martinez
I did as well just add relabelings: - action: labeldrop regex: (pod|service|endpoint|namespace) since looks like up to date prometheusrule uses this for kube-state-metrics. On Wednesday, August 26, 2020 at 9:24:38 PM UTC-5 Rodrigo Martinez wrote: > Have been updating old prome

Re: [prometheus-users] Issues with prometheus rules once thanos is enabled

2020-08-26 Thread Rodrigo Martinez
Have been updating old prometheus rules and have noticed less errors. However there are some taken from other components. Currently I have ceph running and using their prometheusrule from their examples I see issues kube_node_status_condition{condition="Ready",job="kube-state-metrics",status="t

[prometheus-users] Re: go_memstats_alloc_bytes

2020-08-26 Thread mspr...@us.ibm.com
The golang runtime produces Prometheus metrics about https://golang.org/pkg/runtime/#MemStats using https://github.com/prometheus/client_golang/blob/master/prometheus/go_collector.go . A scrape of any golang process will include those metrics. This includes not only Prometheus servers but al

[prometheus-users] /metrics endpoint not showing metrics scraped from applications

2020-08-26 Thread 'Rounak Salim' via Prometheus Users
The /metrics endpoint only shows me the metrics for the Prometheus server and none of the metrics scraped by Prometheus from other applications are shown. How can I get all my metrics to be shown in the /metrics endpoint? This is causing problems getting our metrics consumed by DataDog agents

Re: [prometheus-users] Correlating different alerts to produce single alert

2020-08-26 Thread radhamani...@gmail.com
Thankyou for the response.. I think I didnt explain my question properly.. If pods in two different namespaces go down,then we need to send a alert as an appA is down.. Can I simply write expr as Kubepoddown_in_namespaceA and Kubepoddown_in_namespaceB ,and send alert message as "AppA is down"? I

[prometheus-users] instance_ip_grouping_key python client clarification

2020-08-26 Thread Connor Ladly-Fredeen
I am attempting to use this functionality to add an additional label of IP address to send to the pushgateway. It seems this function fails on mac (which is fine, I can fallback) but also always returns 127.0.0.1 on a kube pod. Am I missing something for this? The documentation seems somewhat s

Re: [prometheus-users] Correlating different alerts to produce single alert

2020-08-26 Thread Christian Hoffmann
Hi, On 8/26/20 11:00 PM, radhamani...@gmail.com wrote: > I want to send alert by  doing some correlation based upon multiple > alerts.For eg: if podA,podB,serviceA are all 100% down in two different > namespaces(namespace1,namespace2),then  I want to send alert like > ApplicationA is down. Is this

[prometheus-users] NPE setting .labelNames while using Gauge.Time.startTimer()

2020-08-26 Thread Imran Ansari
Hello, Not sure why I get a NPE when setting the `labelNames`, I'm able to provide labels just fine using the REST API directly but not with the Java Client, skipping the labels in Java does work fine as well, what gives ? CollectorRegistry registry = new CollectorRegistry(); fi

[prometheus-users] Correlating different alerts to produce single alert

2020-08-26 Thread radhamani...@gmail.com
I want to send alert by doing some correlation based upon multiple alerts.For eg: if podA,podB,serviceA are all 100% down in two different namespaces(namespace1,namespace2),then I want to send alert like ApplicationA is down. Is this possible? How to do the correlation between different aler

Re: [prometheus-users] Alerts / Alert Manager

2020-08-26 Thread Christian Hoffmann
Hi, On 8/20/20 12:47 PM, 'azha...@googlemail.com' via Prometheus Users wrote: > I have 2 alerts  > > - The first being to fire if CPU is more then 70% (WMI) > > - The second  to report whether an instance is down > > 100 - (avg by(instance) (rate(wmi_cpu_time_total{mode="idle"}[2m])) * > 100) >

Re: [prometheus-users] Wal inclusion in retention.size

2020-08-26 Thread Christian Hoffmann
Hi, On 8/21/20 10:59 AM, Venkata Bhagavatula wrote: > In the https://prometheus.io/docs/prometheus/2.20/storage/ link, > Following is mentioned regarding retention.size > > |--storage.tsdb.retention.size|: [EXPERIMENTAL] This determines the > maximum number of bytes that storage blocks can use (n

Re: [prometheus-users] What is the maximum latency for a metric with a timestamp?

2020-08-26 Thread Brian Candler
If you're trying to record the time that some event happened, e.g. the last time that a job ran, then it's probably better to export the timestamp itself as a separate metric (value = Unix time seconds) -- You received this message because you are subscribed to the Google Groups "Prometheus Us

Re: [prometheus-users] What is the maximum latency for a metric with a timestamp?

2020-08-26 Thread Brian Brazil
On Wed, 26 Aug 2020 at 16:32, Rodolphe Ghio wrote: > Hello, > > I work with Prometheus and I saw that we could expose him to metrics with > a timestamp. I did some tests and I saw that a latency of 1 hour worked but > not two hours. Do you know the maximum latency for Prometheus to still > accept

[prometheus-users] What is the maximum latency for a metric with a timestamp?

2020-08-26 Thread Rodolphe Ghio
Hello, I work with Prometheus and I saw that we could expose him to metrics with a timestamp. I did some tests and I saw that a latency of 1 hour worked but not two hours. Do you know the maximum latency for Prometheus to still accept metrics? Best regards. -- You received this message becau

Re: [prometheus-users] blackbox probe : x509: certificate signed by unknown authority even with insecure_skip_verify set to true

2020-08-26 Thread Christian Hoffmann
Hi, On 8/26/20 1:26 PM, Marion Guthmuller wrote: > I'm trying to monitor a website with prometheus and blackbox exporter. > Each of them are running inside a docker (images pulled from official > docker hub https://hub.docker.com/r/prom/prometheus and > https://hub.docker.com/r/prom/blackbox-expor

Re: [prometheus-users] Re: Can't get metric of max and min temperature of Aruba Switch 3810M / JL075A

2020-08-26 Thread Khanh Nguyen Duc
it mean i must added 1 lookups but i have 1 before like below picture. if I added 1 more , it will can not run , right ? [image: lookyp.jpg] Vào lúc 15:31:14 UTC+7 ngày Thứ Tư, 26 tháng 8, 2020, sup...@gmail.com đã viết: > This can be done with regex extracts. > > It looks like the MIB also ha

[prometheus-users] go_memstats_alloc_bytes

2020-08-26 Thread deepak...@gmail.com
Hi All, Can you please provide me some insights on he below, go_memstats_alloc_bytes : does this show up the allocated bytes for the prometheus itself ? Does the Metrics in the drop down list of the prometheus dashboard pull up metrics related to Prometheus ? How can i pull the metrics related

[prometheus-users] blackbox probe : x509: certificate signed by unknown authority even with insecure_skip_verify set to true

2020-08-26 Thread Marion Guthmuller
Hello! I'm trying to monitor a website with prometheus and blackbox exporter. Each of them are running inside a docker (images pulled from official docker hub https://hub.docker.com/r/prom/prometheus and https://hub.docker.com/r/prom/blackbox-exporter/). My prometheus config: [image: promet

[prometheus-users] Re: Is it possible to rename/edit existing label values for targets?

2020-08-26 Thread Tim Schwenke
"Is there anyway to update/edit all historical metrics which match a certain label and value?" No a z schrieb am Montag, 24. August 2020 um 18:51:11 UTC+2: > > We currently add a label to the target scrape job to give a bit of context > to the service the host provides and route alerts based o

Re: [prometheus-users] Re: Can't get metric of max and min temperature of Aruba Switch 3810M / JL075A

2020-08-26 Thread Ben Kochie
This can be done with regex extracts. It looks like the MIB also has incorrect types for the strings, so you also have to override that. Here's an example generator config: https://github.com/SuperQ/tools/tree/master/snmp_exporter/hp On Wed, Aug 26, 2020 at 6:05 AM Khanh Nguyen Duc wrote: >

Re: [prometheus-users] SNMP collected value shows up under label and not the metric value

2020-08-26 Thread Ben Kochie
Here's an example for this MIB: https://github.com/SuperQ/tools/tree/master/snmp_exporter/hp On Wed, Aug 26, 2020 at 10:11 AM Brian Candler wrote: > On Sunday, 16 August 2020 13:00:13 UTC+1, Linkoid01 wrote: >> >> I've manually changed the snmp.yml file from OctetString to >> DisplayString and

Re: [prometheus-users] SNMP collected value shows up under label and not the metric value

2020-08-26 Thread Brian Candler
On Sunday, 16 August 2020 13:00:13 UTC+1, Linkoid01 wrote: > > I've manually changed the snmp.yml file from OctetString to > DisplayString and I am able to see the values and not a hex representation. > I've checked the DELL-RAC-MIB and the file is correct, like you said > DellPowerReading ::= D

Re: [prometheus-users] SNMP collected value shows up under label and not the metric value

2020-08-26 Thread Khanh Nguyen Duc
change from current to Name will be get that value. Vào lúc 19:00:13 UTC+7 ngày Chủ Nhật, 16 tháng 8, 2020, Linkoid01 đã viết: > I've manually changed the snmp.yml file from OctetString to > DisplayString and I am able to see the values and not a hex representation. > I've checked the DELL-RAC-

[prometheus-users] Data partition growing

2020-08-26 Thread deln...@gmail.com
[image: Capture-wal-corruption.PNG] /usr/local/bin/prometheus --version prometheus, version 2.20.0 (branch: HEAD, revision: e5a06b483527d4fe0704b8fa3a2b475b661c526f) build user: r build date: 20200722-18:51:45 go version: go1.14.6 sudo du -Pshx /var/lib/prometheus/* 73G