[prometheus-users] Readiness probe failed: HTTP probe failed with statuscode: 503

[anis ben hassine]

Hello Guys,



   1. I'm getting for the first time a Readiness Probe issue with ly Prom 
   running on AWS EKS, the pod was running nice since weeks ant it suddenly 
   turns into failing.

*$ kdpon monitoring prometheus *(Kube pod describe command )


Unhealthy  4m (x182535 over 6d)  kubelet, ip-100-.ec2.internal  
Readiness 
probe failed: HTTP probe failed with statuscode: 503



 2. However, seems my logs are doing well, and restarting pods doesn't fix 
the issue but this is a normal. behaviour for readiness Prob




*$ klon monitoring prometheus *(Kube log command )


level=info ts=2020-05-25T09:38:56.653582783Z caller=main.go:243 
msg="Starting Prometheus" version="(version=2.6.1, branch=HEAD, 
revision=b639fe140c1f71b2cbad3fc322b17efe60839e7e)"

level=info ts=2020-05-25T09:38:56.653650678Z caller=main.go:244 
build_context="(go=go1.11.4, user=root@4c0e286fe2b3, 
date=20190115-19:12:04)"

level=info ts=2020-05-25T09:38:56.653691321Z caller=main.go:245 
host_details="(Linux 4.14.154-128.181.amzn2.x86_64 #1 SMP Sat Nov 16 
21:49:00 UTC 2019 x86_64 prometheus-0 (none))"

level=info ts=2020-05-25T09:38:56.653727427Z caller=main.go:246 
fd_limits="(soft=65536, hard=65536)"

level=info ts=2020-05-25T09:38:56.653762149Z caller=main.go:247 
vm_limits="(soft=unlimited, hard=unlimited)"

level=info ts=2020-05-25T09:38:56.65457275Z caller=main.go:561 
msg="Starting TSDB ..."

level=info ts=2020-05-25T09:38:56.655755388Z caller=web.go:429 
component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2020-05-25T09:38:56.817276722Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=1583940235972 
maxt=158397120 ulid=01E36HT83DK3TG22QWE02AP50X

level=info ts=2020-05-25T09:38:56.822556254Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158397120 
maxt=158416560 ulid=01E3C4BCPDEWX998E1EMR8RWTB

level=info ts=2020-05-25T09:38:56.826661565Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158416560 
maxt=158436000 ulid=01E3HXRADQ09H3PD3KHPS5GAXK

level=info ts=2020-05-25T09:38:56.831874475Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158436000 
maxt=158455440 ulid=01E3QQ4PWYR2D0TF0ERWR7KTZA

level=info ts=2020-05-25T09:38:56.837772421Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158455440 
maxt=158474880 ulid=01E3XGHHPWR4QHYZBZGGSR41FS

level=info ts=2020-05-25T09:38:56.842330027Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158474880 
maxt=158494320 ulid=01E439YBW2HYS58CK67540JRDP

level=info ts=2020-05-25T09:38:56.847023184Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158494320 
maxt=158513760 ulid=01E493B0FCHAT74EZFVX63AA8J

level=info ts=2020-05-25T09:38:56.852126117Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158513760 
maxt=158533200 ulid=01E4EWV0G5ZV7G0M5TH3XEV3X5

level=info ts=2020-05-25T09:38:56.856215058Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158533200 
maxt=158552640 ulid=01E4MP49E017G3SF46C4MXND0A

level=info ts=2020-05-25T09:38:56.860292298Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158552640 
maxt=158572080 ulid=01E4TFMEZ1YMEM8PJAQJA9WZA7

level=info ts=2020-05-25T09:38:56.865162002Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158572080 
maxt=158591520 ulid=01E508Y5F1V3RBPRMPVYMQCFQM

level=info ts=2020-05-25T09:38:56.869511088Z caller=repair.go:48 
component=tsdb msg="found healthy block" mint=158591520 
maxt=158610960 ulid=01E562AXM9EHH3BV11MQYHGX80




I'm trying to figure if the issue is rrmated to Code or infrastructure layer

Thanks you in advance for you Help

Best,
ABH

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8ae22e72-c753-4af0-b5b1-d28a60a1654e%40googlegroups.com.

[prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

I've installed Prometheus in my windows machine and installed Debian and 
OpenSuse in VMware. I wanted to check the system metrics so, I installed 
node_exporters in both(Debian and OpenSuse).
But from prometheus server I can only access debian metrics, opensuse 
target is showing down. (Error: Context time out).
Can anyone help me to config node_exporter in opensuse.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9dcdbe57-6bdd-4ff7-afb6-465a4183796e%40googlegroups.com.

[prometheus-users] Not get current value from Prometheus using Grok Exporter

[Deep saraswat]

I have a Grok exporter gauge metric which returning mysql checksum 
completion percentage by reading and parsing from a file. I'm using grafana 
dashboard to visualize Prometheus values.



[image: Screenshot 2020-05-24 at 9.51.00 PM.png] 
Problem is Prometheus holding last value of metric until a new value come 
for it. For example if last value was 48% then prometheus will keep on 
displaying that value until getting a new value.My requirement is, only 
return current values of the completion percentage

grok exporter config:

global:
  config_version: 3
input:
  type: file
  paths:
  - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.out
  - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.err
imports:
- type: grok_patterns
  dir: ./patterns
grok_patterns:
- 'CHECKSUMOUT [a-zA-Z ]*'
metrics:
- type: counter
  name: mysql_ptchecksum_checksum_stats
  help: mysql ptchecksum stats.
  match: 
'%{MONTHDAY:month}\-%{MONTHDAY:date}\T%{TIME:logtime}\s.*%{INT:ERRORS}\s.*%{INT:DIFFS}\s.*%{NUMBER:ROWS}\s.*%{INT:DIFF_ROWS}\s.*%{INT:CHUNKS}\s.*%{INT:SKIPPED}\s.*%{NUMBER:time}
 
%{GREEDYDATA:TABLE}'
  labels:
diffs: '{{.DIFFS}}'
table: '{{.TABLE}}'
date: '{{.date}}'
errors: '{{.ERRORS}}'
rows: '{{.ROWS}}'
diff_rows: '{{.DIFF_ROWS}}'
chunks: '{{.CHUNKS}}'
TS: '{{.logtime}}'
time: '{{.time}}'
- type: gauge
  name: mysql_ptchecksum_checksuming_status
  help: mysql ptchecksum cheksuming status.
  match: 
'%{GREEDYDATA:checksum_status}\s%{GREEDYDATA:tabel}\:.*%{NUMBER:completion_percentage}\%\s.*%{NUMBER:TimeRemainingHours}\:%{NUMBER:TimeRemainingMinutes}'
  value: '{{.completion_percentage}}'
  labels:
status: '{{.checksum_status}}'
tabel: '{{.tabel}}'
- type: gauge
  name: mysql_ptchecksum_waiting_status
  help: mysql ptchecksum waiting status.
  match: 'Replica 
lag\s.*\is\s.*%{INT:seconds}\s%{GREEDYDATA:data}\son\s%{GREEDYDATA:server}\.\s'
  value: '{{.seconds}}'
  labels:
server: '{{.server}}'

server:
  protocol: http
  port: 9144

please help me resolve this.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/65f3848d-c4e8-421f-abbc-3faf33bd89d3%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Stuart Clark]

What is the exact command being used (with all the command line options)? 

On 25 May 2020 09:50:16 BST, Harsh bhadoria  wrote:
>./node_exporter. command starts the node_exporter server. 
>On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:
>>
>> What command line options are being used for them? You can use ps to
>check 
>> or look at the systemd unit
>>
>> On 25 May 2020 09:47:11 BST, Harsh bhadoria > > wrote:
>>>
>>> In debian i've installed the node_exporter and ran it, now I can see
>the 
>>> metrics from prometheus which is in windows, same I've installed in 
>>> opensuse also, and running node_exporter, but I'm not getting info
>in 
>>> prometheus.
>>>
>>> On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:

 Yes, i can see the metrics from opensuse while going through 
 localhost:9100, but not getting it from prometheus


 On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>
> Are you able to use curl or wget to access the node exporter from
>the 
> box or the Prometheus server? 
>
> On 25 May 2020 09:22:35 BST, Harsh bhadoria
> 
> wrote:
>>
>> I've installed Prometheus in my windows machine and installed
>Debian 
>> and OpenSuse in VMware. I wanted to check the system metrics so,
>I 
>> installed node_exporters in both(Debian and OpenSuse).
>> But from prometheus server I can only access debian metrics,
>opensuse 
>> target is showing down. (Error: Context time out).
>> Can anyone help me to config node_exporter in opensuse.
>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my
>brevity.
>

>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/e4cdde08-6172-4f00-ac74-3908eb09d47b%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/76F69BE7-BF2F-44C1-A407-A57C96D066F3%40Jahingo.com.

Re: [prometheus-users] Node_exporter in opensuse

[Stuart Clark]

What configuration do you have set for the different node exporters in your 
prometheus.yml?

On 25 May 2020 10:03:03 BST, Harsh bhadoria  wrote:
>I used chrome in opensuse machine to check the metrics.
>
>On Monday, 25 May 2020 14:30:12 UTC+5:30, Stuart Clark wrote:
>>
>> When you say that you check the opensuse metrics with your browser,
>which 
>> machine is that running on? Is that curl or a graphical browser? 
>>
>> On 25 May 2020 09:57:26 BST, Harsh bhadoria > > wrote:
>>>
>>> I just installed the node_exporter in my Suse env, then unzip it,
>went to 
>>> node_exporter folder, there is an node_exporter executable file,
>from there 
>>> I just give command (./node_exporter), it starts the server at port
>9100, 
>>> now from use env if I go to my browser and go to URL(localhost:9100)
>it 
>>> shows it metrics, but in my prometheus server(running in windows
>side), I'm 
>>> not getting access or see the metrics. On same time, same process
>I've done 
>>> in my debian system, and by Prometheus is able to get the metrics.
>Problem 
>>> is with node_exporter at suse side i think
>>>
>>> On Monday, 25 May 2020 14:21:14 UTC+5:30, Stuart Clark wrote:

 What is the exact command being used (with all the command line 
 options)? 

 On 25 May 2020 09:50:16 BST, Harsh bhadoria 
>
 wrote:
>
> ./node_exporter. command starts the node_exporter server. 
> On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:
>>
>> What command line options are being used for them? You can use ps
>to 
>> check or look at the systemd unit
>>
>> On 25 May 2020 09:47:11 BST, Harsh bhadoria
> 
>> wrote:
>>>
>>> In debian i've installed the node_exporter and ran it, now I can
>see 
>>> the metrics from prometheus which is in windows, same I've
>installed in 
>>> opensuse also, and running node_exporter, but I'm not getting
>info in 
>>> prometheus.
>>>
>>> On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:

 Yes, i can see the metrics from opensuse while going through 
 localhost:9100, but not getting it from prometheus


 On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>
> Are you able to use curl or wget to access the node exporter
>from 
> the box or the Prometheus server? 
>
> On 25 May 2020 09:22:35 BST, Harsh bhadoria
> 
> wrote:
>>
>> I've installed Prometheus in my windows machine and installed
>
>> Debian and OpenSuse in VMware. I wanted to check the system
>metrics so, I 
>> installed node_exporters in both(Debian and OpenSuse).
>> But from prometheus server I can only access debian metrics, 
>> opensuse target is showing down. (Error: Context time out).
>> Can anyone help me to config node_exporter in opensuse.
>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my
>brevity.
>

>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my
>brevity.
>>
>
 -- 
 Sent from my Android device with K-9 Mail. Please excuse my
>brevity.

>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/f5b6ef25-79a5-47bb-ab96-ae587992bd24%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1837FA58-2F0A-4A6C-83DF-F468E789F106%40Jahingo.com.

Re: [prometheus-users] Adding extra labels to metrics

[Brian Candler]

That's almost right.  Firstly, all these settings go under 
"relabel_configs", none under "metric_relabel_configs"

Secondly, you need a final rule to set the __address__ to the actual target 
you want to scrape:

- source_labels: ['__address__']
  regex: '(.*);(.*);(.*);(.*)'
  target_label: '__address__'
  replacement: '$1'

A useful pattern is that you can take the port out of the 'targets' file, 
and apply it here - this gives you meaningful instance labels 
.

- source_labels: ['__address__']
  regex: '(.*);(.*);(.*);(.*)'
  target_label: '__address__'
  replacement: '$1:9117'

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e2cf9374-4d65-4a99-9c78-14e7ac74%40googlegroups.com.

Re: [prometheus-users] How to show delta between all data points in the last 1 minute? (Gauge)

[tomeri]

Hi Ben,

I looked into mtail it looks cool but probably doesn't fit my needs
I run a pod (DaemonSet) in k8s that counts the peer-pods (same host) log's 
producing rate


On Monday, May 25, 2020 at 3:14:28 PM UTC+3, Ben Kochie wrote:
>
> In your code, you just add deltas to a counter metric. This tracks the 
> local known state of how many lines it's seen. Prometheus tracks this over 
> time, allowing you to use rate() and increase() functions to calculate the 
> deltas over time.
>
> If you're looking to observe log files, I highly recommend mtail. It's a 
> well tested and reliable option for producing metrics about and from log 
> files.
>
> On Mon, May 25, 2020 at 1:48 PM tomeri  > wrote:
>
>> You right, i care about "how many lines have I added to this file", and 
>> yes I'm actually want to check the rate of log files (append-only).
>> Let's say counter is the best fit here, on the code side how do i get the 
>> diff between the current total value to the previous sample? do i need to 
>> hold it in my app memory? is it something that Prometheus can solve?
>>
>> On Sunday, May 24, 2020 at 8:42:12 PM UTC+3, Julius Volz wrote:
>>>
>>> On Sun, May 24, 2020 at 5:02 PM tomeri  wrote:
>>>
 Hi Julius,

 Thanks for the fast response.

 Do you think Gauge is the right metric kind for this purpose?

>>>
>>> A gauge tracks anything that can naturally go up or down. The length of 
>>> lines in a file can in principle go up or down, so normally that would be a 
>>> gauge.
>>>
>>> If you have files that are basically append-only (like log files), and 
>>> you want to track the rate at which they grow, then it would be better to 
>>> think of the metric conceptually not as "how many lines does this file 
>>> have", but "how many lines have I added to this file" (can only go up, and 
>>> you count lines as they are being added, not just proxying some third-party 
>>> line count), which would be more fitting for a counter metric.
>>>  
>>>
 How do I handle resets? i see drops with deriv/delta

>>>
>>> If it's actually a counter (see above) where drops truly indicate a 
>>> reset, then you will need to use rate() or increase(). In your case 
>>> "increase(my_lines_total[1m])". This will handle counter resets, but will 
>>> still have the same extrapolation caveat as I mentioned earlier.
>>>  
>>>
 I tried both drive and delta with 1m time range multiple by 60 and got 
 wrong results, here is my query:
 sum 
 by(datacenter)(delta(files_total_lines_gauge{datacenter="xxx"}[1m])) * 60 
 > 
 0

 I use >0 to workaround the drops

 I basically try to get the same result as I were to use graphite with 
 nonNegativeDerivative

>>>
>>> It sounds like the closest thing in PromQL is indeed 
>>> "increase(foo[1m])", but it sounds like the Graphite function doesn't do 
>>> the extrapolation bit.
>>>  
>>>
 On Sunday, May 24, 2020 at 5:39:02 PM UTC+3, Julius Volz wrote:
>
> Since it's a gauge (and at least theoretically line counts can 
> decrease, not only increase), you'll want either the delta() (
> https://prometheus.io/docs/prometheus/2.18/querying/functions/#delta) or 
> the deriv() (
> https://prometheus.io/docs/prometheus/2.18/querying/functions/#deriv 
> )
>  
> function, multiplied by 60 (to get from per-second to per-minute).
>
> Note that both functions can give you non-integer results even if the 
> line numbers only change by integer increments/decrements, as delta() 
> extrapolates the observed slope to the edges of the provided time window, 
> and deriv() does a linear regression to estimate how fast a gauge is 
> going 
> up or down.
>
> Another thing you could do (if you care about integer results) is:
>
> my_lines_total - my_lines_total offset 1m
>
> ...to give you the absolute difference between the last sample value 
> seen 1m ago and the currently last-seen sample value. Note that while 
> this 
> returns you an integer result, it might be further away from the "true" 
> rate due to the lack of extrapolation, because the two samples you will 
> be 
> comparing will not be exactly 1m apart.
>
> On Sun, May 24, 2020 at 4:05 PM tomeri  wrote:
>
>> Hi,
>>
>> I run an application that export metrics about the total number of 
>> lines of each file in my directory, basically the product of `wc -l`
>> So on each interval (every 1m) my app counts the total number of 
>> lines and then updates the Gague metric.
>>
>> For example:
>> First iteration: 1000 total lines
>> Second iteration: 1300 total lines
>> Third iteration: 1900 total lines
>> Fourth iteration: 2400 total lines
>> ...
>>
>> 

Re: [prometheus-users] Not get current value from Prometheus using Grok Exporter

[Stuart Clark]

Until a new value is scraped, that 48% is the current value.

As a metrics system Prometheus scrapes values regularly. As a result it is 
unlikely that a scrape will fall exactly on the moment needed for a query to be 
processed. Instead Prometheus will look back to find the latest value and use 
that (with a limit, after which a time series is marked as stale).

What are you trying to achieve? 

On 25 May 2020 09:32:02 BST, Deep saraswat  wrote:
>I have a Grok exporter gauge metric which returning mysql checksum 
>completion percentage by reading and parsing from a file. I'm using
>grafana 
>dashboard to visualize Prometheus values.
>
>
>
>[image: Screenshot 2020-05-24 at 9.51.00 PM.png]
>
>Problem is Prometheus holding last value of metric until a new value
>come 
>for it. For example if last value was 48% then prometheus will keep on 
>displaying that value until getting a new value.My requirement is, only
>
>return current values of the completion percentage
>
>grok exporter config:
>
>global:
>  config_version: 3
>input:
>  type: file
>  paths:
>  - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.out
>  - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.err
>imports:
>- type: grok_patterns
>  dir: ./patterns
>grok_patterns:
>- 'CHECKSUMOUT [a-zA-Z ]*'
>metrics:
>- type: counter
>  name: mysql_ptchecksum_checksum_stats
>  help: mysql ptchecksum stats.
>  match: 
>'%{MONTHDAY:month}\-%{MONTHDAY:date}\T%{TIME:logtime}\s.*%{INT:ERRORS}\s.*%{INT:DIFFS}\s.*%{NUMBER:ROWS}\s.*%{INT:DIFF_ROWS}\s.*%{INT:CHUNKS}\s.*%{INT:SKIPPED}\s.*%{NUMBER:time}
>
>%{GREEDYDATA:TABLE}'
>  labels:
>diffs: '{{.DIFFS}}'
>table: '{{.TABLE}}'
>date: '{{.date}}'
>errors: '{{.ERRORS}}'
>rows: '{{.ROWS}}'
>diff_rows: '{{.DIFF_ROWS}}'
>chunks: '{{.CHUNKS}}'
>TS: '{{.logtime}}'
>time: '{{.time}}'
>- type: gauge
>  name: mysql_ptchecksum_checksuming_status
>  help: mysql ptchecksum cheksuming status.
>  match: 
>'%{GREEDYDATA:checksum_status}\s%{GREEDYDATA:tabel}\:.*%{NUMBER:completion_percentage}\%\s.*%{NUMBER:TimeRemainingHours}\:%{NUMBER:TimeRemainingMinutes}'
>  value: '{{.completion_percentage}}'
>  labels:
>status: '{{.checksum_status}}'
>tabel: '{{.tabel}}'
>- type: gauge
>  name: mysql_ptchecksum_waiting_status
>  help: mysql ptchecksum waiting status.
>  match: 'Replica 
>lag\s.*\is\s.*%{INT:seconds}\s%{GREEDYDATA:data}\son\s%{GREEDYDATA:server}\.\s'
>  value: '{{.seconds}}'
>  labels:
>server: '{{.server}}'
>
>server:
>  protocol: http
>  port: 9144
>
>please help me resolve this.
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/65f3848d-c4e8-421f-abbc-3faf33bd89d3%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/96505C3B-DBED-4F4D-B3F3-FB623E714C67%40Jahingo.com.

Re: [prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

I just installed the node_exporter in my Suse env, then unzip it, went to 
node_exporter folder, there is an node_exporter executable file, from there 
I just give command (./node_exporter), it starts the server at port 9100, 
now from use env if I go to my browser and go to URL(localhost:9100) it 
shows it metrics, but in my prometheus server(running in windows side), I'm 
not getting access or see the metrics. On same time, same process I've done 
in my debian system, and by Prometheus is able to get the metrics. Problem 
is with node_exporter at suse side i think

On Monday, 25 May 2020 14:21:14 UTC+5:30, Stuart Clark wrote:
>
> What is the exact command being used (with all the command line options)? 
>
> On 25 May 2020 09:50:16 BST, Harsh bhadoria  > wrote:
>>
>> ./node_exporter. command starts the node_exporter server. 
>> On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:
>>>
>>> What command line options are being used for them? You can use ps to 
>>> check or look at the systemd unit
>>>
>>> On 25 May 2020 09:47:11 BST, Harsh bhadoria  
>>> wrote:

 In debian i've installed the node_exporter and ran it, now I can see 
 the metrics from prometheus which is in windows, same I've installed in 
 opensuse also, and running node_exporter, but I'm not getting info in 
 prometheus.

 On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:
>
> Yes, i can see the metrics from opensuse while going through 
> localhost:9100, but not getting it from prometheus
>
>
> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>>
>> Are you able to use curl or wget to access the node exporter from the 
>> box or the Prometheus server? 
>>
>> On 25 May 2020 09:22:35 BST, Harsh bhadoria  
>> wrote:
>>>
>>> I've installed Prometheus in my windows machine and installed Debian 
>>> and OpenSuse in VMware. I wanted to check the system metrics so, I 
>>> installed node_exporters in both(Debian and OpenSuse).
>>> But from prometheus server I can only access debian metrics, 
>>> opensuse target is showing down. (Error: Context time out).
>>> Can anyone help me to config node_exporter in opensuse.
>>>
>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/10501952-31a9-4b14-8162-b9753efa1415%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

./node_exporter. command starts the node_exporter server. 
On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:
>
> What command line options are being used for them? You can use ps to check 
> or look at the systemd unit
>
> On 25 May 2020 09:47:11 BST, Harsh bhadoria  > wrote:
>>
>> In debian i've installed the node_exporter and ran it, now I can see the 
>> metrics from prometheus which is in windows, same I've installed in 
>> opensuse also, and running node_exporter, but I'm not getting info in 
>> prometheus.
>>
>> On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:
>>>
>>> Yes, i can see the metrics from opensuse while going through 
>>> localhost:9100, but not getting it from prometheus
>>>
>>>
>>> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:

 Are you able to use curl or wget to access the node exporter from the 
 box or the Prometheus server? 

 On 25 May 2020 09:22:35 BST, Harsh bhadoria  
 wrote:
>
> I've installed Prometheus in my windows machine and installed Debian 
> and OpenSuse in VMware. I wanted to check the system metrics so, I 
> installed node_exporters in both(Debian and OpenSuse).
> But from prometheus server I can only access debian metrics, opensuse 
> target is showing down. (Error: Context time out).
> Can anyone help me to config node_exporter in opensuse.
>
>
 -- 
 Sent from my Android device with K-9 Mail. Please excuse my brevity.

>>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e4cdde08-6172-4f00-ac74-3908eb09d47b%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Stuart Clark]

If you use that IP address rather than localhost on the opensuse machine (in 
the web browser) do you get metrics? 

On 25 May 2020 10:14:03 BST, Harsh bhadoria  wrote:
>In scrape_configs, i added the "job-name: 'node_exporter' " and in
>static 
>config, I've added targets as IP address of debian and opensuse.
>Like this, 
>- target: ['194.167.200.146:9100','194.167.200.147:9100']
>
>On Monday, 25 May 2020 14:36:34 UTC+5:30, Stuart Clark wrote:
>>
>> What configuration do you have set for the different node exporters
>in 
>> your prometheus.yml?
>>
>> On 25 May 2020 10:03:03 BST, Harsh bhadoria > > wrote:
>>>
>>> I used chrome in opensuse machine to check the metrics.
>>>
>>> On Monday, 25 May 2020 14:30:12 UTC+5:30, Stuart Clark wrote:

 When you say that you check the opensuse metrics with your browser,
>
 which machine is that running on? Is that curl or a graphical
>browser? 

 On 25 May 2020 09:57:26 BST, Harsh bhadoria 
>
 wrote:
>
> I just installed the node_exporter in my Suse env, then unzip it,
>went 
> to node_exporter folder, there is an node_exporter executable
>file, from 
> there I just give command (./node_exporter), it starts the server
>at port 
> 9100, now from use env if I go to my browser and go to
>URL(localhost:9100) 
> it shows it metrics, but in my prometheus server(running in
>windows side), 
> I'm not getting access or see the metrics. On same time, same
>process I've 
> done in my debian system, and by Prometheus is able to get the
>metrics. 
> Problem is with node_exporter at suse side i think
>
> On Monday, 25 May 2020 14:21:14 UTC+5:30, Stuart Clark wrote:
>>
>> What is the exact command being used (with all the command line 
>> options)? 
>>
>> On 25 May 2020 09:50:16 BST, Harsh bhadoria
> 
>> wrote:
>>>
>>> ./node_exporter. command starts the node_exporter server. 
>>> On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:

 What command line options are being used for them? You can use
>ps to 
 check or look at the systemd unit

 On 25 May 2020 09:47:11 BST, Harsh bhadoria
> 
 wrote:
>
> In debian i've installed the node_exporter and ran it, now I
>can 
> see the metrics from prometheus which is in windows, same I've
>installed in 
> opensuse also, and running node_exporter, but I'm not getting
>info in 
> prometheus.
>
> On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria
>wrote:
>>
>> Yes, i can see the metrics from opensuse while going through 
>> localhost:9100, but not getting it from prometheus
>>
>>
>> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>>>
>>> Are you able to use curl or wget to access the node exporter
>from 
>>> the box or the Prometheus server? 
>>>
>>> On 25 May 2020 09:22:35 BST, Harsh bhadoria <
>>> harshbha...@gmail.com> wrote:

 I've installed Prometheus in my windows machine and
>installed 
 Debian and OpenSuse in VMware. I wanted to check the system
>metrics so, I 
 installed node_exporters in both(Debian and OpenSuse).
 But from prometheus server I can only access debian
>metrics, 
 opensuse target is showing down. (Error: Context time out).
 Can anyone help me to config node_exporter in opensuse.


>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my 
>>> brevity.
>>>
>>
 -- 
 Sent from my Android device with K-9 Mail. Please excuse my
>brevity.

>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my
>brevity.
>>
>
 -- 
 Sent from my Android device with K-9 Mail. Please excuse my
>brevity.

>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/c77ce763-8e6a-4555-952c-1ba166fdc718%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2E7BB6FA-B30B-4D92-970B-C37E1CC1FBEA%40Jahingo.com.

Re: [prometheus-users] How to show delta between all data points in the last 1 minute? (Gauge)

[Ben Kochie]

In your code, you just add deltas to a counter metric. This tracks the
local known state of how many lines it's seen. Prometheus tracks this over
time, allowing you to use rate() and increase() functions to calculate the
deltas over time.

If you're looking to observe log files, I highly recommend mtail. It's a
well tested and reliable option for producing metrics about and from log
files.

On Mon, May 25, 2020 at 1:48 PM tomeri  wrote:

> You right, i care about "how many lines have I added to this file", and
> yes I'm actually want to check the rate of log files (append-only).
> Let's say counter is the best fit here, on the code side how do i get the
> diff between the current total value to the previous sample? do i need to
> hold it in my app memory? is it something that Prometheus can solve?
>
> On Sunday, May 24, 2020 at 8:42:12 PM UTC+3, Julius Volz wrote:
>>
>> On Sun, May 24, 2020 at 5:02 PM tomeri  wrote:
>>
>>> Hi Julius,
>>>
>>> Thanks for the fast response.
>>>
>>> Do you think Gauge is the right metric kind for this purpose?
>>>
>>
>> A gauge tracks anything that can naturally go up or down. The length of
>> lines in a file can in principle go up or down, so normally that would be a
>> gauge.
>>
>> If you have files that are basically append-only (like log files), and
>> you want to track the rate at which they grow, then it would be better to
>> think of the metric conceptually not as "how many lines does this file
>> have", but "how many lines have I added to this file" (can only go up, and
>> you count lines as they are being added, not just proxying some third-party
>> line count), which would be more fitting for a counter metric.
>>
>>
>>> How do I handle resets? i see drops with deriv/delta
>>>
>>
>> If it's actually a counter (see above) where drops truly indicate a
>> reset, then you will need to use rate() or increase(). In your case
>> "increase(my_lines_total[1m])". This will handle counter resets, but will
>> still have the same extrapolation caveat as I mentioned earlier.
>>
>>
>>> I tried both drive and delta with 1m time range multiple by 60 and got
>>> wrong results, here is my query:
>>> sum by(datacenter)(delta(files_total_lines_gauge{datacenter="xxx"}[1m]))
>>> * 60 > 0
>>>
>>> I use >0 to workaround the drops
>>>
>>> I basically try to get the same result as I were to use graphite with
>>> nonNegativeDerivative
>>>
>>
>> It sounds like the closest thing in PromQL is indeed "increase(foo[1m])",
>> but it sounds like the Graphite function doesn't do the extrapolation bit.
>>
>>
>>> On Sunday, May 24, 2020 at 5:39:02 PM UTC+3, Julius Volz wrote:

 Since it's a gauge (and at least theoretically line counts can
 decrease, not only increase), you'll want either the delta() (
 https://prometheus.io/docs/prometheus/2.18/querying/functions/#delta) or
 the deriv() (
 https://prometheus.io/docs/prometheus/2.18/querying/functions/#deriv
 )
 function, multiplied by 60 (to get from per-second to per-minute).

 Note that both functions can give you non-integer results even if the
 line numbers only change by integer increments/decrements, as delta()
 extrapolates the observed slope to the edges of the provided time window,
 and deriv() does a linear regression to estimate how fast a gauge is going
 up or down.

 Another thing you could do (if you care about integer results) is:

 my_lines_total - my_lines_total offset 1m

 ...to give you the absolute difference between the last sample value
 seen 1m ago and the currently last-seen sample value. Note that while this
 returns you an integer result, it might be further away from the "true"
 rate due to the lack of extrapolation, because the two samples you will be
 comparing will not be exactly 1m apart.

 On Sun, May 24, 2020 at 4:05 PM tomeri  wrote:

> Hi,
>
> I run an application that export metrics about the total number of
> lines of each file in my directory, basically the product of `wc -l`
> So on each interval (every 1m) my app counts the total number of lines
> and then updates the Gague metric.
>
> For example:
> First iteration: 1000 total lines
> Second iteration: 1300 total lines
> Third iteration: 1900 total lines
> Fourth iteration: 2400 total lines
> ...
>
> Prometheus scraps my app's metric every 15s/25s (depends on the env)
> What i want to plot is a graph that will show the rate per minute -
> how many lines produced in each file for that last 1 minute.
>
> No matter what I tried, I couldn't make the graph to show to correct
> results.
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive 

Re: [prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

In debian i've installed the node_exporter and ran it, now I can see the 
metrics from prometheus which is in windows, same I've installed in 
opensuse also, and running node_exporter, but I'm not getting info in 
prometheus.

On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:
>
> Yes, i can see the metrics from opensuse while going through 
> localhost:9100, but not getting it from prometheus
>
>
> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>>
>> Are you able to use curl or wget to access the node exporter from the box 
>> or the Prometheus server? 
>>
>> On 25 May 2020 09:22:35 BST, Harsh bhadoria  
>> wrote:
>>>
>>> I've installed Prometheus in my windows machine and installed Debian and 
>>> OpenSuse in VMware. I wanted to check the system metrics so, I installed 
>>> node_exporters in both(Debian and OpenSuse).
>>> But from prometheus server I can only access debian metrics, opensuse 
>>> target is showing down. (Error: Context time out).
>>> Can anyone help me to config node_exporter in opensuse.
>>>
>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/887c33d9-b57a-4bdd-9b66-c7ab80187eda%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

No, firewall is off. 

On Monday, 25 May 2020 14:16:39 UTC+5:30, Stuart Clark wrote:
>
> What is the configuration being used for the node exporter? Could you be 
> binding to only localhost rather than and external network interface?
>
> Are there any firewalls between the server and Prometheus? 
>
> On 25 May 2020 09:44:54 BST, Harsh bhadoria  > wrote:
>>
>> Yes, i can see the metrics from opensuse while going through 
>> localhost:9100, but not getting it from prometheus
>>
>>
>> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>>>
>>> Are you able to use curl or wget to access the node exporter from the 
>>> box or the Prometheus server? 
>>>
>>> On 25 May 2020 09:22:35 BST, Harsh bhadoria  
>>> wrote:

 I've installed Prometheus in my windows machine and installed Debian 
 and OpenSuse in VMware. I wanted to check the system metrics so, I 
 installed node_exporters in both(Debian and OpenSuse).
 But from prometheus server I can only access debian metrics, opensuse 
 target is showing down. (Error: Context time out).
 Can anyone help me to config node_exporter in opensuse.


>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/5e47eebc-6cd3-4b39-9eaa-7eb2b99c9842%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Stuart Clark]

What command line options are being used for them? You can use ps to check or 
look at the systemd unit

On 25 May 2020 09:47:11 BST, Harsh bhadoria  wrote:
>In debian i've installed the node_exporter and ran it, now I can see
>the 
>metrics from prometheus which is in windows, same I've installed in 
>opensuse also, and running node_exporter, but I'm not getting info in 
>prometheus.
>
>On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:
>>
>> Yes, i can see the metrics from opensuse while going through 
>> localhost:9100, but not getting it from prometheus
>>
>>
>> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>>>
>>> Are you able to use curl or wget to access the node exporter from
>the box 
>>> or the Prometheus server? 
>>>
>>> On 25 May 2020 09:22:35 BST, Harsh bhadoria  
>>> wrote:

 I've installed Prometheus in my windows machine and installed
>Debian and 
 OpenSuse in VMware. I wanted to check the system metrics so, I
>installed 
 node_exporters in both(Debian and OpenSuse).
 But from prometheus server I can only access debian metrics,
>opensuse 
 target is showing down. (Error: Context time out).
 Can anyone help me to config node_exporter in opensuse.


>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/887c33d9-b57a-4bdd-9b66-c7ab80187eda%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/877BA553-16F5-4E07-8139-933AC02A3916%40Jahingo.com.

Re: [prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

I used chrome in opensuse machine to check the metrics.

On Monday, 25 May 2020 14:30:12 UTC+5:30, Stuart Clark wrote:
>
> When you say that you check the opensuse metrics with your browser, which 
> machine is that running on? Is that curl or a graphical browser? 
>
> On 25 May 2020 09:57:26 BST, Harsh bhadoria  > wrote:
>>
>> I just installed the node_exporter in my Suse env, then unzip it, went to 
>> node_exporter folder, there is an node_exporter executable file, from there 
>> I just give command (./node_exporter), it starts the server at port 9100, 
>> now from use env if I go to my browser and go to URL(localhost:9100) it 
>> shows it metrics, but in my prometheus server(running in windows side), I'm 
>> not getting access or see the metrics. On same time, same process I've done 
>> in my debian system, and by Prometheus is able to get the metrics. Problem 
>> is with node_exporter at suse side i think
>>
>> On Monday, 25 May 2020 14:21:14 UTC+5:30, Stuart Clark wrote:
>>>
>>> What is the exact command being used (with all the command line 
>>> options)? 
>>>
>>> On 25 May 2020 09:50:16 BST, Harsh bhadoria  
>>> wrote:

 ./node_exporter. command starts the node_exporter server. 
 On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:
>
> What command line options are being used for them? You can use ps to 
> check or look at the systemd unit
>
> On 25 May 2020 09:47:11 BST, Harsh bhadoria  
> wrote:
>>
>> In debian i've installed the node_exporter and ran it, now I can see 
>> the metrics from prometheus which is in windows, same I've installed in 
>> opensuse also, and running node_exporter, but I'm not getting info in 
>> prometheus.
>>
>> On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:
>>>
>>> Yes, i can see the metrics from opensuse while going through 
>>> localhost:9100, but not getting it from prometheus
>>>
>>>
>>> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:

 Are you able to use curl or wget to access the node exporter from 
 the box or the Prometheus server? 

 On 25 May 2020 09:22:35 BST, Harsh bhadoria  
 wrote:
>
> I've installed Prometheus in my windows machine and installed 
> Debian and OpenSuse in VMware. I wanted to check the system metrics 
> so, I 
> installed node_exporters in both(Debian and OpenSuse).
> But from prometheus server I can only access debian metrics, 
> opensuse target is showing down. (Error: Context time out).
> Can anyone help me to config node_exporter in opensuse.
>
>
 -- 
 Sent from my Android device with K-9 Mail. Please excuse my brevity.

>>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f5b6ef25-79a5-47bb-ab96-ae587992bd24%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Stuart Clark]

Are you able to use curl or wget to access the node exporter from the box or 
the Prometheus server? 

On 25 May 2020 09:22:35 BST, Harsh bhadoria  wrote:
>I've installed Prometheus in my windows machine and installed Debian
>and 
>OpenSuse in VMware. I wanted to check the system metrics so, I
>installed 
>node_exporters in both(Debian and OpenSuse).
>But from prometheus server I can only access debian metrics, opensuse 
>target is showing down. (Error: Context time out).
>Can anyone help me to config node_exporter in opensuse.
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/9dcdbe57-6bdd-4ff7-afb6-465a4183796e%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6EA4DA83-254A-40E8-9DFC-57ECC1B6199D%40Jahingo.com.

Re: [prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

Yes, i can see the metrics from opensuse while going through 
localhost:9100, but not getting it from prometheus


On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>
> Are you able to use curl or wget to access the node exporter from the box 
> or the Prometheus server? 
>
> On 25 May 2020 09:22:35 BST, Harsh bhadoria  > wrote:
>>
>> I've installed Prometheus in my windows machine and installed Debian and 
>> OpenSuse in VMware. I wanted to check the system metrics so, I installed 
>> node_exporters in both(Debian and OpenSuse).
>> But from prometheus server I can only access debian metrics, opensuse 
>> target is showing down. (Error: Context time out).
>> Can anyone help me to config node_exporter in opensuse.
>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/100c6a28-e45e-4530-adeb-5a2b23f200d3%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Stuart Clark]

What is the configuration being used for the node exporter? Could you be 
binding to only localhost rather than and external network interface?

Are there any firewalls between the server and Prometheus? 

On 25 May 2020 09:44:54 BST, Harsh bhadoria  wrote:
>Yes, i can see the metrics from opensuse while going through 
>localhost:9100, but not getting it from prometheus
>
>
>On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>>
>> Are you able to use curl or wget to access the node exporter from the
>box 
>> or the Prometheus server? 
>>
>> On 25 May 2020 09:22:35 BST, Harsh bhadoria > > wrote:
>>>
>>> I've installed Prometheus in my windows machine and installed Debian
>and 
>>> OpenSuse in VMware. I wanted to check the system metrics so, I
>installed 
>>> node_exporters in both(Debian and OpenSuse).
>>> But from prometheus server I can only access debian metrics,
>opensuse 
>>> target is showing down. (Error: Context time out).
>>> Can anyone help me to config node_exporter in opensuse.
>>>
>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/100c6a28-e45e-4530-adeb-5a2b23f200d3%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/48F053AF-F3D8-477B-9932-D32046F78FB4%40Jahingo.com.

Re: [prometheus-users] Node_exporter in opensuse

[Stuart Clark]

When you say that you check the opensuse metrics with your browser, which 
machine is that running on? Is that curl or a graphical browser? 

On 25 May 2020 09:57:26 BST, Harsh bhadoria  wrote:
>I just installed the node_exporter in my Suse env, then unzip it, went
>to 
>node_exporter folder, there is an node_exporter executable file, from
>there 
>I just give command (./node_exporter), it starts the server at port
>9100, 
>now from use env if I go to my browser and go to URL(localhost:9100) it
>
>shows it metrics, but in my prometheus server(running in windows side),
>I'm 
>not getting access or see the metrics. On same time, same process I've
>done 
>in my debian system, and by Prometheus is able to get the metrics.
>Problem 
>is with node_exporter at suse side i think
>
>On Monday, 25 May 2020 14:21:14 UTC+5:30, Stuart Clark wrote:
>>
>> What is the exact command being used (with all the command line
>options)? 
>>
>> On 25 May 2020 09:50:16 BST, Harsh bhadoria > > wrote:
>>>
>>> ./node_exporter. command starts the node_exporter server. 
>>> On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:

 What command line options are being used for them? You can use ps
>to 
 check or look at the systemd unit

 On 25 May 2020 09:47:11 BST, Harsh bhadoria 
>
 wrote:
>
> In debian i've installed the node_exporter and ran it, now I can
>see 
> the metrics from prometheus which is in windows, same I've
>installed in 
> opensuse also, and running node_exporter, but I'm not getting info
>in 
> prometheus.
>
> On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:
>>
>> Yes, i can see the metrics from opensuse while going through 
>> localhost:9100, but not getting it from prometheus
>>
>>
>> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:
>>>
>>> Are you able to use curl or wget to access the node exporter
>from the 
>>> box or the Prometheus server? 
>>>
>>> On 25 May 2020 09:22:35 BST, Harsh bhadoria
> 
>>> wrote:

 I've installed Prometheus in my windows machine and installed
>Debian 
 and OpenSuse in VMware. I wanted to check the system metrics
>so, I 
 installed node_exporters in both(Debian and OpenSuse).
 But from prometheus server I can only access debian metrics, 
 opensuse target is showing down. (Error: Context time out).
 Can anyone help me to config node_exporter in opensuse.


>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my
>brevity.
>>>
>>
 -- 
 Sent from my Android device with K-9 Mail. Please excuse my
>brevity.

>>>
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/10501952-31a9-4b14-8162-b9753efa1415%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/0F03A39E-C366-4119-9095-F7104C80DF26%40Jahingo.com.

Re: [prometheus-users] Not get current value from Prometheus using Grok Exporter

[Deep saraswat]

Hi stuart, 

Thanks for your prompt response.

I m trying to create a tabel dashboard on grafana showing current tabel 
undergoing checksumming and its completion percentage. As i collect these 
metrics from a file where these values get printed as things progress. But when 
i plot this tabel dashboard it shows stale values for tabels which have already 
completed checksuming that too even 10-12 hrs back.

Because of these i end up getting too many values on my dashboard whereas i m 
only intrested in current one.

Regards 
Deependra 

Sent from my iPhone

> On 25-May-2020, at 2:10 PM, Stuart Clark  wrote:
> 
> Until a new value is scraped, that 48% is the current value.
> 
> As a metrics system Prometheus scrapes values regularly. As a result it is 
> unlikely that a scrape will fall exactly on the moment needed for a query to 
> be processed. Instead Prometheus will look back to find the latest value and 
> use that (with a limit, after which a time series is marked as stale).
> 
> What are you trying to achieve? 
> 
>> On 25 May 2020 09:32:02 BST, Deep saraswat  wrote:
>> I have a Grok exporter gauge metric which returning mysql checksum 
>> completion percentage by reading and parsing from a file. I'm using grafana 
>> dashboard to visualize Prometheus values.
>> 
>> 
>> 
>> 
>> 
>> Problem is Prometheus holding last value of metric until a new value come 
>> for it. For example if last value was 48% then prometheus will keep on 
>> displaying that value until getting a new value.My requirement is, only 
>> return current values of the completion percentage
>> 
>> grok exporter config:
>> 
>> global:
>>   config_version: 3
>> input:
>>   type: file
>>   paths:
>>   - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.out
>>   - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.err
>> imports:
>> - type: grok_patterns
>>   dir: ./patterns
>> grok_patterns:
>> - 'CHECKSUMOUT [a-zA-Z ]*'
>> metrics:
>> - type: counter
>>   name: mysql_ptchecksum_checksum_stats
>>   help: mysql ptchecksum stats.
>>   match: 
>> '%{MONTHDAY:month}\-%{MONTHDAY:date}\T%{TIME:logtime}\s.*%{INT:ERRORS}\s.*%{INT:DIFFS}\s.*%{NUMBER:ROWS}\s.*%{INT:DIFF_ROWS}\s.*%{INT:CHUNKS}\s.*%{INT:SKIPPED}\s.*%{NUMBER:time}
>>  %{GREEDYDATA:TABLE}'
>>   labels:
>> diffs: '{{.DIFFS}}'
>> table: '{{.TABLE}}'
>> date: '{{.date}}'
>> errors: '{{.ERRORS}}'
>> rows: '{{.ROWS}}'
>> diff_rows: '{{.DIFF_ROWS}}'
>> chunks: '{{.CHUNKS}}'
>> TS: '{{.logtime}}'
>> time: '{{.time}}'
>> - type: gauge
>>   name: mysql_ptchecksum_checksuming_status
>>   help: mysql ptchecksum cheksuming status.
>>   match: 
>> '%{GREEDYDATA:checksum_status}\s%{GREEDYDATA:tabel}\:.*%{NUMBER:completion_percentage}\%\s.*%{NUMBER:TimeRemainingHours}\:%{NUMBER:TimeRemainingMinutes}'
>>   value: '{{.completion_percentage}}'
>>   labels:
>> status: '{{.checksum_status}}'
>> tabel: '{{.tabel}}'
>> - type: gauge
>>   name: mysql_ptchecksum_waiting_status
>>   help: mysql ptchecksum waiting status.
>>   match: 'Replica 
>> lag\s.*\is\s.*%{INT:seconds}\s%{GREEDYDATA:data}\son\s%{GREEDYDATA:server}\.\s'
>>   value: '{{.seconds}}'
>>   labels:
>> server: '{{.server}}'
>> 
>> server:
>>   protocol: http
>>   port: 9144
>> 
>> please help me resolve this.
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/EC9BDCE7-83B7-4CA8-A625-7914974095F8%40gmail.com.

[prometheus-users] Recomended EC2 instance type for running latest Prometheus

[Vivek Lanjekar]

i was thinking of Running Prometheus server in i3en.xlarge as it has local 
storage and will get better IO performance than ebs. Please can any one 
share their opinion on this or may be suggest better EC2 instance type for 
running Prometheus in Production environment. Thanks in advance

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c6741e37-f193-43ce-9feb-53d3c5f360ab%40googlegroups.com.

Re: [prometheus-users] Not get current value from Prometheus using Grok Exporter

[Stuart Clark]

In which case I'd suggest producing an additional metric where the value is the 
unix timestamp of the last update. You can then update your dashboard query to 
exclude those which haven't been updated for a certain length of time.

However I would caution against getting too close to trying to use Prometheus 
as an event recording system. If that is really what you are wanting other 
tools may be a better fit. 

On 25 May 2020 13:34:28 BST, Deep saraswat  wrote:
>Hi stuart, 
>
>Thanks for your prompt response.
>
>I m trying to create a tabel dashboard on grafana showing current tabel
>undergoing checksumming and its completion percentage. As i collect
>these metrics from a file where these values get printed as things
>progress. But when i plot this tabel dashboard it shows stale values
>for tabels which have already completed checksuming that too even 10-12
>hrs back.
>
>Because of these i end up getting too many values on my dashboard
>whereas i m only intrested in current one.
>
>Regards 
>Deependra 
>
>Sent from my iPhone
>
>> On 25-May-2020, at 2:10 PM, Stuart Clark 
>wrote:
>> 
>> Until a new value is scraped, that 48% is the current value.
>> 
>> As a metrics system Prometheus scrapes values regularly. As a result
>it is unlikely that a scrape will fall exactly on the moment needed for
>a query to be processed. Instead Prometheus will look back to find the
>latest value and use that (with a limit, after which a time series is
>marked as stale).
>> 
>> What are you trying to achieve? 
>> 
>>> On 25 May 2020 09:32:02 BST, Deep saraswat 
>wrote:
>>> I have a Grok exporter gauge metric which returning mysql checksum
>completion percentage by reading and parsing from a file. I'm using
>grafana dashboard to visualize Prometheus values.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Problem is Prometheus holding last value of metric until a new value
>come for it. For example if last value was 48% then prometheus will
>keep on displaying that value until getting a new value.My requirement
>is, only return current values of the completion percentage
>>> 
>>> grok exporter config:
>>> 
>>> global:
>>>   config_version: 3
>>> input:
>>>   type: file
>>>   paths:
>>>   - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.out
>>>   - /root/grok_exporter-1.0.0.RC3.linux-amd64/example/ptchecksum.err
>>> imports:
>>> - type: grok_patterns
>>>   dir: ./patterns
>>> grok_patterns:
>>> - 'CHECKSUMOUT [a-zA-Z ]*'
>>> metrics:
>>> - type: counter
>>>   name: mysql_ptchecksum_checksum_stats
>>>   help: mysql ptchecksum stats.
>>>   match:
>'%{MONTHDAY:month}\-%{MONTHDAY:date}\T%{TIME:logtime}\s.*%{INT:ERRORS}\s.*%{INT:DIFFS}\s.*%{NUMBER:ROWS}\s.*%{INT:DIFF_ROWS}\s.*%{INT:CHUNKS}\s.*%{INT:SKIPPED}\s.*%{NUMBER:time}
>%{GREEDYDATA:TABLE}'
>>>   labels:
>>> diffs: '{{.DIFFS}}'
>>> table: '{{.TABLE}}'
>>> date: '{{.date}}'
>>> errors: '{{.ERRORS}}'
>>> rows: '{{.ROWS}}'
>>> diff_rows: '{{.DIFF_ROWS}}'
>>> chunks: '{{.CHUNKS}}'
>>> TS: '{{.logtime}}'
>>> time: '{{.time}}'
>>> - type: gauge
>>>   name: mysql_ptchecksum_checksuming_status
>>>   help: mysql ptchecksum cheksuming status.
>>>   match:
>'%{GREEDYDATA:checksum_status}\s%{GREEDYDATA:tabel}\:.*%{NUMBER:completion_percentage}\%\s.*%{NUMBER:TimeRemainingHours}\:%{NUMBER:TimeRemainingMinutes}'
>>>   value: '{{.completion_percentage}}'
>>>   labels:
>>> status: '{{.checksum_status}}'
>>> tabel: '{{.tabel}}'
>>> - type: gauge
>>>   name: mysql_ptchecksum_waiting_status
>>>   help: mysql ptchecksum waiting status.
>>>   match: 'Replica
>lag\s.*\is\s.*%{INT:seconds}\s%{GREEDYDATA:data}\son\s%{GREEDYDATA:server}\.\s'
>>>   value: '{{.seconds}}'
>>>   labels:
>>> server: '{{.server}}'
>>> 
>>> server:
>>>   protocol: http
>>>   port: 9144
>>> 
>>> please help me resolve this.
>> 
>> -- 
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8CF587FB-C649-415E-9DEF-9382A647CB3B%40Jahingo.com.

[prometheus-users] Re: Adding extra labels to metrics

[Juan Rosero]

Awesome! That worked! Thanks for everything. Leaving the solution here.

  - targets:
  - server1;ad-1;domain-1;emea-1
  - server2;ad-2;domain-2;emea-1
relabel_configs:
- source_labels: ['__address__']
  regex: '(.*);(.*);(.*);(.*)'
  target_label: 'ad'
  replacement: '$2'
- source_labels: ['__address__']
  regex: '(.*);(.*);(.*);(.*)'
  target_label: 'dom'
  replacement: '$3'
- source_labels: ['__address__']
  regex: '(.*);(.*);(.*);(.*)'
  target_label: 'reg'
  replacement: '$4'
- source_labels: ['__address__']
  regex: '(.*);(.*);(.*);(.*)'
  target_label: '__address__'
  replacement: '$1:9117'


On Thursday, May 21, 2020 at 2:00:35 PM UTC-7, Juan Rosero wrote:
>
> Hello,
>
> I'm trying to add location labels to my Apache Exporter metrics and other 
> exporters that don't expose them when Prometheus scrapes them from the 
> server nodes. Here is my Prometheus job and the labels (ad, dom, reg) that 
> I'm looking to add depending on the server location. Please let me know if 
> it's possible to be added on the exporter end or maybe in Prometheus while 
> it ingest those metrics. Many thanks!
>
>
> - job_name: apache_server
>   static_configs:
>   - targets: ["server1:9117", "server2:9117", "server3:9117", 
> "server4:9117", "server5:9117", "server6:9117",]
> labels:
>   env: "int"
>   app: "apache"
>   version: "2.4.43"
>   ad: ""
>   dom: ""
>   reg: ""
>
> Here how I'm looking for the metrics to look like:
> {env="int", ad="ad-1", dom="domain-1", reg="emea-1", 
> instance="server1:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-2", dom="domain-2", reg="emea-1", 
> instance="server2:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-1", reg="emea-1", 
> instance="server3:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-1", reg="emea-1", 
> instance="server4:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-2", reg="emea-1", 
> instance="server5:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-1", reg="emea-1", 
> instance="server6:9117", app="apache", version="2.4.43"}
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/3a7b86ce-fa97-437d-a931-c48b17b3da96%40googlegroups.com.

[prometheus-users] Cannot save metrics with timestamp

[shimpeko]

Hello,

Metrics are not saved when I put a timestamps with metrics.
I've got through github issue and some of prometheus codes but not yet 
getting the root cause.

I would be happy if I can get why I'm getting below errors.

Errors
*Error 1*
I got below error when executing prometheus with prometheus.yml (v1). 
Please see blow for the prometheus.yml (v1).

prometheus_1  | level=warn ts=2020-05-25T12:29:17.216Z caller=scrape.go:1240 
component="scrape manager" scrape_pool=my-app target=https://:443/metrics msg="Error on ingesting samples that are too 
old or are too far into the future" num_dropped=600

*Error2*
I got massive amount of below log when running prometheus with 
prometheus.yml (v2). Please see below for prometheus.yml (v2)

prometheus_1  | level=info ts=2020-05-25T15:57:25.857Z caller=head.go:732 
component=tsdb msg="WAL checkpoint complete" first=1164 last=1165 
duration=2.1818ms
prometheus_1  | level=info ts=2020-05-25T15:57:25.867Z caller=head.go:662 
component=tsdb msg="head GC completed" duration=509.9µs
prometheus_1  | level=info ts=2020-05-25T15:57:25.879Z caller=head.go:662 
component=tsdb msg="head GC completed" duration=549.7µs
prometheus_1  | level=info ts=2020-05-25T15:57:25.882Z caller=head.go:732 
component=tsdb msg="WAL checkpoint complete" first=1166 last=1167 
duration=2.7017ms
prometheus_1  | level=info ts=2020-05-25T15:57:25.892Z caller=head.go:662 
component=tsdb msg="head GC completed" duration=564.7µs
prometheus_1  | level=info ts=2020-05-25T15:57:25.903Z caller=head.go:662 
component=tsdb msg="head GC completed" duration=651.9µs
prometheus_1  | level=info ts=2020-05-25T15:57:25.906Z caller=head.go:732 
component=tsdb msg="WAL checkpoint complete" first=1168 last=1169 
duration=2.8861ms
prometheus_1  | level=info ts=2020-05-25T15:57:25.916Z caller=head.go:662 
component=tsdb msg="head GC completed" duration=562.8µs
prometheus_1  | level=info ts=2020-05-25T15:57:25.925Z caller=head.go:662 
component=tsdb msg="head GC completed" duration=633µs


Background

I'm testing prometheus for custom app.
I've created endpoint which expose metrics in text-based format for the app.

Since metrics are calculated with batch job, I've put a timestamps to 
metrics to show when the metrics was calculated.
The batch job is executed in every 5 min, so the timestamp will not be 
older than 5 min compared to the current timestamp.

Here is an example of exposed data. There are about 600 lines in real data.

#HELP processed_rows_total the total number of processed rows
#TYPE processed_rows_total counter
processed_rows_total{stream_name="a", result="success"} 93163 1590422899
processed_rows_total{stream_name="a", result="error"} 0 1590422899
processed_rows_total{stream_name="b", result="success"} 45695851 1590422899

processed_rows_total{stream_name="b", result="error"} 0 1590422899


Environment

Running with docker.

docker-compose.yml

prometheus:
  image: prom/prometheus:v2.17.2
  entrypoint:
- "/bin/prometheus"
- "--log.level=debug"
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.console.libraries=/usr/share/prometheus/console_libraries"
- "--web.console.templates=/usr/share/prometheus/consoles"
  volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
  ports:
- "127.0.0.1:9090:9090"

prometheus.yml (v1)

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. 
Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default 
is every 1 minute.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries 
scraped from this config.
  - job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

  - job_name: 'my-app'
metrics_path: '/metrics'
scrape_interval: 1m
scheme: 'https'
static_configs:
- targets: ['']

prometheus.yml (v2)

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. 
Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default 
is every 1 minute.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'my-app'
metrics_path: '/metrics'
scrape_interval: 1m
scheme: 'https'
static_configs:
- targets: ['']

Other notes

With *honor-timestamps: false* configuration in prometheus.yml, metrics are 
saved correctly.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

[prometheus-users] Alertmanager issue

[piyush sharma]

Hi all,

Thanks for all the guidance you all have given me.
I came across something regarding alert manager.
I have provisioned alertmanager to send alerts to slack.
Sometimes I notice that I get resolved notification for the alert that never 
triggered.

Why is this happening. Is there something wrong in my configuration. How to get 
rid of it.
.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/aa9d8db7-278e-4e94-80dd-09e9d645f118%40googlegroups.com.

[prometheus-users] How to exclude all labels containing '3scale' in their name during FEDERATION phase

[Fabio Marzotti]

Hi, I need to exclude all labels containing an error name. For example 
those starting with '3scale'.
This in order to avoid this error during federation phase:

"prometheus   | level=warn ts=2020-05-25T18:15:57.094907355Z 
caller=scrape.go:867 component="scrape manager" scrape_pool= 
target="https://x:443/federate?match%5B%5D=%7B__name__%3D~%22.%2B%22%7D;
 
msg="append failed" err="expected label name, got \"INVALID\""

May be is the match phase in  any case after federation process and then 
I'll get the same error ??

Thanks for the support, 

Fabio
  



-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/3ccfa579-360e-4b56-b7a6-8aae153315fe%40googlegroups.com.

[prometheus-users] Thanos Object Storage Size

[Saurabh Jain]

Hi,

How much approx size should be allocated for Thanos Object Storage where we 
have to monitor OpenStack environment on 5 sites.

Also we have to use Node and Collectd Exporter to monitor system parameters 
as well.

Pls advice.

Thanks,
Saurabh

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d3eebefe-d541-4b7f-bd1b-b30c2477714b%40googlegroups.com.

Re: [prometheus-users] Node_exporter in opensuse

[Harsh bhadoria]

Figured out the issue, there was firewall issue.
By the way, thanks for your suggestions.

On Monday, 25 May 2020 14:45:42 UTC+5:30, Stuart Clark wrote:
>
> If you use that IP address rather than localhost on the opensuse machine 
> (in the web browser) do you get metrics? 
>
> On 25 May 2020 10:14:03 BST, Harsh bhadoria  > wrote:
>>
>> In scrape_configs, i added the "job-name: 'node_exporter' " and in static 
>> config, I've added targets as IP address of debian and opensuse.
>> Like this, 
>> - target: ['194.167.200.146:9100','194.167.200.147:9100']
>>
>> On Monday, 25 May 2020 14:36:34 UTC+5:30, Stuart Clark wrote:
>>>
>>> What configuration do you have set for the different node exporters in 
>>> your prometheus.yml?
>>>
>>> On 25 May 2020 10:03:03 BST, Harsh bhadoria  
>>> wrote:

 I used chrome in opensuse machine to check the metrics.

 On Monday, 25 May 2020 14:30:12 UTC+5:30, Stuart Clark wrote:
>
> When you say that you check the opensuse metrics with your browser, 
> which machine is that running on? Is that curl or a graphical browser? 
>
> On 25 May 2020 09:57:26 BST, Harsh bhadoria  
> wrote:
>>
>> I just installed the node_exporter in my Suse env, then unzip it, 
>> went to node_exporter folder, there is an node_exporter executable file, 
>> from there I just give command (./node_exporter), it starts the server 
>> at 
>> port 9100, now from use env if I go to my browser and go to 
>> URL(localhost:9100) it shows it metrics, but in my prometheus 
>> server(running in windows side), I'm not getting access or see the 
>> metrics. 
>> On same time, same process I've done in my debian system, and by 
>> Prometheus 
>> is able to get the metrics. Problem is with node_exporter at suse side i 
>> think
>>
>> On Monday, 25 May 2020 14:21:14 UTC+5:30, Stuart Clark wrote:
>>>
>>> What is the exact command being used (with all the command line 
>>> options)? 
>>>
>>> On 25 May 2020 09:50:16 BST, Harsh bhadoria  
>>> wrote:

 ./node_exporter. command starts the node_exporter server. 
 On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote:
>
> What command line options are being used for them? You can use ps 
> to check or look at the systemd unit
>
> On 25 May 2020 09:47:11 BST, Harsh bhadoria  
> wrote:
>>
>> In debian i've installed the node_exporter and ran it, now I can 
>> see the metrics from prometheus which is in windows, same I've 
>> installed in 
>> opensuse also, and running node_exporter, but I'm not getting info 
>> in 
>> prometheus.
>>
>> On Monday, 25 May 2020 14:14:54 UTC+5:30, Harsh bhadoria wrote:
>>>
>>> Yes, i can see the metrics from opensuse while going through 
>>> localhost:9100, but not getting it from prometheus
>>>
>>>
>>> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote:

 Are you able to use curl or wget to access the node exporter 
 from the box or the Prometheus server? 

 On 25 May 2020 09:22:35 BST, Harsh bhadoria <
 harshbha...@gmail.com> wrote:
>
> I've installed Prometheus in my windows machine and installed 
> Debian and OpenSuse in VMware. I wanted to check the system 
> metrics so, I 
> installed node_exporters in both(Debian and OpenSuse).
> But from prometheus server I can only access debian metrics, 
> opensuse target is showing down. (Error: Context time out).
> Can anyone help me to config node_exporter in opensuse.
>
>
 -- 
 Sent from my Android device with K-9 Mail. Please excuse my 
 brevity.

>>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my 
> brevity.
>

>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

>>> -- 
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>
>>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8196f4fe-043b-4251-91b5-421503639dc4%40googlegroups.com.

[prometheus-users] Re: Cannot save metrics with timestamp

[shimpeko]

Resolved.

Precision of timestamp must be millisecond but I've had put second.

On Tuesday, May 26, 2020 at 1:30:39 AM UTC+9, shimpeko wrote:
>
> Hello,
>
> Metrics are not saved when I put a timestamps with metrics.
> I've got through github issue and some of prometheus codes but not yet 
> getting the root cause.
>
> I would be happy if I can get why I'm getting below errors.
>
> Errors
> *Error 1*
> I got below error when executing prometheus with prometheus.yml (v1). 
> Please see blow for the prometheus.yml (v1).
>
> prometheus_1  | level=warn ts=2020-05-25T12:29:17.216Z caller=scrape.go:
> 1240 component="scrape manager" scrape_pool=my-app target=https:// server address>:443/metrics msg="Error on ingesting samples that are too 
> old or are too far into the future" num_dropped=600
>
> *Error2*
> I got massive amount of below log when running prometheus with 
> prometheus.yml (v2). Please see below for prometheus.yml (v2)
>
> prometheus_1  | level=info ts=2020-05-25T15:57:25.857Z caller=head.go:732 
> component=tsdb msg="WAL checkpoint complete" first=1164 last=1165 
> duration=2.1818ms
> prometheus_1  | level=info ts=2020-05-25T15:57:25.867Z caller=head.go:662 
> component=tsdb msg="head GC completed" duration=509.9µs
> prometheus_1  | level=info ts=2020-05-25T15:57:25.879Z caller=head.go:662 
> component=tsdb msg="head GC completed" duration=549.7µs
> prometheus_1  | level=info ts=2020-05-25T15:57:25.882Z caller=head.go:732 
> component=tsdb msg="WAL checkpoint complete" first=1166 last=1167 
> duration=2.7017ms
> prometheus_1  | level=info ts=2020-05-25T15:57:25.892Z caller=head.go:662 
> component=tsdb msg="head GC completed" duration=564.7µs
> prometheus_1  | level=info ts=2020-05-25T15:57:25.903Z caller=head.go:662 
> component=tsdb msg="head GC completed" duration=651.9µs
> prometheus_1  | level=info ts=2020-05-25T15:57:25.906Z caller=head.go:732 
> component=tsdb msg="WAL checkpoint complete" first=1168 last=1169 
> duration=2.8861ms
> prometheus_1  | level=info ts=2020-05-25T15:57:25.916Z caller=head.go:662 
> component=tsdb msg="head GC completed" duration=562.8µs
> prometheus_1  | level=info ts=2020-05-25T15:57:25.925Z caller=head.go:662 
> component=tsdb msg="head GC completed" duration=633µs
>
>
> Background
>
> I'm testing prometheus for custom app.
> I've created endpoint which expose metrics in text-based format for the 
> app.
>
> Since metrics are calculated with batch job, I've put a timestamps to 
> metrics to show when the metrics was calculated.
> The batch job is executed in every 5 min, so the timestamp will not be 
> older than 5 min compared to the current timestamp.
>
> Here is an example of exposed data. There are about 600 lines in real data.
>
> #HELP processed_rows_total the total number of processed rows
> #TYPE processed_rows_total counter
> processed_rows_total{stream_name="a", result="success"} 93163 1590422899
> processed_rows_total{stream_name="a", result="error"} 0 1590422899
> processed_rows_total{stream_name="b", result="success"} 45695851 1590422899
>
> processed_rows_total{stream_name="b", result="error"} 0 1590422899
>
>
> Environment
>
> Running with docker.
>
> docker-compose.yml
>
> prometheus:
>   image: prom/prometheus:v2.17.2
>   entrypoint:
> - "/bin/prometheus"
> - "--log.level=debug"
> - "--config.file=/etc/prometheus/prometheus.yml"
> - "--storage.tsdb.path=/prometheus"
> - "--web.console.libraries=/usr/share/prometheus/console_libraries"
> - "--web.console.templates=/usr/share/prometheus/consoles"
>   volumes:
> - ./prometheus.yml:/etc/prometheus/prometheus.yml
>   ports:
> - "127.0.0.1:9090:9090"
>
> prometheus.yml (v1)
>
> # my global config
> global:
>   scrape_interval: 15s # Set the scrape interval to every 15 seconds. 
> Default is every 1 minute.
>   evaluation_interval: 15s # Evaluate rules every 15 seconds. The default 
> is every 1 minute.
>
> # A scrape configuration containing exactly one endpoint to scrape:
> # Here it's Prometheus itself.
> scrape_configs:
>   # The job name is added as a label `job=` to any timeseries 
> scraped from this config.
>   - job_name: 'prometheus'
> static_configs:
> - targets: ['localhost:9090']
>
>   - job_name: 'my-app'
> metrics_path: '/metrics'
> scrape_interval: 1m
> scheme: 'https'
> static_configs:
> - targets: ['']
>
> prometheus.yml (v2)
>
> # my global config
> global:
>   scrape_interval: 15s # Set the scrape interval to every 15 seconds. 
> Default is every 1 minute.
>   evaluation_interval: 15s # Evaluate rules every 15 seconds. The default 
> is every 1 minute.
>
> # A scrape configuration containing exactly one endpoint to scrape:
> # Here it's Prometheus itself.
> scrape_configs:
>   - job_name: 'my-app'
> metrics_path: '/metrics'
> scrape_interval: 1m
> scheme: 'https'
> static_configs:
> - targets: ['']
>
> Other notes
>
> With *honor-timestamps: false* configuration in prometheus.yml, metrics 
> are 

Re: [prometheus-users] Hosted Prometheus with Grafana Cloud

[Ben Kochie]

On Tue, May 19, 2020 at 1:42 PM Colton Conor  wrote:

> We are exploring the option of paying for Grafana Cloud's service. In
> addition to hosting Grafana, it comes with the ability to store metrics
> from Prometheus and Graphite. The documentation says:
>
> To send data using Prometheus you need the following:
>
>- A running instance of Prometheus.
>- In your Prometheus configuration, add a remote_write section.
>
> So it doesn't really sound like this Cloud option actually host
> Prometheus itself, but instead just collect metrics from an onsite
> Prometheus server which then sends it to the cloud using remote_write.
>
> My question is how beneficial is this to just running our own onsite
> assuming we are going to have to install and run Prometheus anyways? Would
> the one-site Prometheus sever just not need that much storage space since
> it would remote write to the cloud, and then delete the onsite metrics?
>

That's one of the big benefits. The local Prometheus servers only need to
keep a short retention time. Only enough to execute alerts to your
alertmanager cluster. Another benefit is that with the external Grafana
service, query load for dashboards don't hit your Prometheus server. It
only needs to collect and run rules. This can reduce CPU and memory
requirements for the local service.


>
> Also, since Prometheus also supports exporting to Graphite, which is also
> supported by the cloud service, would it be better to send the metrics
> using Prometheus with remote_write, or Prometheus export to
> Graphite using carbon-relay-ng?
>

Definitely better to use the native Prometheus protocol.


>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAMDdSzOYazWpaTuZY21Ob0quWSPoqQgNKD0HvH_GU9PwjA%3DDDw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmoVdXvrXs9krrpLaqbLEYUqRzFCRP5wHtNXsxDRL65mXA%40mail.gmail.com.

Re: [prometheus-users] Recomended EC2 instance type for running latest Prometheus

[Ben Kochie]

Prometheus doesn't typically require high performance storage. HDD or EBS
work just fine.

Sizing your instance depends on how much you're monitoring.

This might help you:
https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

On Mon, May 25, 2020 at 5:27 PM Vivek Lanjekar 
wrote:

> i was thinking of Running Prometheus server in i3en.xlarge as it has local
> storage and will get better IO performance than ebs. Please can any one
> share their opinion on this or may be suggest better EC2 instance type for
> running Prometheus in Production environment. Thanks in advance
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c6741e37-f193-43ce-9feb-53d3c5f360ab%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmpPb6hXsXe-0E-T6tYJ5zTKpx5c%2BS1qVg5Hp26Xc3%3DS-A%40mail.gmail.com.