For context and maybe upcomming errors here is my prom.yml so far:
# my global config
global:
scrape_interval: 1m # Set the scrape interval to every 15 seconds.
Default is every 1 minute.
evaluation_interval: 30s # Evaluate rules every 15 seconds. The default
is every 1 minute.
scrape_timeout: 30s
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
# Load rules once and periodically evaluate them according to the global
'evaluation_interval'.
rule_files:
- rules.yml
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries
scraped from this config.
- job_name: 'Windows-Exporter Host-Server'
file_sd_configs:
- files:
- C:\Prometheus\targets\hosts.yml
relabel_configs:
# When __address__ is of the form "name address", extract
# name to "instance" label and address to "__address__"
- source_labels: [__address__]
regex: '(.+) (.+)'
target_label: instance
replacement: '${1}'
- source_labels: [__address__]
regex: '(.+) (.+)'
target_label: __address__
replacement: '${2}'
[...]
- job_name: 'Ping (ICMP)'
metrics_path: /probe
params:
module: [icmp_ttl5]
file_sd_configs:
- files:
- C:\Prometheus\targets\hosts.yml
- C:\Prometheus\targets\netcomponents.yml
- C:\Prometheus\targets\vmhost-05.yml
- C:\Prometheus\targets\vmhost-06.yml
- C:\Prometheus\targets\vmhost-08.yml
- C:\Prometheus\targets\vmhost-09.yml
- C:\Prometheus\targets\vmhost-10.yml
- C:\Prometheus\targets\vmopheo-01.yml
- C:\Prometheus\targets\vmopheo-02.yml
- C:\Prometheus\targets\vmopheo-04.yml
- C:\Prometheus\targets\vmopheo-06.yml
- C:\Prometheus\targets\vmopheo-07.yml
- C:\Prometheus\targets\vmopheo-08.yml
relabel_configs:
- source_labels: [__address__]
regex: '([^ ]+)' # name or address only
target_label: instance
- source_labels: [__address__]
regex: '([^ ]+)' # name or address only
target_label: __param_target
- source_labels: [__address__]
regex: '(.+) (.+)' # name address
target_label: instance
replacement: '${1}'
- source_labels: [__address__]
regex: '(.+) (.+)' # name address
target_label: __param_target
replacement: '${2}'
- source_labels: [module]
target_label: __param_module
- target_label: __address__
replacement: 127.0.0.1:9115 # Blackbox exporter
Kolja Krückmann schrieb am Freitag, 28. April 2023 um 14:10:00 UTC+2:
> Hi Brian, thanks for your quick response.
>
> Actually I just pasted the relabeling you attached and it immediately
> worked :)
>
> Brian Candler schrieb am Freitag, 28. April 2023 um 13:37:33 UTC+2:
>
>> Do you want to show what you've done so far?
>>
>> You can read these:
>>
>>
>> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-blackbox-exporter.html
>>
>> # simple config
>>
>> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-relabeling.html
>>
>> # node_exporter with relabelling
>>
>> If you combine the two, you could end up with something like this:
>>
>> - job_name: blackbox
>> file_sd_configs:
>> - files:
>> - /etc/prometheus/blackbox.d/*.yml
>> metrics_path: /probe
>> relabel_configs:
>> - source_labels: [__address__]
>> regex: '([^ ]+)' # name or address only
>> target_label: instance
>> - source_labels: [__address__]
>> regex: '([^ ]+)' # name or address only
>> target_label: __param_target
>> - source_labels: [__address__]
>> regex: '(.+) (.+)' # name address
>>
>> target_label: instance
>> replacement: '${1}'
>> - source_labels: [__address__]
>> regex: '(.+) (.+)' # name address
>> target_label: __param_target
>> replacement: '${2}'
>> - source_labels: [module]
>> target_label: __param_module
>> - target_label: __address__
>> replacement: 127.0.0.1:9115 # Blackbox exporter
>>
>> Where /etc/prometheus/blackbox.d/ping.yml contains, for example:
>>
>> - labels:
>> module: icmp
>> targets:
>> - google-pri 8.8.8.8
>> - google-sec 8.8.4.4
>>
>> Then you should get metrics like this:
>>
>> probe_success{job="blackbox",instance="google-pri",module="icmp"} 1
>>
>> On Friday, 28 April 2023 at 09:49:21 UTC+1 Kolja Krückmann wrote:
>>
>>> Hi y'all, now i got my metrics running (Thanks Brian)
>>>
>>> For the next step I want to use the Blackbox_exporter with the same
>>> targets (where the target.yml is build like "- targetname targetIP")
>>> I want to have the same labeling as in the question above where each
>>> target has a name lable but I'm not quite getting there. Either the
>>> Endpoint URL is missing the target IP or the label is missing the name of
>>> the endpoint.
>>>
>>> Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:38:59 UTC+2:
>>>
>>>> nevermind - sorry for the question, should have just googled learn
>>>> regular expression.
>>>> I'm fine for now.
>>>> Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:35:21 UTC+2:
>>>>
>>>>> Thanks so much!
>>>>>
>>>>> Can you tell me where I could kinda "learn" regex? I find it very
>>>>> difficult to get known to regex because it's nothing im using on daily
>>>>> basis ^^
>>>>>
>>>>>
>>>>>
>>>>> Brian Candler schrieb am Donnerstag, 27. April 2023 um 17:24:26 UTC+2:
>>>>>
>>>>>> So to be clear, if you want your targets file to look like this:
>>>>>>
>>>>>> - targets:
>>>>>> - SVR-DS01 172.25.X0.XXX:9182
>>>>>> - SRV-DS02 172.21.X1.XXX:9182
>>>>>>
>>>>>> you can do something like this:
>>>>>>
>>>>>> relabel_configs:
>>>>>> # When __address__ is a single item, set the instance
>>>>>> # label to the part without the port
>>>>>> - source_labels: [__address__]
>>>>>> regex: '([^ ]+):[0-9]+'
>>>>>> replacement: '${1}'
>>>>>> target_label: instance
>>>>>>
>>>>>> # When __address__ is space-separated "name address:port",
>>>>>> # put the first part in the "instance" label and leave the
>>>>>> second part
>>>>>> # in "__address__"
>>>>>> - source_labels: [__address__]
>>>>>> regex: '(.+) (.+)'
>>>>>> target_label: instance
>>>>>> replacement: '${1}'
>>>>>> - source_labels: [__address__]
>>>>>> regex: '(.+) (.+)'
>>>>>> target_label: __address__
>>>>>> replacement: '${2}'
>>>>>>
>>>>>> If your target entries *always* consist of two items separated by a
>>>>>> space, then you don't need the first rule. It's only there in case you
>>>>>> have
>>>>>> entries in the old format, i.e.
>>>>>>
>>>>>> - targets:
>>>>>> - 172.25.X0.XXX:9182
>>>>>>
>>>>>> (in which case, the instance is set to "172.25.X0.XXX")
>>>>>>
>>>>>> On Thursday, 27 April 2023 at 16:16:38 UTC+1 Brian Candler wrote:
>>>>>>
>>>>>>> Well, there is a good reason to keep the port out of the "instance"
>>>>>>> label: it's very awkward to make a query to join two different
>>>>>>> different
>>>>>>> metrics collected from two different exporters on the same host, if one
>>>>>>> has
>>>>>>> (say) instance="foo:9100" and another has instance="foo:9104". A
>>>>>>> second
>>>>>>> reason is that it's nicer in dashboards to see "foo" rather than
>>>>>>> "foo:9100"
>>>>>>>
>>>>>>> See https://www.robustperception.io/controlling-the-instance-label
>>>>>>>
>>>>>>> IMO the cleanest way to achieve this is to leave the port out of the
>>>>>>> targets file, copy __address__ to instance, and then append the
>>>>>>> exporter
>>>>>>> port statically to __address__.
>>>>>>>
>>>>>>> However, you could leave the port number in the targets file, and
>>>>>>> strip it out to create the instance label:
>>>>>>>
>>>>>>> - source_labels: [__address__]
>>>>>>> regex: '(.*):[0-9]+'
>>>>>>> target_label: instance
>>>>>>> replacement: '${1}'
>>>>>>>
>>>>>>> This would be useful if the same exporter were running on different
>>>>>>> ports on different hosts (fairly uncommon I think).
>>>>>>>
>>>>>>> Also, you risk a collision of metrics if targets list includes the
>>>>>>> same host twice on two different ports in the same scrape job (which
>>>>>>> should
>>>>>>> be very unlikely).
>>>>>>>
>>>>>>> On Thursday, 27 April 2023 at 15:14:03 UTC+1 Kolja Krückmann wrote:
>>>>>>>
>>>>>>>> Hi Brian,
>>>>>>>>
>>>>>>>> thank you very much for your response!
>>>>>>>> I just wanted to clarify, if it's better to "regex" the port to the
>>>>>>>> end of the address, or if it is equally fine to just add the port in
>>>>>>>> the
>>>>>>>> target.yml to each target? (That's what im currently doing) Is it just
>>>>>>>> best
>>>>>>>> practice with the regex expression or is there actually a technical
>>>>>>>> reason
>>>>>>>> behind that?
>>>>>>>>
>>>>>>>> Kind regards
>>>>>>>> Kolja
>>>>>>>>
>>>>>>>> Brian Candler schrieb am Dienstag, 18. April 2023 um 13:34:53 UTC+2:
>>>>>>>>
>>>>>>>>> Alternatively, you use rewriting rules.
>>>>>>>>>
>>>>>>>>> - targets:
>>>>>>>>> - SVR-DS01 172.25.X0.XXX
>>>>>>>>> - SRV-DS02 172.21.X1.XXX
>>>>>>>>> ... etc
>>>>>>>>>
>>>>>>>>> and the corresponding relabel_configs in your scrape job could be
>>>>>>>>> something like this:
>>>>>>>>>
>>>>>>>>> relabel_configs:
>>>>>>>>> # When __address__ consists of just a name or IP address,
>>>>>>>>> # copy it to the "instance" label. This keeps the port
>>>>>>>>> # number out of the instance label.
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> regex: '([^ ]+)'
>>>>>>>>> target_label: instance
>>>>>>>>>
>>>>>>>>> # When __address__ is of the form "name address", extract
>>>>>>>>> # name to "instance" label and address to "__address__"
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> regex: '(.+) (.+)'
>>>>>>>>> target_label: instance
>>>>>>>>> replacement: '${1}'
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> regex: '(.+) (.+)'
>>>>>>>>> target_label: __address__
>>>>>>>>> replacement: '${2}'
>>>>>>>>>
>>>>>>>>> # Append port number to __address__ so that scrape gets
>>>>>>>>> # sent to the right port
>>>>>>>>> - source_labels: [__address__]
>>>>>>>>> target_label: __address__
>>>>>>>>> replacement: '${1}:9182'
>>>>>>>>>
>>>>>>>>> On Tuesday, 18 April 2023 at 11:14:23 UTC+1 Julius Volz wrote:
>>>>>>>>>
>>>>>>>>>> The labels in the SD file can only be provided for each target
>>>>>>>>>> group (the top-level list item type of the YAML file), so you'd have
>>>>>>>>>> to do
>>>>>>>>>> something like this:
>>>>>>>>>>
>>>>>>>>>> - targets:
>>>>>>>>>> - [...first list of targets...]
>>>>>>>>>> labels:
>>>>>>>>>> instance: "A"
>>>>>>>>>> - targets:
>>>>>>>>>> - [...second list of targets...]
>>>>>>>>>> labels:
>>>>>>>>>> instance: "B"
>>>>>>>>>> - targets:
>>>>>>>>>> - [...third list of targets...]
>>>>>>>>>> labels:
>>>>>>>>>> instance: "C"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 18, 2023 at 11:20 AM Kolja Krückmann <
>>>>>>>>>> kolja.kr...@opheo.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi y'all I'm trying to get a label for each target in my
>>>>>>>>>>> file_sd_config
>>>>>>>>>>>
>>>>>>>>>>> my prom.yml is:
>>>>>>>>>>> (redacted)
>>>>>>>>>>>
>>>>>>>>>>> # my global config
>>>>>>>>>>> global:
>>>>>>>>>>> scrape_interval: 1m # Set the scrape interval to every 15
>>>>>>>>>>> seconds. Default is every 1 minute.
>>>>>>>>>>> evaluation_interval: 30s # Evaluate rules every 15 seconds.
>>>>>>>>>>> The default is every 1 minute.
>>>>>>>>>>> scrape_timeout: 30s
>>>>>>>>>>>
>>>>>>>>>>> scrape_configs:
>>>>>>>>>>> # The job name is added as a label `job=<job_name>` to any
>>>>>>>>>>> timeseries scraped from this config.
>>>>>>>>>>> - job_name: 'node'
>>>>>>>>>>> file_sd_configs:
>>>>>>>>>>> - files:
>>>>>>>>>>> -
>>>>>>>>>>> C:/Prometheus/prometheus-2.41.0.windows-amd64/target_cluster_b.yml
>>>>>>>>>>>
>>>>>>>>>>> my target_cluster_b.yml:
>>>>>>>>>>> - targets:
>>>>>>>>>>> - 172.25.X0.XXX:9182
>>>>>>>>>>> labels:
>>>>>>>>>>> instance: "SVR-DS01"
>>>>>>>>>>> - 172.21.X1.XXX:9182
>>>>>>>>>>> - 172.25.X2.XXX:9182
>>>>>>>>>>> - 172.25.X3.XXX:9182
>>>>>>>>>>>
>>>>>>>>>>> as seen in the target.yml I want to try to get a label to each
>>>>>>>>>>> target - to see in the prom dashboard which ip is which targetname
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>> Kind regards Kolja
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "Prometheus Users" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to prometheus-use...@googlegroups.com.
>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com
>>>>>>>>>>>
>>>>>>>>>>> <https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Julius Volz
>>>>>>>>>> PromLabs - promlabs.com
>>>>>>>>>>
>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7d53a32c-7ee8-4390-9251-f9e15141bcb6n%40googlegroups.com.
