Thank you for the response Brian,
with both config a and config b, i was unable to see the source label.
suggested config a)
metric_relabel_configs:
- source_labels: [ source, __address__ ]
regex: ";(.*)(:.*)?"
target_label: source
replacement: '${1}'
output:
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="run"} 3213
*query: up*
response:
up{instance="localhost:9104", job="nodeexporter"} 1
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", type="Processor"} 0
source= metrics was not available.
now to debug, i changed the query slightly , and it seems that the
__address__ is not available with metric_relabel_configs?
modified config a)
- source_labels: [ source, __address__ ]
regex: "(.*)"
target_label: source
replacement: '${1}'
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com;", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com;", status="run"} 3213
*query: up*
up{instance="localhost:9104", job="nodeexporter"}
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", source=";", type="Processor"} 0
so from the last query, it seems that __address__ is blank, and since
source= is also blank, we see source=";" which appears to be concatenation
of 2 blank values with ; as delimiter.
Q1: here i am unable to understand that how ; was appended after source
value in *custom_metrics_from_server* .
config b) :
metric_relabel_configs:
- source_labels: [ source ]
regex: ""
target_label: __tmp_source_empty
replacement: Y
- source_labels: [ __tmp_source_empty, __address__ ]
regex: "Y;(.*)(:.*)?"
target_label: source
replacement: "${1}"
- target_label: __tmp_source_empty
replacement: ""
output:
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="run"} 3213
*query: up*
response:
up{instance="localhost:9104", job="nodeexporter"} 1
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", type="Processor"} 0
source= metrics was not available.
now to debug, i changed the query slightly , and it seems that the
__address__ is not available with metric_relabel_configs with config b?
modified config b)
*query: custom_metrics_from_server*
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source=";", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source=";", status="run"} 3213
*query: up*
up{instance="localhost:9104", job="nodeexporter"}
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", source="Y;", type="Processor"}
so from the last query, it seems that __address__ is blank, and since
source= is also blank, we see source=";" which appears to be concatenation
of 2 blank values with ; as delimiter.
so far , it appears that the __address__ field is not available with
metrics_relabel_config?
Please advice.
On Wednesday, March 12, 2025 at 12:24:11 AM UTC+5:30 Brian Candler wrote:
> It's better to avoid honor_labels unless you really need it. It allows the
> exporter to do bad things like overriding the "job" and "instance" labels.
>
> Better to use metric_relabel_configs to relabel, at which point you have
> the combined set of labels from service discovery and the scrape. A missing
> label is the same as a label whose value is the empty string. So you can do
> (untested):
>
> metric_relabel_configs:
> - source_labels: [ source, __address__ ]
> regex: ";(.*)(:.*)?"
>
> target_label: source
> replacement: "${1}"
>
> This works as long as the source label's value cannot start with a
> semicolon - if it can, define a different "separator" character. If you
> don't like that, another approach is to set a temporary label as a flag:
>
> metric_relabel_configs:
> - source_labels: [ source ]
> regex: ""
> target_label: __tmp_source_empty
> replacement: Y
> - source_labels: [ __tmp_source_empty, __address__ ]
> regex: "Y;(.*)(:.*)?"
> target_label: source
> replacement: "${1}"
> - target_label: __tmp_source_empty
> replacement: ""
>
> On Tuesday, 11 March 2025 at 17:39:58 UTC mohan garden wrote:
>
>> Hi Everyone,
>>
>> I need to handle a scenario where I add a label to ingested metrics, but
>> only if the label does not already exist in the collected data. If the
>> label is present in a metric, it should remain unchanged.
>>
>> For example, given a set of metrics:
>>
>> custom_metrics_from_server{ source="server1.example.com",status="run" }
>> 3213
>> custom_metrics_from_server{ source="server1.example.com",status="pend" }
>> 3215
>>
>> When additional metrics are collected from a node exporter, some will
>> already have a "source=" label, while others will not:
>>
>> custom_metrics_from_server{ source="server1.example.com",status="run" }
>> 3213
>> custom_metrics_from_server{ source="server1.example.com",status="pend" }
>> 3215
>> node_cooling_device_max_state{name="7",type="Processor"} 0
>>
>> To ensure that all metrics contain the "source=" label without overriding
>> existing ones, I figured the following approach using honor_labels and
>> relabel_configs:
>>
>> honor_labels: true
>> relabel_configs:
>> - source_labels: [ _address_ ]
>> regex: "(.*):(.*)"
>> target_label: "source"
>> replacement: '${1}'
>>
>> Is this the best way to handle this, or is there a more optimal approach?
>> Please advice.
>>
>> Thanks!
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/ff6ef6e6-3431-4ee7-93c9-82333de86537n%40googlegroups.com.
