The label "dnshost" is totally different pn Prometheus UI with the label
which we want to be shown on Grafana. Is this understanding correct?
On Wednesday, September 15, 2021 at 6:11:58 PM UTC+8 nina guo wrote:
> Could you please to review if the step 1 and step 2 are right?
>
> On Wednesday, September 15, 2021 at 6:11:12 PM UTC+8 nina guo wrote:
>
>> Sorry Brian disturb you again.
>> We still faced problems regarding this issue.
>>
>> With file discovery we defined as below:
>> Target:[]
>> Labels: dnshost
>>
>> Then the dnshost can be shown on Prometheus UI like below:
>> [image: Screenshot 2021-09-15 180030.png]
>>
>> *1. Even the labels can be shown on Prometheus UI, we still need to write
>> a custom metric:*
>> #!/bin/bash
>> #
>> # Description: Expose metrics from virtualip.
>> #
>> #
>> OUTPUT=$(cat "/etc/sysconfig/network/virtualip")
>> ET=$（cut -d":" -f 1 OUTPUT）
>> SHORT_NAME=$(cut -d":" -f 2 OUTPUT)
>>
>> echo '# HELP show virtualip.'
>> echo '# TYPE virtualip'
>>
>> echo "virtualip{interface=\"$ET\",short_name=\"$SHORT_NAME\"} 1">
>> /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
>> mv /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
>> /var/lib/node_exporter/textfile_collector/virtualip.prom
>>
>> *2. with the query (mymetric) * on(instance)
>> group_left(hostname,domainname) node_uname_info to get the value on
>> Grafana. *
>> On Monday, September 13, 2021 at 3:26:37 PM UTC+8 nina guo wrote:
>>
>>> Got it. Thank you very much Brian.
>>>
>>> On Monday, September 13, 2021 at 3:03:24 PM UTC+8 Brian Candler wrote:
>>>
>>>> Yes, I showed that before. You write your PromQL query like this:
>>>>
>>>> (mymetric) * *on(instance)
>>>> group_left(hostname,domainname) node_uname_info*
>>>>
>>>> where "mymetric" is whatever query you would have written before.
>>>>
>>>> See also:
>>>> https://www.robustperception.io/how-to-have-labels-for-machine-roles
>>>>
>>>> https://www.robustperception.io/exposing-the-software-version-to-prometheus
>>>>
>>>> This works as long as mymetric and node_uname_info have *exactly* the
>>>> same value for the "instance" label. In particular, you need to make sure
>>>> that the exporter port number (:9100) *doesn't* appear in the instance
>>>> label of node_uname info.
>>>> If that's not the case, then you need to fix it:
>>>>
>>>> https://www.robustperception.io/controlling-the-instance-label
>>>> https://groups.google.com/forum/#!topic/prometheus-users/I4dVtwI5YrM
>>>>
>>>> https://groups.google.com/forum/#!msg/prometheus-users/PPwF-2CQiSc/FsSvfCm3AwAJ
>>>>
>>>> On Monday, 13 September 2021 at 06:47:14 UTC+1 [email protected]
>>>> wrote:
>>>>
>>>>> Hi Brian,
>>>>>
>>>>> We are trying another approach regarding the case.
>>>>>
>>>>> Our targets are discoverd by file discvoery. If we define some of the
>>>>> labels by ourselves, whether the value of the label can be obtained by
>>>>> node_uname_info?
>>>>>
>>>>> [image: Screenshot 2021-09-13 134538.png]
>>>>>
>>>>> eg.:
>>>>> If adding a new label label_A="label_a_value", then whether the value
>>>>> of the label can be got with node_uname_info?
>>>>>
>>>>> On Friday, September 10, 2021 at 4:03:37 PM UTC+8 nina guo wrote:
>>>>>
>>>>>> Thank you very much Brian
>>>>>> On Friday, September 10, 2021 at 3:49:07 PM UTC+8 Brian Candler wrote:
>>>>>>
>>>>>>> The script will go wherever node_exporter runs. Normally this would
>>>>>>> be on the host, but some people run node_exporter inside a pod; if it's
>>>>>>> in
>>>>>>> a pod, then you'll need to bind-mount /etc/sysconfig/network/virtualip
>>>>>>> so
>>>>>>> that the pod can read it.
>>>>>>>
>>>>>>> There are some bugs in that script. In particular you'll need
>>>>>>> double-quotes around the echo which does expansion, and you'll need to
>>>>>>> put
>>>>>>> double-quotes around the label values too.
>>>>>>> echo "virtualip{interface=\"$ET\",short_name=\"$SHORT_NAME\"} 1" >
>>>>>>> ...etc
>>>>>>>
>>>>>>> On Friday, 10 September 2021 at 07:23:59 UTC+1 [email protected]
>>>>>>> wrote:
>>>>>>>
>>>>>>>> #!/bin/bash
>>>>>>>> #
>>>>>>>> # Description: Expose metrics from virtualip.
>>>>>>>> #
>>>>>>>> #
>>>>>>>> OUTPUT=$(cat "/etc/sysconfig/network/virtualip")
>>>>>>>> ET=$（cut -d":" -f 1 OUTPUT）
>>>>>>>> SHORT_NAME=$(cut -d":" -f 2 OUTPUT)
>>>>>>>>
>>>>>>>> echo '# HELP show virtualip.'
>>>>>>>> echo '# TYPE virtualip'
>>>>>>>>
>>>>>>>> echo 'virtualip{interface=ET,short_name=SHORT_NAME} 1' >
>>>>>>>> /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
>>>>>>>> mv /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
>>>>>>>> /var/lib/node_exporter/textfile_collector/virtualip.prom
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------------------------------------------------------------------------
>>>>>>>> I'm writing the script as above.
>>>>>>>> Our Prometheus solution is deployed in k8s cluster. I have add
>>>>>>>> ----collector.textfile.directory to node exporter deployment file. Do
>>>>>>>> we
>>>>>>>> also need to copy this script to node exporter POD?
>>>>>>>> On Thursday, September 9, 2021 at 8:17:54 PM UTC+8 Brian Candler
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> So you could simply return
>>>>>>>>>
>>>>>>>>> virtualip{interface="eth0",function_name="foo"} 1
>>>>>>>>>
>>>>>>>>> and then prometheus itself would add "instance" and "job" labels
>>>>>>>>> when scraping.
>>>>>>>>>
>>>>>>>>> I have searched for /etc/sysconfig/network/virtualip and I see no
>>>>>>>>> reference to it on the Internet, so I am guessing this is a custom
>>>>>>>>> file
>>>>>>>>> you've created, and your own script is interpreting it.
>>>>>>>>>
>>>>>>>>> On Thursday, 9 September 2021 at 10:43:30 UTC+1 [email protected]
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thank you. The output is as below.
>>>>>>>>>> (function_name) username@hostname:~> cat
>>>>>>>>>> /etc/sysconfig/network/virtualip
>>>>>>>>>> eth0:function_name
>>>>>>>>>> On Thursday, September 9, 2021 at 5:06:27 PM UTC+8 Brian Candler
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> > We want to get this output "cat
>>>>>>>>>>> /etc/sysconfig/network/virtualip". So which collectors may contain
>>>>>>>>>>> this
>>>>>>>>>>> value?
>>>>>>>>>>>
>>>>>>>>>>> I am unaware of any. That looks like a RedHat-specific config
>>>>>>>>>>> file.
>>>>>>>>>>>
>>>>>>>>>>> Remember that prometheus collects *metrics* which are floating
>>>>>>>>>>> point numbers only. Anything which is not in that format has to
>>>>>>>>>>> appear in
>>>>>>>>>>> the label of a metric, and the unique set of labels defines a new
>>>>>>>>>>> timeseries. If you don't need any floating-point value for a
>>>>>>>>>>> metric, then
>>>>>>>>>>> the convention is to return a static value of "1" which makes it
>>>>>>>>>>> easy to
>>>>>>>>>>> combine with other metrics, using the multiplication operator.
>>>>>>>>>>> node_uname_info is a very good example of this, as are others like
>>>>>>>>>>> node_exporter_build_info and node_network_info.
>>>>>>>>>>>
>>>>>>>>>>> If you show us what the content of this file looks like, maybe
>>>>>>>>>>> we can suggest what a suitable metric would look like.
>>>>>>>>>>>
>>>>>>>>>>> > I checked again that the info we require can be shown with
>>>>>>>>>>> ifconfig command. We want to get "eth0:AAA" . Is there any metric
>>>>>>>>>>> include
>>>>>>>>>>> this value?
>>>>>>>>>>>
>>>>>>>>>>> You can see all of the metrics returned by node_exporter using
>>>>>>>>>>> this command:
>>>>>>>>>>>
>>>>>>>>>>> curl localhost:9100/metrics
>>>>>>>>>>>
>>>>>>>>>>> I can see metrics giving layer 2 info (MAC addresses):
>>>>>>>>>>> node_network_info{address="0a:4e:86:6c:ab:ed",broadcast="ff:ff:ff:ff:ff:ff",device="veth6698c6af",duplex="full",ifalias="",operstate="up"}
>>>>>>>>>>>
>>>>>>>>>>> 1
>>>>>>>>>>>
>>>>>>>>>>> - there is more info here
>>>>>>>>>>> <https://www.robustperception.io/network-interface-metrics-from-the-node-exporter>
>>>>>>>>>>>
>>>>>>>>>>> - but none giving layer 3 info (IP addresses). Either this is
>>>>>>>>>>> available in
>>>>>>>>>>> an optional node_exporter collector which is disabled by default -
>>>>>>>>>>> I
>>>>>>>>>>> pointed you to the documentation on those before - or you'd need to
>>>>>>>>>>> create
>>>>>>>>>>> a new metric yourself (e.g. with textfile collector), or find
>>>>>>>>>>> another
>>>>>>>>>>> exporter that does what you want.
>>>>>>>>>>>
>>>>>>>>>>> > So we only want to get a specific static value to be shown on
>>>>>>>>>>> Grafana, it may not be a generanl metric.
>>>>>>>>>>>
>>>>>>>>>>> Everything in prometheus is a metric. Static values have to be
>>>>>>>>>>> labels on metrics. Again, see how node_uname_info does this.
>>>>>>>>>>>
>>>>>>>>>>> Of course, since Grafana is a separate piece of software, it
>>>>>>>>>>> *might* be possible in Grafana to extract information from some
>>>>>>>>>>> other source and combine it with other info in your dashboard. I
>>>>>>>>>>> don't
>>>>>>>>>>> know how you'd do that, and you'd have to ask elsewhere, because
>>>>>>>>>>> this is a
>>>>>>>>>>> mailing list for prometheus, not grafana. (Grafana has its own
>>>>>>>>>>> discussion
>>>>>>>>>>> forum).
>>>>>>>>>>>
>>>>>>>>>>> On Thursday, 9 September 2021 at 05:30:00 UTC+1
>>>>>>>>>>> [email protected] wrote:
>>>>>>>>>>>
>>>>>>>>>>>> So we only want to get a specific static value to be shown on
>>>>>>>>>>>> Grafana, it may not be a generanl metric.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thursday, September 9, 2021 at 12:02:01 PM UTC+8 nina guo
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I checked again that the info we require can be shown with
>>>>>>>>>>>>> ifconfig command. We want to get "eth0:AAA" . Is there any metric
>>>>>>>>>>>>> include
>>>>>>>>>>>>> this value?
>>>>>>>>>>>>>
>>>>>>>>>>>>> username@hostname:~> ifconfig
>>>>>>>>>>>>> eth0 Link encap:Ethernet HWaddr FA:
>>>>>>>>>>>>> xxxxxxx
>>>>>>>>>>>>> ..........................
>>>>>>>>>>>>>
>>>>>>>>>>>>> *eth0:AAA* Link encap:Ethernet HWaddr FA:
>>>>>>>>>>>>> inet addr: Bcast: Mask:
>>>>>>>>>>>>> UP BROADCAST RUNNING MULTICAST Metric:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thursday, September 9, 2021 at 10:53:46 AM UTC+8 nina guo
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you very much for your detailed reply.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We want to get this output "cat
>>>>>>>>>>>>>> /etc/sysconfig/network/virtualip". So which collectors may
>>>>>>>>>>>>>> contain this
>>>>>>>>>>>>>> value?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wednesday, September 8, 2021 at 6:26:45 PM UTC+8 Brian
>>>>>>>>>>>>>> Candler wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wednesday, 8 September 2021 at 10:00:42 UTC+1
>>>>>>>>>>>>>>> [email protected] wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> May I know if there is an example for how to write this
>>>>>>>>>>>>>>>> cronjob?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There are lots of examples in this repo:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/prometheus-community/node-exporter-textfile-collector-scripts
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Basically you just write out metrics in the prometheus
>>>>>>>>>>>>>>> text-based exposition format
>>>>>>>>>>>>>>> <https://prometheus.io/docs/instrumenting/exposition_formats/>
>>>>>>>>>>>>>>> to a text file, and enable the node_exporter textfile
>>>>>>>>>>>>>>> collector
>>>>>>>>>>>>>>> <https://github.com/prometheus/node_exporter#textfile-collector>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to read it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > With node exporter, whether we can obtain the info of
>>>>>>>>>>>>>>> virtual IP?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't know, because you haven't said what sort of virtual
>>>>>>>>>>>>>>> IPs you're using, except for some CLI tool "virtualip list"
>>>>>>>>>>>>>>> which I've
>>>>>>>>>>>>>>> never seen before.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can look at the list of node_exporter collectors which
>>>>>>>>>>>>>>> are enabled by default
>>>>>>>>>>>>>>> <https://github.com/prometheus/node_exporter#enabled-by-default>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and disabled by default
>>>>>>>>>>>>>>> <https://github.com/prometheus/node_exporter#disabled-by-default>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - maybe there's one which meets your needs. For example, if
>>>>>>>>>>>>>>> you're using
>>>>>>>>>>>>>>> IPVS then I see some metrics for that. If you're using
>>>>>>>>>>>>>>> keepalived, then it
>>>>>>>>>>>>>>> exports its status via SNMP. (You would run snmpd on that
>>>>>>>>>>>>>>> host, and
>>>>>>>>>>>>>>> snmp_exporter on your prometheus server).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > One more question is with node_uname_info, we get the
>>>>>>>>>>>>>>> nodename, this nodename is the hostname of the server by
>>>>>>>>>>>>>>> default?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I believe it's whatever "uname -n" or "hostname" shows.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/e106813e-4ebc-4dfd-bdd7-f28027011c33n%40googlegroups.com.
