Could you please to review if the step 1 and step 2 are right?
On Wednesday, September 15, 2021 at 6:11:12 PM UTC+8 nina guo wrote:
> Sorry Brian disturb you again.
> We still faced problems regarding this issue.
>
> With file discovery we defined as below:
> Target:[]
> Labels: dnshost
>
> Then the dnshost can be shown on Prometheus UI like below:
> [image: Screenshot 2021-09-15 180030.png]
>
> *1. Even the labels can be shown on Prometheus UI, we still need to write
> a custom metric:*
> #!/bin/bash
> #
> # Description: Expose metrics from virtualip.
> #
> #
> OUTPUT=$(cat "/etc/sysconfig/network/virtualip")
> ET=$（cut -d":" -f 1 OUTPUT）
> SHORT_NAME=$(cut -d":" -f 2 OUTPUT)
>
> echo '# HELP show virtualip.'
> echo '# TYPE virtualip'
>
> echo "virtualip{interface=\"$ET\",short_name=\"$SHORT_NAME\"} 1">
> /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
> mv /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
> /var/lib/node_exporter/textfile_collector/virtualip.prom
>
> *2. with the query (mymetric) * on(instance)
> group_left(hostname,domainname) node_uname_info to get the value on
> Grafana. *
> On Monday, September 13, 2021 at 3:26:37 PM UTC+8 nina guo wrote:
>
>> Got it. Thank you very much Brian.
>>
>> On Monday, September 13, 2021 at 3:03:24 PM UTC+8 Brian Candler wrote:
>>
>>> Yes, I showed that before. You write your PromQL query like this:
>>>
>>> (mymetric) * *on(instance)
>>> group_left(hostname,domainname) node_uname_info*
>>>
>>> where "mymetric" is whatever query you would have written before.
>>>
>>> See also:
>>> https://www.robustperception.io/how-to-have-labels-for-machine-roles
>>>
>>> https://www.robustperception.io/exposing-the-software-version-to-prometheus
>>>
>>> This works as long as mymetric and node_uname_info have *exactly* the
>>> same value for the "instance" label. In particular, you need to make sure
>>> that the exporter port number (:9100) *doesn't* appear in the instance
>>> label of node_uname info.
>>> If that's not the case, then you need to fix it:
>>>
>>> https://www.robustperception.io/controlling-the-instance-label
>>> https://groups.google.com/forum/#!topic/prometheus-users/I4dVtwI5YrM
>>>
>>> https://groups.google.com/forum/#!msg/prometheus-users/PPwF-2CQiSc/FsSvfCm3AwAJ
>>>
>>> On Monday, 13 September 2021 at 06:47:14 UTC+1 [email protected] wrote:
>>>
>>>> Hi Brian,
>>>>
>>>> We are trying another approach regarding the case.
>>>>
>>>> Our targets are discoverd by file discvoery. If we define some of the
>>>> labels by ourselves, whether the value of the label can be obtained by
>>>> node_uname_info?
>>>>
>>>> [image: Screenshot 2021-09-13 134538.png]
>>>>
>>>> eg.:
>>>> If adding a new label label_A="label_a_value", then whether the value
>>>> of the label can be got with node_uname_info?
>>>>
>>>> On Friday, September 10, 2021 at 4:03:37 PM UTC+8 nina guo wrote:
>>>>
>>>>> Thank you very much Brian
>>>>> On Friday, September 10, 2021 at 3:49:07 PM UTC+8 Brian Candler wrote:
>>>>>
>>>>>> The script will go wherever node_exporter runs. Normally this would
>>>>>> be on the host, but some people run node_exporter inside a pod; if it's
>>>>>> in
>>>>>> a pod, then you'll need to bind-mount /etc/sysconfig/network/virtualip
>>>>>> so
>>>>>> that the pod can read it.
>>>>>>
>>>>>> There are some bugs in that script. In particular you'll need
>>>>>> double-quotes around the echo which does expansion, and you'll need to
>>>>>> put
>>>>>> double-quotes around the label values too.
>>>>>> echo "virtualip{interface=\"$ET\",short_name=\"$SHORT_NAME\"} 1" >
>>>>>> ...etc
>>>>>>
>>>>>> On Friday, 10 September 2021 at 07:23:59 UTC+1 [email protected]
>>>>>> wrote:
>>>>>>
>>>>>>> #!/bin/bash
>>>>>>> #
>>>>>>> # Description: Expose metrics from virtualip.
>>>>>>> #
>>>>>>> #
>>>>>>> OUTPUT=$(cat "/etc/sysconfig/network/virtualip")
>>>>>>> ET=$（cut -d":" -f 1 OUTPUT）
>>>>>>> SHORT_NAME=$(cut -d":" -f 2 OUTPUT)
>>>>>>>
>>>>>>> echo '# HELP show virtualip.'
>>>>>>> echo '# TYPE virtualip'
>>>>>>>
>>>>>>> echo 'virtualip{interface=ET,short_name=SHORT_NAME} 1' >
>>>>>>> /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
>>>>>>> mv /var/lib/node_exporter/textfile_collector/virtualip.prom.$$
>>>>>>> /var/lib/node_exporter/textfile_collector/virtualip.prom
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------------------------------------------------------------------------
>>>>>>> I'm writing the script as above.
>>>>>>> Our Prometheus solution is deployed in k8s cluster. I have add
>>>>>>> ----collector.textfile.directory to node exporter deployment file. Do
>>>>>>> we
>>>>>>> also need to copy this script to node exporter POD?
>>>>>>> On Thursday, September 9, 2021 at 8:17:54 PM UTC+8 Brian Candler
>>>>>>> wrote:
>>>>>>>
>>>>>>>> So you could simply return
>>>>>>>>
>>>>>>>> virtualip{interface="eth0",function_name="foo"} 1
>>>>>>>>
>>>>>>>> and then prometheus itself would add "instance" and "job" labels
>>>>>>>> when scraping.
>>>>>>>>
>>>>>>>> I have searched for /etc/sysconfig/network/virtualip and I see no
>>>>>>>> reference to it on the Internet, so I am guessing this is a custom
>>>>>>>> file
>>>>>>>> you've created, and your own script is interpreting it.
>>>>>>>>
>>>>>>>> On Thursday, 9 September 2021 at 10:43:30 UTC+1 [email protected]
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thank you. The output is as below.
>>>>>>>>> (function_name) username@hostname:~> cat
>>>>>>>>> /etc/sysconfig/network/virtualip
>>>>>>>>> eth0:function_name
>>>>>>>>> On Thursday, September 9, 2021 at 5:06:27 PM UTC+8 Brian Candler
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> > We want to get this output "cat
>>>>>>>>>> /etc/sysconfig/network/virtualip". So which collectors may contain
>>>>>>>>>> this
>>>>>>>>>> value?
>>>>>>>>>>
>>>>>>>>>> I am unaware of any. That looks like a RedHat-specific config
>>>>>>>>>> file.
>>>>>>>>>>
>>>>>>>>>> Remember that prometheus collects *metrics* which are floating
>>>>>>>>>> point numbers only. Anything which is not in that format has to
>>>>>>>>>> appear in
>>>>>>>>>> the label of a metric, and the unique set of labels defines a new
>>>>>>>>>> timeseries. If you don't need any floating-point value for a
>>>>>>>>>> metric, then
>>>>>>>>>> the convention is to return a static value of "1" which makes it
>>>>>>>>>> easy to
>>>>>>>>>> combine with other metrics, using the multiplication operator.
>>>>>>>>>> node_uname_info is a very good example of this, as are others like
>>>>>>>>>> node_exporter_build_info and node_network_info.
>>>>>>>>>>
>>>>>>>>>> If you show us what the content of this file looks like, maybe we
>>>>>>>>>> can suggest what a suitable metric would look like.
>>>>>>>>>>
>>>>>>>>>> > I checked again that the info we require can be shown with
>>>>>>>>>> ifconfig command. We want to get "eth0:AAA" . Is there any metric
>>>>>>>>>> include
>>>>>>>>>> this value?
>>>>>>>>>>
>>>>>>>>>> You can see all of the metrics returned by node_exporter using
>>>>>>>>>> this command:
>>>>>>>>>>
>>>>>>>>>> curl localhost:9100/metrics
>>>>>>>>>>
>>>>>>>>>> I can see metrics giving layer 2 info (MAC addresses):
>>>>>>>>>> node_network_info{address="0a:4e:86:6c:ab:ed",broadcast="ff:ff:ff:ff:ff:ff",device="veth6698c6af",duplex="full",ifalias="",operstate="up"}
>>>>>>>>>>
>>>>>>>>>> 1
>>>>>>>>>>
>>>>>>>>>> - there is more info here
>>>>>>>>>> <https://www.robustperception.io/network-interface-metrics-from-the-node-exporter>
>>>>>>>>>>
>>>>>>>>>> - but none giving layer 3 info (IP addresses). Either this is
>>>>>>>>>> available in
>>>>>>>>>> an optional node_exporter collector which is disabled by default - I
>>>>>>>>>> pointed you to the documentation on those before - or you'd need to
>>>>>>>>>> create
>>>>>>>>>> a new metric yourself (e.g. with textfile collector), or find
>>>>>>>>>> another
>>>>>>>>>> exporter that does what you want.
>>>>>>>>>>
>>>>>>>>>> > So we only want to get a specific static value to be shown on
>>>>>>>>>> Grafana, it may not be a generanl metric.
>>>>>>>>>>
>>>>>>>>>> Everything in prometheus is a metric. Static values have to be
>>>>>>>>>> labels on metrics. Again, see how node_uname_info does this.
>>>>>>>>>>
>>>>>>>>>> Of course, since Grafana is a separate piece of software, it
>>>>>>>>>> *might* be possible in Grafana to extract information from some
>>>>>>>>>> other source and combine it with other info in your dashboard. I
>>>>>>>>>> don't
>>>>>>>>>> know how you'd do that, and you'd have to ask elsewhere, because
>>>>>>>>>> this is a
>>>>>>>>>> mailing list for prometheus, not grafana. (Grafana has its own
>>>>>>>>>> discussion
>>>>>>>>>> forum).
>>>>>>>>>>
>>>>>>>>>> On Thursday, 9 September 2021 at 05:30:00 UTC+1
>>>>>>>>>> [email protected] wrote:
>>>>>>>>>>
>>>>>>>>>>> So we only want to get a specific static value to be shown on
>>>>>>>>>>> Grafana, it may not be a generanl metric.
>>>>>>>>>>>
>>>>>>>>>>> On Thursday, September 9, 2021 at 12:02:01 PM UTC+8 nina guo
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I checked again that the info we require can be shown with
>>>>>>>>>>>> ifconfig command. We want to get "eth0:AAA" . Is there any metric
>>>>>>>>>>>> include
>>>>>>>>>>>> this value?
>>>>>>>>>>>>
>>>>>>>>>>>> username@hostname:~> ifconfig
>>>>>>>>>>>> eth0 Link encap:Ethernet HWaddr FA:
>>>>>>>>>>>> xxxxxxx
>>>>>>>>>>>> ..........................
>>>>>>>>>>>>
>>>>>>>>>>>> *eth0:AAA* Link encap:Ethernet HWaddr FA:
>>>>>>>>>>>> inet addr: Bcast: Mask:
>>>>>>>>>>>> UP BROADCAST RUNNING MULTICAST Metric:
>>>>>>>>>>>>
>>>>>>>>>>>> On Thursday, September 9, 2021 at 10:53:46 AM UTC+8 nina guo
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you very much for your detailed reply.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We want to get this output "cat
>>>>>>>>>>>>> /etc/sysconfig/network/virtualip". So which collectors may
>>>>>>>>>>>>> contain this
>>>>>>>>>>>>> value?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wednesday, September 8, 2021 at 6:26:45 PM UTC+8 Brian
>>>>>>>>>>>>> Candler wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wednesday, 8 September 2021 at 10:00:42 UTC+1
>>>>>>>>>>>>>> [email protected] wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> May I know if there is an example for how to write this
>>>>>>>>>>>>>>> cronjob?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There are lots of examples in this repo:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/prometheus-community/node-exporter-textfile-collector-scripts
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Basically you just write out metrics in the prometheus
>>>>>>>>>>>>>> text-based exposition format
>>>>>>>>>>>>>> <https://prometheus.io/docs/instrumenting/exposition_formats/>
>>>>>>>>>>>>>> to a text file, and enable the node_exporter textfile
>>>>>>>>>>>>>> collector
>>>>>>>>>>>>>> <https://github.com/prometheus/node_exporter#textfile-collector>
>>>>>>>>>>>>>> to read it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > With node exporter, whether we can obtain the info of
>>>>>>>>>>>>>> virtual IP?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't know, because you haven't said what sort of virtual
>>>>>>>>>>>>>> IPs you're using, except for some CLI tool "virtualip list"
>>>>>>>>>>>>>> which I've
>>>>>>>>>>>>>> never seen before.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can look at the list of node_exporter collectors which
>>>>>>>>>>>>>> are enabled by default
>>>>>>>>>>>>>> <https://github.com/prometheus/node_exporter#enabled-by-default>
>>>>>>>>>>>>>> and disabled by default
>>>>>>>>>>>>>> <https://github.com/prometheus/node_exporter#disabled-by-default>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - maybe there's one which meets your needs. For example, if
>>>>>>>>>>>>>> you're using
>>>>>>>>>>>>>> IPVS then I see some metrics for that. If you're using
>>>>>>>>>>>>>> keepalived, then it
>>>>>>>>>>>>>> exports its status via SNMP. (You would run snmpd on that host,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> snmp_exporter on your prometheus server).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > One more question is with node_uname_info, we get the
>>>>>>>>>>>>>> nodename, this nodename is the hostname of the server by default?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I believe it's whatever "uname -n" or "hostname" shows.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7f4cca86-1ce0-4218-b4a0-96264d1d5b3fn%40googlegroups.com.
