> And I don't want to user a network service, because my scrap must be
handled by a blackbox exporter on a specific node.
So how are you going to identify which particular node or pod you want to
talk to? If it's always the same physical node, then just scrape that
node's IP address, and bind the service to a fixed port on that node.
Otherwise, what are you going to do to identify the pod - search for it by
name? By a specific label?
But really, this is exactly what a "service" is for in Kubernetes: to
provide an external access point for a service. This would allow k8s to
move that pod to a different node, and keep it accessible on the same
IP/port. Your blackbox_exporter pod definitely provides a "service" like
this, in my opinion. A k8s Service will identify the pod in the same way
as you'd do manually anyway (e.g. by linking to a pod with a particular
label). All it does then is forward the traffic to that pod.
> Is there a way to mix the 2 discovery services (kubernetes and static ) ?
Not really. In label rewriting rules, a single __address__ cannot be
rewritten to multiple __address__es - it's a 1:1 mapping (or 1:0 if you
drop the target). Therefore, if you use kubernetes SD to select one
specific pod, then that scrape job can only scrape a single blackbox target.
What I think you'd have to do is:
1. Write a separate program to determine the IP/port of the blackbox
exporter. Say it finds 1.2.3.4:9115
2. From this program, write out a YAML file which merges this with a list
of targets you want to probe:
- labels:
__param_target: 10.129.100.213
targets:
- 1.2.3.4:9115
- labels:
__param_target: 10.129.100.214
targets:
- 1.2.3.4:9115
- labels:
__param_target: 10.129.100.215
targets:
- 1.2.3.4:9115
(note: you must also use rewriting rules to copy __param_target to some
other persistent label, e.g. "instance" or "target", to ensure each
timeseries has a unique set of labels)
3. Use file_sd_configs to read this file
But why go to all that trouble, when you could create a single Service
resource definition for your blackbox_exporter pod and be done with it?
On Monday, 12 December 2022 at 10:08:17 UTC SebV wrote:
> Hi Brian,
>
> Yes, I want to probe 10.129.100.213, 10.129.100.214, 10.129.100.215, so
> it is not blackbox address.
> And I don't want to user a network service, because my scrap must be
> handled by a blackbox exporter on a specific node.
> This is why, it try to find a way to set dynamicly the last step :
> - target_label: __address__
> replacement: my blackbox pod ip on a specific node
>
> Is there a way to mix the 2 discovery services (kubernetes and static ) ?
>
> Thank you in advance.
> On Sunday, December 11, 2022 at 7:20:51 PM UTC+1 Brian Candler wrote:
>
>> On Sunday, 11 December 2022 at 17:50:38 UTC SebV wrote:
>>
>>> what am I missing ?
>>>
>>
>> Nothing is setting the labels __meta_kubernetes_pod_ip or
>> __meta_kubernetes_pod_container_port_number. Therefore, the target
>> __address__ you are constructing is simply ":" - which is invalid, as the
>> error says.
>>
>> This in turn is because the service discovery mechanism you've chosen to
>> use is static_configs. This only sets the __address__ label to the values
>> you give, and nothing else.
>>
>> I think you first need to be clear about which address to send the scrape
>> to (which is the final value of __address__ after relabeling, and will be
>> the address:port where blackbox_exporter is listening), and the address you
>> want blackbox_exporter to probe (which is __param_target). What are
>> 10.129.100.213, 10.129.100.214, 10.129.100.215? Are these three instances
>> of blackbox_exporter, or three targets that you want blackbox_exporter to
>> probe?
>>
>> If you want the to use the labels __meta_kubernetes_pod_ip and
>> __meta_kubernetes_pod_container_port_number then you need to use
>> kubernetes_sd_configs
>> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config>
>>
>> instead. You'd need to configure this block to talk to the Kubernetes API,
>> and it will return a list of targets (e.g. pods or services, whatever you
>> ask for) with those meta labels, from which you can construct your scraping
>> targets. This might make sense if what you're trying to do is to scrape or
>> probe each pod. It could also be used to find your blackbox_exporter pod
>> but it would be inefficient, unless you can apply some sort of label to
>> your blackbox_exporter service and tell the API you're only interested in
>> services with that label.
>>
>> It sounds to me, though, that what you want is simply:
>>
>> - target_label: __address__
>> replacement: my_service_ip:9115 <http://127.0.0.1:9115/> # The
>> blackbox exporter's real hostname:port.
>>
>> That is, create blackbox_exporter in kubernetes with an attached service,
>> and give the service either a static port or a static IP address. Then if
>> the pod is recreated on a different IP, the service IP:port is unchanged.
>> (And: if you have multiple blackbox_exporter pods, the load should be
>> distributed between them)
>>
>> https://kubernetes.io/blog/2022/05/23/service-ip-dynamic-and-static-allocation/
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/75007d9f-ec11-4cd3-b349-d4e396c27317n%40googlegroups.com.
