Hello, folks. The story. In my org we have several k8s clusters and quite unreliable security department, who has control over firewall and have a habbit of corrupting the rules on said firewall. The confusion is immense. The issue is that at any point in time *one or several nodes can lose access to one or several external resources*.
So, we had a brilliant idea to mitigate the confusion by deploying a *blackbox daemonset *and *configure Prometheus to query external resources*. But! We have many-to-many relationship here. We want to query multiple resources from all of the worker nodes in cluster. This means, that we need auto discovery (to dynamically get instances of daemon set) and multiple targets. In docs we found an example of multi target requests though static configs. Unfortunately, as the name suggest, it is a static config and we can't get the *metadata of node* that lost connection or *reliably* determine the fact that connection was lost at all (no way to properly configure alert threshold, so it wouldn't misfire). On the other hand - there is Kubernetes service discovery. But the only way we found is to have a separate job for each of the remote resources. Is there, maybe, a better way to approach the problem? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/32246179-4d6f-4fab-92fb-192e2abd154dn%40googlegroups.com.
