Hi.
I'm using Apache Camel-K to run integration logics.
Camel-K is runing inside a Rancher 2.4.3 in k8s, in the namespace
"platform".
This Rancher version has a embedded prometheus stack on it enabled.
So, there is a Prometheus Operator 0.38.1 running...
This prometheus stack is at the namespace "cattle-prometheus".
So the first thing I done was configure the Prometheus resource for the
operator, adding to it this:
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchExpressions:
- key: camel.apache.org/integration
operator: Exists
This tells to the operator that I want scan all namespaces for PodMontiors
with the label key: camel.apache.org/integration.
So, when I start a Camel-K integration, a new pod is created, as so a
PodMonitor resource to describe to the Prometheus Operator where Prometheus
Server can scrape metrics of my integration pod.
Is expected, in a while, that the Prometheus Operator discover this
PodMonitor and reconfigure the Prometheus Server to scrape it.
Here is where I'm getting in trouble...
The prometheus operator is discovering that a new PodMontior and logging
this:
level=debug ts=2021-11-30T16:42:55.490845245Z caller=operator.go:734
component=prometheusoperator msg="PodMonitor added"
level=debug ts=2021-11-30T16:42:57.017240979Z caller=operator.go:748
component=prometheusoperator msg="PodMonitor updated"
level=debug ts=2021-11-30T16:43:13.978452732Z caller=operator.go:748
component=prometheusoperator msg="PodMonitor updated"
level=debug ts=2021-11-30T16:43:14.207084511Z caller=operator.go:748
component=prometheusoperator msg="PodMonitor updated"
But nothing occurs on the Prometheus Server. It does not start to scrape my
pod metrics.
BUT, if I restart the Prometheus Operator, it start to scrape it (if my pod
is already running). So, after restart the operator at got this log:
level=debug ts=2021-11-30T13:11:50.399104614Z caller=operator.go:1840
component=prometheusoperator msg="filtering namespaces to select
PodMonitors from"
namespaces=kube-node-lease,weave,cattle-prometheus,infrastructure,kube-system,security-scan,cattle-system,kube-public,platform,poc,default,aplicativos
namespace=cattle-prometheus prometheus=cluster-monitoring
level=debug ts=2021-11-30T13:11:50.399141448Z caller=operator.go:1853
component=prometheusoperator msg="selected PodMonitors"
podmonitors=platform/micro-profile-metrics2,platform/create-role-route
namespace=cattle-prometheus prometheus=cluster-monitoring
I'm also reporting this at the prometheus operator project:
https://github.com/prometheus-operator/prometheus-operator/issues/4428
But I'm stuck, I need to solve this... Can someone help me?
Anyone knows why the operator isn't firing the regemeration of the
prometheus config to scrape my new pods?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/30c9d9bd-3bb1-4ccc-a70a-8b0a26927366n%40googlegroups.com.
