Antonin, hi. I'm getting a problem in my prometheus operator about the podmonitor discovery....
When I start a new integration, it's not discovered by the prometheus operator! I need to restart the pod operator to my integration to be discovered and scraped This is my Prometheus object in k8s: apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{"project.cattle.io/namespaces":"[\"cattle-prometheus\",\"cattle-system\",\"kube-node-lease\",\"kube-public\",\"kube-sy$ project.cattle.io/namespaces: '["cattle-prometheus","cattle-system","kube-node-lease","kube-public","kube-system","security-scan"]' generation: 23 labels: app: prometheus chart: prometheus-0.0.1 heritage: Tiller io.cattle.field/appId: cluster-monitoring release: cluster-monitoring name: cluster-monitoring namespace: cattle-prometheus resourceVersion: "68969602" spec: additionalAlertManagerConfigs: key: additional-alertmanager-configs.yaml name: prometheus-cluster-monitoring-additional-alertmanager-configs additionalScrapeConfigs: key: additional-scrape-configs.yaml name: prometheus-cluster-monitoring-additional-scrape-configs affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchLabels: app: prometheus prometheus: cluster-monitoring topologyKey: kubernetes.io/hostname weight: 100 arbitraryFSAccessThroughSMs: {} baseImage: rancher/prom-prometheus configMaps: - prometheus-cluster-monitoring-nginx containers: - command: - /bin/sh - -c - cp /nginx/run-sh.tmpl /var/cache/nginx/nginx-start.sh; chmod +x /var/cache/nginx/nginx-start.sh; /var/cache/nginx/nginx-start.sh env: - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP image: rancher/nginx:1.17.4-alpine name: prometheus-proxy ports: - containerPort: 8080 name: nginx-http protocol: TCP resources: limits: cpu: 100m memory: 100Mi requests: cpu: 50m memory: 50Mi securityContext: runAsGroup: 101 runAsUser: 101 volumeMounts: - mountPath: /nginx name: configmap-prometheus-cluster-monitoring-nginx - mountPath: /var/cache/nginx name: nginx-home - args: - --proxy-url=http://127.0.0.1:9090 - --listen-address=$(POD_IP):9090 - --filter-reader-labels=prometheus - --filter-reader-labels=prometheus_replica command: - prometheus-auth env: - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP image: rancher/prometheus-auth:v0.2.0 livenessProbe: failureThreshold: 6 httpGet: path: /-/healthy port: web scheme: HTTP initialDelaySeconds: 300 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 name: prometheus-agent ports: - containerPort: 9090 name: web protocol: TCP readinessProbe: failureThreshold: 10 httpGet: path: /-/ready port: web scheme: HTTP initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 resources: limits: cpu: 500m memory: 200Mi requests: cpu: 100m memory: 100Mi evaluationInterval: 60s externalLabels: prometheus_from: arquitetura listenLocal: true logFormat: logfmt logLevel: info nodeSelector: kubernetes.io/os: linux podMetadata: labels: labels: app: prometheus chart: prometheus-0.0.1 release: cluster-monitoring podMonitorNamespaceSelector: {} podMonitorSelector: matchLabels: app: camel-k replicas: 1 resources: limits: cpu: "1" memory: 1000Mi requests: cpu: 750m memory: 750Mi retention: 12h ruleNamespaceSelector: matchExpressions: - key: field.cattle.io/projectId operator: In values: - p-gzj4v - key: field.cattle.io/projectId operator: In values: - p-gzj4v ruleSelector: matchExpressions: - key: source operator: In values: - rancher-alert - rancher-monitoring rules: alert: {} scrapeInterval: 60s secrets: - exporter-etcd-cert securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: cluster-monitoring serviceMonitorNamespaceSelector: matchExpressions: - key: field.cattle.io/projectId operator: In values: - p-gzj4v - key: field.cattle.io/projectId operator: In values: - p-gzj4v serviceMonitorSelector: {} tolerations: - effect: NoSchedule key: cattle.io/os operator: Equal value: linux version: v2.17.2 volumes: - emptyDir: {} name: nginx-home On Fri, Nov 26, 2021 at 11:41 AM Roberto Camelk <betonetotbo.cam...@gmail.com> wrote: > > Thanks again Antonin. > > You save my black-friday! > > On Fri, Nov 26, 2021 at 11:02 AM Antonin Stefanutti > <anto...@stefanutti.fr.invalid> wrote: > > > > Great! > > > > For the metrics, there should be the following registered by the > > MicroProfile Metrics Camel extension: > > > > https://camel.apache.org/camel-quarkus/2.4.x/reference/extensions/microprofile-metrics.html#_usage > > > > However, the final name is determined by the MicroProfile Metrics > > specification, so for example the following metric: > > > > camel.context.exchanges.completed.total > > > > Have its name translated to: > > > > application_camel_context_exchanges_completed_total > > > > To check also if the metrics endpoint do have the metrics registered, you > > can run: > > > > $ kubectl exec deployment/<integration_name> -- curl -s > > http://localhost:8080/q/metrics | grep > > application_camel_route_exchanges_completed_total > > > > # HELP application_camel_route_exchanges_completed_total The total number > > of completed exchanges for a route or Camel Context > > # TYPE application_camel_route_exchanges_completed_total counter > > application_camel_route_exchanges_completed_total{camelContext="camel-1",routeId="route1"} > > 33.0 > > > > > On 26 Nov 2021, at 14:34, Roberto Camelk <betonetotbo.cam...@gmail.com> > > > wrote: > > > > > > Thanks a lot again Antonin. I owe you a beer or coffee!!! > > > > > > This worked 100%. > > > > > > To end, another question.... after this, prometheus start scraping the > > > metrics from my running route pod, but... > > > > > > Some metrics IDs are not there, some metrics starting with > > > "org_apache_camel_" as I have see in this grafana dashboard sample: > > > https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145 > > > > > > The other metrics (camel_k_* - > > > https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics) > > > are there as so the jvm metrics > > > (https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics). > > > > > > Is this correct? Why am I not getting that "org_apache_camel_" metrics? > > > > > > On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti > > > <anto...@stefanutti.fr.invalid> wrote: > > >> > > >> It's likely that you need to configure the Prometheus instance, by > > >> editing the Prometheus resource, e.g.: > > >> > > >> apiVersion: monitoring.coreos.com/v1 > > >> kind: Prometheus > > >> metadata: > > >> name: prometheus > > >> spec: > > >> serviceAccountName: prometheus > > >> podMonitorSelector: > > >> matchLabels: > > >> app: camel-k > > >> podMonitorNamespaceSelector: {} > > >> > > >> It's important to have podMonitorNamespaceSelector: {} to discover all > > >> the namespaces, otherwise it's only the resource's namespace. > > >> > > >> Also you can use the Prometheus trait to set the labels on the > > >> Integration PodMonitor accordingly, e.g.: > > >> > > >> $ kamel run -t prometheus.enabled=true -t > > >> prometheus.pod-monitor-labels="app=camel-k" > > >> > > >> There are some examples in the Prometheus operator documentation: > > >> > > >> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started > > >> > > >> And the troubleshooting guide at: > > >> > > >> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md > > >> > > >> > > >>> On 26 Nov 2021, at 12:32, Roberto Camelk <betonetotbo.cam...@gmail.com> > > >>> wrote: > > >>> > > >>> Antonin. Thanks ! > > >>> > > >>> But I continue stuck, please let me share some extra info about my > > >>> prometheus operator log: > > >>> > > >>> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840 > > >>> component=prometheusoperator msg="filtering namespaces to select > > >>> PodMonitors from" namespaces=cattle-prometheus > > >>> namespace=cattle-prometheus prometheus=cluster-monitoring > > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z > > >>> caller=operator.go:1853 component=prometheusoperator msg="selected > > >>> PodMonitors" podmonitors= namespace=cattle-prometheus > > >>> prometheus=cluster-monitoring > > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z > > >>> caller=operator.go:1677 component=prometheusoperator msg="updating > > >>> Prometheus configuration secret skipped, no configuration change" > > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z > > >>> caller=operator.go:1776 component=prometheusoperator msg="filtering > > >>> namespaces to select ServiceMonitors from" > > >>> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system > > >>> namespace=cattle-prometheus prometheus=cluster-monitoring > > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z > > >>> caller=operator.go:1810 component=prometheusoperator msg="selected > > >>> ServiceMonitors" > > >>> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring > > >>> namespace=cattle-prometheus prometheus=cluster-monitoring > > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z > > >>> caller=operator.go:1741 component=prometheusoperator msg="updated > > >>> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets > > >>> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z > > >>> caller=operator.go:1169 component=prometheusoperator msg="new > > >>> statefulset generation inputs match current, skipping any actions" > > >>> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z > > >>> caller=operator.go:734 component=prometheusoperator msg="PodMonitor > > >>> added" > > >>> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z > > >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor > > >>> updated" > > >>> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z > > >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor > > >>> updated" > > >>> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z > > >>> caller=operator.go:759 component=prometheusoperator msg="PodMonitor > > >>> delete" > > >>> > > >>> This last 4 lines is about my camel-k route, that I ran and stopped. > > >>> > > >>> So, there are some problems, I think, about the logs above about the > > >>> podmonitor selectors telling: "selected PodMonitors" podmonitors= > > >>> namespace=cattle-prometheus prometheus=cluster-monitoring > > >>> > > >>> My camel-k route is running at namespace "platform" and has no label > > >>> like "prometheus=cluster-monitoring". > > >>> > > >>> Do you know how can I fix this? Adding additional scrape configs to > > >>> prometheus can solve this? Can you provide a snipet code? > > >>> > > >>> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti > > >>> <anto...@stefanutti.fr.invalid> wrote: > > >>>> > > >>>> When run an Integration with `kamel run -t prometheus.enabled=true`, a > > >>>> PodMonitor resource is created for the Prometheus operator to > > >>>> reconcile and configure Prometheus to scrape the Integration metrics > > >>>> endpoint. > > >>>> > > >>>> The PodMonitor metadata must match that of the Prometheus operator, > > >>>> like the namespace, the labels, ... > > >>>> > > >>>> Some documentation is available at: > > >>>> > > >>>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery > > >>>> > > >>>> That contains some links to the Prometheus operator documentation for > > >>>> troubleshooting why the metrics endpoint is not discovered. > > >>>> > > >>>>> On 25 Nov 2021, at 12:25, Roberto Camelk > > >>>>> <betonetotbo.cam...@gmail.com> wrote: > > >>>>> > > >>>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster > > >>>>> monitoring enabled in rancher, so that exists a Prometheus instance > > >>>>> running, so as a Prometheus Operator. > > >>>>> > > >>>>> Recently I deployed a Apache Camel-K operator, and now I want to > > >>>>> enable the prometheus integration for collect metrics about my camel > > >>>>> routes. > > >>>>> > > >>>>> So, my Camel-K operator is running in namescape camel-k and the > > >>>>> rancher embedded prometheus stack in cattle-prometheus namespace. > > >>>>> > > >>>>> I just have launched my route with the trait --trait > > >>>>> prometheus.enabled=true, but the camel metrics aren't listing at my > > >>>>> prometheus. > > >>>>> > > >>>>> Anyone knows why or what I need to configure to my camel-k route > > >>>>> deploy it's metrics at the rancher embedded prometheus? > > >>>> > > >> > >