Great! For the metrics, there should be the following registered by the MicroProfile Metrics Camel extension:
https://camel.apache.org/camel-quarkus/2.4.x/reference/extensions/microprofile-metrics.html#_usage However, the final name is determined by the MicroProfile Metrics specification, so for example the following metric: camel.context.exchanges.completed.total Have its name translated to: application_camel_context_exchanges_completed_total To check also if the metrics endpoint do have the metrics registered, you can run: $ kubectl exec deployment/<integration_name> -- curl -s http://localhost:8080/q/metrics | grep application_camel_route_exchanges_completed_total # HELP application_camel_route_exchanges_completed_total The total number of completed exchanges for a route or Camel Context # TYPE application_camel_route_exchanges_completed_total counter application_camel_route_exchanges_completed_total{camelContext="camel-1",routeId="route1"} 33.0 > On 26 Nov 2021, at 14:34, Roberto Camelk <betonetotbo.cam...@gmail.com> wrote: > > Thanks a lot again Antonin. I owe you a beer or coffee!!! > > This worked 100%. > > To end, another question.... after this, prometheus start scraping the > metrics from my running route pod, but... > > Some metrics IDs are not there, some metrics starting with > "org_apache_camel_" as I have see in this grafana dashboard sample: > https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145 > > The other metrics (camel_k_* - > https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics) > are there as so the jvm metrics > (https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics). > > Is this correct? Why am I not getting that "org_apache_camel_" metrics? > > On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti > <anto...@stefanutti.fr.invalid> wrote: >> >> It's likely that you need to configure the Prometheus instance, by editing >> the Prometheus resource, e.g.: >> >> apiVersion: monitoring.coreos.com/v1 >> kind: Prometheus >> metadata: >> name: prometheus >> spec: >> serviceAccountName: prometheus >> podMonitorSelector: >> matchLabels: >> app: camel-k >> podMonitorNamespaceSelector: {} >> >> It's important to have podMonitorNamespaceSelector: {} to discover all the >> namespaces, otherwise it's only the resource's namespace. >> >> Also you can use the Prometheus trait to set the labels on the Integration >> PodMonitor accordingly, e.g.: >> >> $ kamel run -t prometheus.enabled=true -t >> prometheus.pod-monitor-labels="app=camel-k" >> >> There are some examples in the Prometheus operator documentation: >> >> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started >> >> And the troubleshooting guide at: >> >> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md >> >> >>> On 26 Nov 2021, at 12:32, Roberto Camelk <betonetotbo.cam...@gmail.com> >>> wrote: >>> >>> Antonin. Thanks ! >>> >>> But I continue stuck, please let me share some extra info about my >>> prometheus operator log: >>> >>> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840 >>> component=prometheusoperator msg="filtering namespaces to select >>> PodMonitors from" namespaces=cattle-prometheus >>> namespace=cattle-prometheus prometheus=cluster-monitoring >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z >>> caller=operator.go:1853 component=prometheusoperator msg="selected >>> PodMonitors" podmonitors= namespace=cattle-prometheus >>> prometheus=cluster-monitoring >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z >>> caller=operator.go:1677 component=prometheusoperator msg="updating >>> Prometheus configuration secret skipped, no configuration change" >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z >>> caller=operator.go:1776 component=prometheusoperator msg="filtering >>> namespaces to select ServiceMonitors from" >>> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system >>> namespace=cattle-prometheus prometheus=cluster-monitoring >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z >>> caller=operator.go:1810 component=prometheusoperator msg="selected >>> ServiceMonitors" >>> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring >>> namespace=cattle-prometheus prometheus=cluster-monitoring >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z >>> caller=operator.go:1741 component=prometheusoperator msg="updated >>> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets >>> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z >>> caller=operator.go:1169 component=prometheusoperator msg="new >>> statefulset generation inputs match current, skipping any actions" >>> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z >>> caller=operator.go:734 component=prometheusoperator msg="PodMonitor >>> added" >>> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor >>> updated" >>> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor >>> updated" >>> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z >>> caller=operator.go:759 component=prometheusoperator msg="PodMonitor >>> delete" >>> >>> This last 4 lines is about my camel-k route, that I ran and stopped. >>> >>> So, there are some problems, I think, about the logs above about the >>> podmonitor selectors telling: "selected PodMonitors" podmonitors= >>> namespace=cattle-prometheus prometheus=cluster-monitoring >>> >>> My camel-k route is running at namespace "platform" and has no label >>> like "prometheus=cluster-monitoring". >>> >>> Do you know how can I fix this? Adding additional scrape configs to >>> prometheus can solve this? Can you provide a snipet code? >>> >>> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti >>> <anto...@stefanutti.fr.invalid> wrote: >>>> >>>> When run an Integration with `kamel run -t prometheus.enabled=true`, a >>>> PodMonitor resource is created for the Prometheus operator to reconcile >>>> and configure Prometheus to scrape the Integration metrics endpoint. >>>> >>>> The PodMonitor metadata must match that of the Prometheus operator, like >>>> the namespace, the labels, ... >>>> >>>> Some documentation is available at: >>>> >>>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery >>>> >>>> That contains some links to the Prometheus operator documentation for >>>> troubleshooting why the metrics endpoint is not discovered. >>>> >>>>> On 25 Nov 2021, at 12:25, Roberto Camelk <betonetotbo.cam...@gmail.com> >>>>> wrote: >>>>> >>>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster >>>>> monitoring enabled in rancher, so that exists a Prometheus instance >>>>> running, so as a Prometheus Operator. >>>>> >>>>> Recently I deployed a Apache Camel-K operator, and now I want to >>>>> enable the prometheus integration for collect metrics about my camel >>>>> routes. >>>>> >>>>> So, my Camel-K operator is running in namescape camel-k and the >>>>> rancher embedded prometheus stack in cattle-prometheus namespace. >>>>> >>>>> I just have launched my route with the trait --trait >>>>> prometheus.enabled=true, but the camel metrics aren't listing at my >>>>> prometheus. >>>>> >>>>> Anyone knows why or what I need to configure to my camel-k route >>>>> deploy it's metrics at the rancher embedded prometheus? >>>> >>
