Antonin, hi.
I'm getting a problem in my prometheus operator about the podmonitor
discovery....
When I start a new integration, it's not discovered by the prometheus
operator! I need to restart the pod operator to my integration to be
discovered and scraped
This is my Prometheus object in k8s:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{"project.cattle.io/namespaces":"[\"cattle-prometheus\",\"cattle-system\",\"kube-node-lease\",\"kube-public\",\"kube-sy$
project.cattle.io/namespaces:
'["cattle-prometheus","cattle-system","kube-node-lease","kube-public","kube-system","security-scan"]'
generation: 23
labels:
app: prometheus
chart: prometheus-0.0.1
heritage: Tiller
io.cattle.field/appId: cluster-monitoring
release: cluster-monitoring
name: cluster-monitoring
namespace: cattle-prometheus
resourceVersion: "68969602"
spec:
additionalAlertManagerConfigs:
key: additional-alertmanager-configs.yaml
name: prometheus-cluster-monitoring-additional-alertmanager-configs
additionalScrapeConfigs:
key: additional-scrape-configs.yaml
name: prometheus-cluster-monitoring-additional-scrape-configs
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app: prometheus
prometheus: cluster-monitoring
topologyKey: kubernetes.io/hostname
weight: 100
arbitraryFSAccessThroughSMs: {}
baseImage: rancher/prom-prometheus
configMaps:
- prometheus-cluster-monitoring-nginx
containers:
- command:
- /bin/sh
- -c
- cp /nginx/run-sh.tmpl /var/cache/nginx/nginx-start.sh; chmod +x
/var/cache/nginx/nginx-start.sh;
/var/cache/nginx/nginx-start.sh
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: rancher/nginx:1.17.4-alpine
name: prometheus-proxy
ports:
- containerPort: 8080
name: nginx-http
protocol: TCP
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 50m
memory: 50Mi
securityContext:
runAsGroup: 101
runAsUser: 101
volumeMounts:
- mountPath: /nginx
name: configmap-prometheus-cluster-monitoring-nginx
- mountPath: /var/cache/nginx
name: nginx-home
- args:
- --proxy-url=http://127.0.0.1:9090
- --listen-address=$(POD_IP):9090
- --filter-reader-labels=prometheus
- --filter-reader-labels=prometheus_replica
command:
- prometheus-auth
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: rancher/prometheus-auth:v0.2.0
livenessProbe:
failureThreshold: 6
httpGet:
path: /-/healthy
port: web
scheme: HTTP
initialDelaySeconds: 300
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
name: prometheus-agent
ports:
- containerPort: 9090
name: web
protocol: TCP
readinessProbe:
failureThreshold: 10
httpGet:
path: /-/ready
port: web
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
resources:
limits:
cpu: 500m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
evaluationInterval: 60s
externalLabels:
prometheus_from: arquitetura
listenLocal: true
logFormat: logfmt
logLevel: info
nodeSelector:
kubernetes.io/os: linux
podMetadata:
labels:
labels:
app: prometheus
chart: prometheus-0.0.1
release: cluster-monitoring
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchLabels:
app: camel-k
replicas: 1
resources:
limits:
cpu: "1"
memory: 1000Mi
requests:
cpu: 750m
memory: 750Mi
retention: 12h
ruleNamespaceSelector:
matchExpressions:
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
ruleSelector:
matchExpressions:
- key: source
operator: In
values:
- rancher-alert
- rancher-monitoring
rules:
alert: {}
scrapeInterval: 60s
secrets:
- exporter-etcd-cert
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: cluster-monitoring
serviceMonitorNamespaceSelector:
matchExpressions:
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
serviceMonitorSelector: {}
tolerations:
- effect: NoSchedule
key: cattle.io/os
operator: Equal
value: linux
version: v2.17.2
volumes:
- emptyDir: {}
name: nginx-home
On Fri, Nov 26, 2021 at 11:41 AM Roberto Camelk
<betonetotbo.cam...@gmail.com> wrote:
>
> Thanks again Antonin.
>
> You save my black-friday!
>
> On Fri, Nov 26, 2021 at 11:02 AM Antonin Stefanutti
> <anto...@stefanutti.fr.invalid> wrote:
> >
> > Great!
> >
> > For the metrics, there should be the following registered by the
> > MicroProfile Metrics Camel extension:
> >
> > https://camel.apache.org/camel-quarkus/2.4.x/reference/extensions/microprofile-metrics.html#_usage
> >
> > However, the final name is determined by the MicroProfile Metrics
> > specification, so for example the following metric:
> >
> > camel.context.exchanges.completed.total
> >
> > Have its name translated to:
> >
> > application_camel_context_exchanges_completed_total
> >
> > To check also if the metrics endpoint do have the metrics registered, you
> > can run:
> >
> > $ kubectl exec deployment/<integration_name> -- curl -s
> > http://localhost:8080/q/metrics | grep
> > application_camel_route_exchanges_completed_total
> >
> > # HELP application_camel_route_exchanges_completed_total The total number
> > of completed exchanges for a route or Camel Context
> > # TYPE application_camel_route_exchanges_completed_total counter
> > application_camel_route_exchanges_completed_total{camelContext="camel-1",routeId="route1"}
> > 33.0
> >
> > > On 26 Nov 2021, at 14:34, Roberto Camelk <betonetotbo.cam...@gmail.com>
> > > wrote:
> > >
> > > Thanks a lot again Antonin. I owe you a beer or coffee!!!
> > >
> > > This worked 100%.
> > >
> > > To end, another question.... after this, prometheus start scraping the
> > > metrics from my running route pod, but...
> > >
> > > Some metrics IDs are not there, some metrics starting with
> > > "org_apache_camel_" as I have see in this grafana dashboard sample:
> > > https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145
> > >
> > > The other metrics (camel_k_* -
> > > https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics)
> > > are there as so the jvm metrics
> > > (https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics).
> > >
> > > Is this correct? Why am I not getting that "org_apache_camel_" metrics?
> > >
> > > On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti
> > > <anto...@stefanutti.fr.invalid> wrote:
> > >>
> > >> It's likely that you need to configure the Prometheus instance, by
> > >> editing the Prometheus resource, e.g.:
> > >>
> > >> apiVersion: monitoring.coreos.com/v1
> > >> kind: Prometheus
> > >> metadata:
> > >> name: prometheus
> > >> spec:
> > >> serviceAccountName: prometheus
> > >> podMonitorSelector:
> > >> matchLabels:
> > >> app: camel-k
> > >> podMonitorNamespaceSelector: {}
> > >>
> > >> It's important to have podMonitorNamespaceSelector: {} to discover all
> > >> the namespaces, otherwise it's only the resource's namespace.
> > >>
> > >> Also you can use the Prometheus trait to set the labels on the
> > >> Integration PodMonitor accordingly, e.g.:
> > >>
> > >> $ kamel run -t prometheus.enabled=true -t
> > >> prometheus.pod-monitor-labels="app=camel-k"
> > >>
> > >> There are some examples in the Prometheus operator documentation:
> > >>
> > >> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started
> > >>
> > >> And the troubleshooting guide at:
> > >>
> > >> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md
> > >>
> > >>
> > >>> On 26 Nov 2021, at 12:32, Roberto Camelk <betonetotbo.cam...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> Antonin. Thanks !
> > >>>
> > >>> But I continue stuck, please let me share some extra info about my
> > >>> prometheus operator log:
> > >>>
> > >>> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
> > >>> component=prometheusoperator msg="filtering namespaces to select
> > >>> PodMonitors from" namespaces=cattle-prometheus
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
> > >>> caller=operator.go:1853 component=prometheusoperator msg="selected
> > >>> PodMonitors" podmonitors= namespace=cattle-prometheus
> > >>> prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
> > >>> caller=operator.go:1677 component=prometheusoperator msg="updating
> > >>> Prometheus configuration secret skipped, no configuration change"
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
> > >>> caller=operator.go:1776 component=prometheusoperator msg="filtering
> > >>> namespaces to select ServiceMonitors from"
> > >>> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
> > >>> caller=operator.go:1810 component=prometheusoperator msg="selected
> > >>> ServiceMonitors"
> > >>> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
> > >>> caller=operator.go:1741 component=prometheusoperator msg="updated
> > >>> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
> > >>> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
> > >>> caller=operator.go:1169 component=prometheusoperator msg="new
> > >>> statefulset generation inputs match current, skipping any actions"
> > >>> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
> > >>> caller=operator.go:734 component=prometheusoperator msg="PodMonitor
> > >>> added"
> > >>> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
> > >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> > >>> updated"
> > >>> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
> > >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> > >>> updated"
> > >>> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
> > >>> caller=operator.go:759 component=prometheusoperator msg="PodMonitor
> > >>> delete"
> > >>>
> > >>> This last 4 lines is about my camel-k route, that I ran and stopped.
> > >>>
> > >>> So, there are some problems, I think, about the logs above about the
> > >>> podmonitor selectors telling: "selected PodMonitors" podmonitors=
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>>
> > >>> My camel-k route is running at namespace "platform" and has no label
> > >>> like "prometheus=cluster-monitoring".
> > >>>
> > >>> Do you know how can I fix this? Adding additional scrape configs to
> > >>> prometheus can solve this? Can you provide a snipet code?
> > >>>
> > >>> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
> > >>> <anto...@stefanutti.fr.invalid> wrote:
> > >>>>
> > >>>> When run an Integration with `kamel run -t prometheus.enabled=true`, a
> > >>>> PodMonitor resource is created for the Prometheus operator to
> > >>>> reconcile and configure Prometheus to scrape the Integration metrics
> > >>>> endpoint.
> > >>>>
> > >>>> The PodMonitor metadata must match that of the Prometheus operator,
> > >>>> like the namespace, the labels, ...
> > >>>>
> > >>>> Some documentation is available at:
> > >>>>
> > >>>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
> > >>>>
> > >>>> That contains some links to the Prometheus operator documentation for
> > >>>> troubleshooting why the metrics endpoint is not discovered.
> > >>>>
> > >>>>> On 25 Nov 2021, at 12:25, Roberto Camelk
> > >>>>> <betonetotbo.cam...@gmail.com> wrote:
> > >>>>>
> > >>>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster
> > >>>>> monitoring enabled in rancher, so that exists a Prometheus instance
> > >>>>> running, so as a Prometheus Operator.
> > >>>>>
> > >>>>> Recently I deployed a Apache Camel-K operator, and now I want to
> > >>>>> enable the prometheus integration for collect metrics about my camel
> > >>>>> routes.
> > >>>>>
> > >>>>> So, my Camel-K operator is running in namescape camel-k and the
> > >>>>> rancher embedded prometheus stack in cattle-prometheus namespace.
> > >>>>>
> > >>>>> I just have launched my route with the trait --trait
> > >>>>> prometheus.enabled=true, but the camel metrics aren't listing at my
> > >>>>> prometheus.
> > >>>>>
> > >>>>> Anyone knows why or what I need to configure to my camel-k route
> > >>>>> deploy it's metrics at the rancher embedded prometheus?
> > >>>>
> > >>
> >
