On 04/02/2021 16:42, Jeff Tippey wrote:
Thanks Stuart,
I have looked around and I can't find it. I'd be grateful if
someone could take a look at one of these dropped pods labels and my
scrape config and let me know if you see where it is dropped.
It looks like this bit might be dropping the target:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: keep
target_label: __metrics_path__
regex: (.+)
It is looking for an annotation called prometheus_io_path which is
copied to the __metrics_path__ label (which sets the path to scrape).
As this annotation doesn't seem to exist in your example this doesn't
match, and because the action is keep the target is dropped.
So try adding that annotation with the correct metric path value.
Not working Config
__address__="xxxxxxxx"
__meta_kubernetes_endpoint_address_target_kind="Pod"
__meta_kubernetes_endpoint_address_target_name="rcax-ios0-br-1-56z6v"
__meta_kubernetes_endpoint_port_name="queuemonitor"
__meta_kubernetes_endpoint_port_protocol="TCP"
__meta_kubernetes_endpoint_ready="true"
__meta_kubernetes_endpoints_name="rcax-ios-0"
__meta_kubernetes_namespace="envb"
__meta_kubernetes_pod_annotation_openshift_io_deployment_config_latest_version="1"
__meta_kubernetes_pod_annotation_openshift_io_deployment_config_name="rcax-ios0-br"
__meta_kubernetes_pod_annotation_openshift_io_deployment_name="rcax-ios0-br-1"
__meta_kubernetes_pod_annotation_openshift_io_scc="restricted"
__meta_kubernetes_pod_container_name="queuemonitor"
__meta_kubernetes_pod_container_port_name=""
__meta_kubernetes_pod_container_port_number="8001"
__meta_kubernetes_pod_container_port_protocol="TCP"
__meta_kubernetes_pod_controller_kind="ReplicationController"
__meta_kubernetes_pod_controller_name="rcax-ios0-br-1"
__meta_kubernetes_pod_host_ip="xxxxxxxx"
__meta_kubernetes_pod_ip="xxxxxxxx"
__meta_kubernetes_pod_label_component="rcax-ios0-br"
__meta_kubernetes_pod_label_deployment="rcax-ios0-br-1"
__meta_kubernetes_pod_label_deploymentconfig="rcax-ios0-br"
__meta_kubernetes_pod_label_scac="rcax"
__meta_kubernetes_pod_label_subsystem="rcax-ios0"
__meta_kubernetes_pod_name="rcax-ios0-br-1-56z6v"
__meta_kubernetes_pod_node_name="stb-node002"
__meta_kubernetes_pod_ready="true"
__meta_kubernetes_pod_uid="356fe668-5a73-11eb-bf8a-0050568584bc"
__meta_kubernetes_service_annotation_prometheus_io_port="8001"
__meta_kubernetes_service_annotation_prometheus_io_scheme="http"
__meta_kubernetes_service_annotation_prometheus_io_scrape="true"
__meta_kubernetes_service_label_name="rcax-ios0"
__meta_kubernetes_service_name="rcax-ios-0"
__metrics_path__="/metrics"
__scheme__="http"
job="kubernetes-service-endpoints"
Scrape Config
- job_name: 'kubernetes-service-endpoints'
tls_config:
ca_file:
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
<http://kubernetes.io/serviceaccount/ca.crt>
# TODO: this should be per target
insecure_skip_verify: true
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
# only scrape infrastructure components
- source_labels: [__meta_kubernetes_namespace]
action: keep
regex:
'default|logging|metrics|kube-.+|openshift|openshift-.+|envb|envc'
# drop infrastructure components managed by other scrape targets
- source_labels: [__meta_kubernetes_service_name]
action: drop
regex: 'prometheus-node-exporter'
# only those that have requested scraping
- source_labels:
[__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels:
[__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels:
[__meta_kubernetes_service_annotation_prometheus_io_path]
action: keep
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__,
__meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+)(?::\d+);(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
On Monday, January 25, 2021 at 4:35:28 PM UTC-6 Stuart Clark wrote:
On 25/01/2021 22:15, Jeff Tippey wrote:
Hi,
I have a custom python container I wrote that exports metrics
on port 8001 at "/metrics". The setup used to work in Openshift
3.6 with Prometheus. I'm using the Prometheus configuration
from github at prometheus github link
<https://github.com/openshift/origin/tree/release-3.11/examples/prometheus>.
In my deployment yamls for my container, For the list (in
deployment yaml) I have annotation
annotations:
prometheus.io/path <http://prometheus.io/path>: /metrics
prometheus.io/port <http://prometheus.io/port>: "8001"
prometheus.io/scrape <http://prometheus.io/scrape>: "true"
and the service definition I have
annotations:
prometheus.io/scrape <http://prometheus.io/scrape>: "true"
prometheus.io/scheme <http://prometheus.io/scheme>: http
prometheus.io/port <http://prometheus.io/port>: "8001"
To repeat, this worked in 3.6 configuration. However, when I
open up Prometheus this is not a target. Under Targets, none of
my service endpoints are showing up. I do see them in the
service discovery but they are being dropped. When, I look in
the prometheus.yaml, it says that it should scrape anything that
is io.scrape = "true". Has anyone experienced this issue or
know why it would be dropping my service endpoint scrapes that
are defined as io.scrape="true".
If not, does anyone know where the reason they are dropped is
logged or how to access? I have been looking around and am at a
loss right now.
Your help is greatly appreciated,
If you look through the relabling rules for the job you should be
able to see what is going on. Look at the details on the SD page
and work out which rule is dropping it. It should then suggest
what you need to do to make it work.
--
You received this message because you are subscribed to the Google
Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/0598e514-fdd5-449c-aa77-5241957c3446n%40googlegroups.com
<https://groups.google.com/d/msgid/prometheus-users/0598e514-fdd5-449c-aa77-5241957c3446n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/40370de2-b9e5-0553-f701-4d323525e9c0%40Jahingo.com.
