Thanks Stuart,
I have looked around and I can't find it. I'd be grateful if someone
could take a look at one of these dropped pods labels and my scrape config
and let me know if you see where it is dropped.
Not working Config
__address__="xxxxxxxx"
__meta_kubernetes_endpoint_address_target_kind="Pod"
__meta_kubernetes_endpoint_address_target_name="rcax-ios0-br-1-56z6v"
__meta_kubernetes_endpoint_port_name="queuemonitor"
__meta_kubernetes_endpoint_port_protocol="TCP"
__meta_kubernetes_endpoint_ready="true"
__meta_kubernetes_endpoints_name="rcax-ios-0"
__meta_kubernetes_namespace="envb"
__meta_kubernetes_pod_annotation_openshift_io_deployment_config_latest_version="1"
__meta_kubernetes_pod_annotation_openshift_io_deployment_config_name="rcax-ios0-br"
__meta_kubernetes_pod_annotation_openshift_io_deployment_name="rcax-ios0-br-1"
__meta_kubernetes_pod_annotation_openshift_io_scc="restricted"
__meta_kubernetes_pod_container_name="queuemonitor"
__meta_kubernetes_pod_container_port_name=""
__meta_kubernetes_pod_container_port_number="8001"
__meta_kubernetes_pod_container_port_protocol="TCP"
__meta_kubernetes_pod_controller_kind="ReplicationController"
__meta_kubernetes_pod_controller_name="rcax-ios0-br-1"
__meta_kubernetes_pod_host_ip="xxxxxxxx"
__meta_kubernetes_pod_ip="xxxxxxxx"
__meta_kubernetes_pod_label_component="rcax-ios0-br"
__meta_kubernetes_pod_label_deployment="rcax-ios0-br-1"
__meta_kubernetes_pod_label_deploymentconfig="rcax-ios0-br"
__meta_kubernetes_pod_label_scac="rcax"
__meta_kubernetes_pod_label_subsystem="rcax-ios0"
__meta_kubernetes_pod_name="rcax-ios0-br-1-56z6v"
__meta_kubernetes_pod_node_name="stb-node002"
__meta_kubernetes_pod_ready="true"
__meta_kubernetes_pod_uid="356fe668-5a73-11eb-bf8a-0050568584bc"
__meta_kubernetes_service_annotation_prometheus_io_port="8001"
__meta_kubernetes_service_annotation_prometheus_io_scheme="http"
__meta_kubernetes_service_annotation_prometheus_io_scrape="true"
__meta_kubernetes_service_label_name="rcax-ios0"
__meta_kubernetes_service_name="rcax-ios-0"
__metrics_path__="/metrics"
__scheme__="http"
job="kubernetes-service-endpoints"
Scrape Config
- job_name: 'kubernetes-service-endpoints'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# TODO: this should be per target
insecure_skip_verify: true
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
# only scrape infrastructure components
- source_labels: [__meta_kubernetes_namespace]
action: keep
regex:
'default|logging|metrics|kube-.+|openshift|openshift-.+|envb|envc'
# drop infrastructure components managed by other scrape targets
- source_labels: [__meta_kubernetes_service_name]
action: drop
regex: 'prometheus-node-exporter'
# only those that have requested scraping
- source_labels:
[__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels:
[__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels:
[__meta_kubernetes_service_annotation_prometheus_io_path]
action: keep
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__,
__meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+)(?::\d+);(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
On Monday, January 25, 2021 at 4:35:28 PM UTC-6 Stuart Clark wrote:
> On 25/01/2021 22:15, Jeff Tippey wrote:
>
> Hi,
>
> I have a custom python container I wrote that exports metrics on port
> 8001 at "/metrics". The setup used to work in Openshift 3.6 with
> Prometheus. I'm using the Prometheus configuration from github at
> prometheus
> github link
> <https://github.com/openshift/origin/tree/release-3.11/examples/prometheus>.
>
>
> In my deployment yamls for my container, For the list (in deployment
> yaml) I have annotation
>
> annotations:
> prometheus.io/path: /metrics
> prometheus.io/port: "8001"
> prometheus.io/scrape: "true"
>
> and the service definition I have
>
> annotations:
> prometheus.io/scrape: "true"
> prometheus.io/scheme: http
> prometheus.io/port: "8001"
>
> To repeat, this worked in 3.6 configuration. However, when I open up
> Prometheus this is not a target. Under Targets, none of my service
> endpoints are showing up. I do see them in the service discovery but they
> are being dropped. When, I look in the prometheus.yaml, it says that it
> should scrape anything that is io.scrape = "true". Has anyone experienced
> this issue or know why it would be dropping my service endpoint scrapes
> that are defined as io.scrape="true".
>
> If not, does anyone know where the reason they are dropped is logged or
> how to access? I have been looking around and am at a loss right now.
>
> Your help is greatly appreciated,
>
> If you look through the relabling rules for the job you should be able to
> see what is going on. Look at the details on the SD page and work out which
> rule is dropping it. It should then suggest what you need to do to make it
> work.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/0598e514-fdd5-449c-aa77-5241957c3446n%40googlegroups.com.
