I did as well just add
relabelings:
- action: labeldrop
regex: (pod|service|endpoint|namespace)
since looks like up to date prometheusrule uses this for
kube-state-metrics.
On Wednesday, August 26, 2020 at 9:24:38 PM UTC-5 Rodrigo Martinez wrote:
> Have been updating old prometheus rules and have noticed less errors.
> However there are some taken from other components.
>
> Currently I have ceph running and using their prometheusrule from their
> examples
> I see issues
>
> kube_node_status_condition{condition="Ready",job="kube-state-metrics",status="true"}
>
> * on (node) group_right()
> max(label_replace(ceph_disk_occupation{job="rook-ceph-mgr"},"node","$1","exported_instance","(.*)"))
>
> by (node)
>
> on cluster with no thanos integration no issues. But with thanos i see
> collision .
>
> when looking at left side of metric I do see that it returns
> metrics twice
>
> kube_node_status_condition{condition="Ready",instance="0.0.0.0:8080
> ",job="kube-state-metrics",node="test-node",status="true"}
> kube_node_status_condition{condition="Ready",endpoint="http",instance="
> 0.0.0.0:8080
> ",job="kube-state-metrics",namespace="monitoring",node="test-node",pod="kube-state-metrics-567789848b-9d77w",service="kube-state-metrics",status="true"}
>
> Have to see what I`m doing wrong on my end. But if its noticeable from
> what is being displayed do let me know . As this issue was not seen until I
> enabled thanos side car.
>
> Thanks
>
> On Tuesday, August 25, 2020 at 9:49:36 AM UTC-5 [email protected] wrote:
>
>> - source_labels: [__meta_kubernetes_service_name]
>> separator: ;
>> regex: (.*)
>> target_label: job
>> replacement: ${1}
>> action: replace
>> - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
>> separator: ;
>> regex: (.+)
>> target_label: job
>> replacement: ${1}
>> action: replace
>>
>>
>> I'm not sure what you're attempting to do here, but it is risky to mess
>> with the "job" label. This is the one prometheus itself sets to identify
>> the scrape job where the metric originated, and if you end up scraping the
>> same target multiple times from different jobs, this label ensures that the
>> timeseries have unique label sets.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/05de6707-2911-4d48-8b3b-53eb8cba8d25n%40googlegroups.com.
