> If I am reading the documentation correctly as documented here - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config - any existing labels on a time series should not be over-written by the external label value if honor labels is set to true.
I presume you are referring to this part of the documentation? # Note that any globally configured "external_labels" are unaffected by this # setting. In communication with external systems, they are always applied only # when a time series does not have a given label yet and are ignored otherwise. The first part explicitly says that honor_labels (i.e. the scrape job setting) has no effect on external labels. honor_labels applies to a scrape job and only affects scraping, not reading. On the other hand, the second part does say that external labels are only applied when the label does not already exist. Is that the issue you're raising? If so, perhaps you can describe things like the versions of prometheus and thanos you're using. I think it would also be worth trying to query the federate endpoint, to see whether external labels behave the way you expect there (i.e. is this specifically a problem with thanos?) On Wednesday, 10 August 2022 at 21:23:48 UTC+1 Jimmy the Greek wrote: > If I am reading the documentation correctly as documented here - > https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config > > - any existing labels on a time series should not be over-written by the > external label value if honor labels is set to true. > > However in reality I am not seeing this as the case. Configuration snippet: > > global: > scrape_interval: 15s > scrape_timeout: 10s > evaluation_interval: 30s > external_labels: > account: services > environment: infrastructure > prometheus: prometheus/prometheus prometheus_replica: > prometheus-prometheus-0 > region: us-west-2 > > scrape_configs: > - job_name: kafka/node > honor_labels: true > honor_timestamps: true > scrape_interval: 15s > scrape_timeout: 10s > metrics_path: /metrics > scheme: http > follow_redirects: true > enable_http2: true > static_configs: > - targets: > - b-1.kafka.xxxxx.kafka.us-west-2.amazonaws.com:11002 > - b-2.kafka.xxxxxxx.kafka.us-west-2.amazonaws.com:11002 > labels: > account: non-prod > > > When I query these metrics from the prometheus server directly the account > name is non-prod. However when pulled through Thanos which aggregates > across many clusters the value is set to services. > > Thanks > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5c0f8c4b-8218-43a0-9907-e43d89888577n%40googlegroups.com.
