Hello,
We are monitoring some 30+ Kafka clusters.
For this we are using the prometheus configuration described on Confluent
Cloud metrics api. Confluent Cloud Metrics API
<https://api.telemetry.confluent.cloud/docs?_ga=2.223081368.2091268830.1644217135-162094420.1635422624&_gac=1.220521132.1644217682.EAIaIQobChMI0I71koTt9QIVAfSzCh026QrpEAAYASAAEgLQPPD_BwE#tag/Version-2/paths/~1v2~1metrics~1%7Bdataset%7D~1export/get>
scrape_configs:
- job_name: Confluent Cloud
scrape_interval: 1m
scrape_timeout: 1m
honor_timestamps: true
static_configs:
- targets: - api.telemetry.confluent.cloud
scheme: https
basic_auth:
username: <Cloud API Key>
password: <Cloud API Secret>
metrics_path: /v2/metrics/cloud/export
params:
"resource.kafka.id":
- lkc-1
- lkc-2
- lkc-lots-more
This works well and all metrics are visual in both Promtheus and Grafana.
But once they are sent of to an Elastic agent running the prometheus
remote_write module, something is happening which i cant figure out.
Metrics are consistently arriving in Elasticsearch with 5 mins delay.
prometheus remote write queue config is pretty much the defaults from
prometheus documenation.
queue_config:
capacity: 2500
max_samples_per_send: 1000
max_shards: 200
min_shards: 1
batch_send_deadline: 5s
min_backoff: 30ms
max_backoff: 5s
retry_on_http_429: true
metadata_config:
send: true
send_interval: 1m
max_samples_per_send: 1000
Other metrics such as metrics from Cofluent kafka rest proxy and black box
exporter is ingested into elastic in a timely manner. Only the Confluent
Cloud metrics are delayed.
Prometheus is deployed using Helm.
Any of you fine minds have any idea how i can eliminate or at least reduce
this latency?
Best regards
Christian Oelsner
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/1d04f298-1b10-4750-ad2b-b99f7c2427efn%40googlegroups.com.
