Env info: prometheus version: 2.16.0 linux: centos7 prometheus config: system config: 8 core 16g ram
``` --web.max-connections=1024 --web.read-timeout=5m --storage.tsdb.retention.time=30d --query.max-concurrency=200 --query.timeout=2m --storage.tsdb.path=/home/admin/app/prometheus --config.file=/home/admin/app/prometheus/conf/prometheus.yml --log.level=debug ``` We use springboot together with okhttp to query with prometheus's http api. However, the query performance is poor. We have about 30 queries sending to prometheus server in parallel every 10 seconds. However, after waiting for about 1 minutes, we can see socket timeout exception in spring boot, because the sended query meet the http client's timeout setting 10 seconds. I can't find any exception in prometheus logs and I see all query is very fast in query log(around 30 ms),only some query can see info 'prometheus query was canceled in query queue'. And when these request are timeout, we find the prometheus's host cpu usage is very low. It seems that the prometheus haven't try its best to process these request. It is strange that such many requests are timeout when the prometheus host system load is low like this. All the after http request are timeout one by one. Here come my question: 1. Are there any query performance benchmark data for prometheus? I know it works well with high write performance, but there is little info about its query performance. 2. Is it normal that prometheus can only support such a little requests in parallel? 3. How do we utilize all the cpu system resource for prometheus to speed up its query performance 4. How can i analyze why the query execute so slow in prometheus server? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/737a5640-ca50-4bb6-b7dd-05b002f40346%40googlegroups.com.
