I find the root cause is that our http client didn't use connection pool, On Saturday, March 7, 2020 at 12:03:49 PM UTC+8, Kaiming Wan wrote: > > Env info: > prometheus version: 2.16.0 > linux: centos7 > prometheus config: > system config: 8 core 16g ram > > ``` > --web.max-connections=1024 --web.read-timeout=5m > --storage.tsdb.retention.time=30d --query.max-concurrency=200 > --query.timeout=2m --storage.tsdb.path=/home/admin/app/prometheus > --config.file=/home/admin/app/prometheus/conf/prometheus.yml > --log.level=debug > ``` > > > > We use springboot together with okhttp to query with prometheus's http > api. However, the query performance is poor. We have about 30 queries > sending to prometheus server in parallel every 10 seconds. However, after > waiting for about 1 minutes, we can see socket timeout exception in spring > boot, because the sended query meet the http client's timeout setting 10 > seconds. I can't find any exception in prometheus logs and I see all query > is very fast in query log(around 30 ms),only some query can see info > 'prometheus query was canceled in query queue'. And when these request are > timeout, we find the prometheus's host cpu usage is very low. It seems that > the prometheus haven't try its best to process these request. It is strange > that such many requests are timeout when the prometheus host system load is > low like this. > > > All the after http request are timeout one by one. Here come my question: > > 1. Are there any query performance benchmark data for prometheus? I know > it works well with high write performance, but there is little info about > its query performance. > > 2. Is it normal that prometheus can only support such a little requests in > parallel? > > 3. How do we utilize all the cpu system resource for prometheus to speed > up its query performance > > 4. How can i analyze why the query execute so slow in prometheus server? >
-- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cdc8b6d4-3529-485a-b000-64cadc241142%40googlegroups.com.
