Without any more info, it looks like someone / something is abusing your Prometheus server API endpoint. Or something has changed such that your API endpoint is very slow.
I would look at the "prometheus_http_request_duration_seconds" histogram to find out if there is a specific API endpoint that is being abused. On Sat, Oct 23, 2021 at 11:09 AM weidong zou <[email protected]> wrote: > version : prometheus:v2.28.1 > > Suddenly this problem occurred in my prometheus,Causes the memory to burst. > > Something like this: > > - level=error ts=2021-10-23T08:31:39.022Z caller=api.go:1491 > component=web msg="error writing response" bytesWritten=0 err="write tcp > 10.194.15.xx:9090->10.194.25.xx:57378: write: broken pipe" > - level=error ts=2021-10-23T08:31:41.561Z caller=api.go:1491 > component=web msg="error writing response" bytesWritten=0 err="write tcp > 10.194.15.xx:9090->10.194.24.xx:23920: write: broken pipe" > - level=error ts=2021-10-23T08:31:42.893Z caller=api.go:1491 > component=web msg="error writing response" bytesWritten=0 err="write tcp > 10.194.15.xx:9090->10.194.25.xx:10736: write: broken pipe" > > [image: err.PNG] > > What is the problem？ > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/ddbefbe2-d13b-410e-b90b-12ae837abbd6n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/ddbefbe2-d13b-410e-b90b-12ae837abbd6n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpt8KoEy08sHYuic%3DrM5ZF6RedzTX2Z_TBohzk%3DEpDH4Q%40mail.gmail.com.
