Hello Brian, Thanks for replying. Yes I understand that this is unexpected load and 20-30 QPS is beyond the limits of jmx exporter. I was proposing to add configuration to limit the connection rates from the exporter itself (when running exporter as stand alone sever).
To answer your question about reconsidering the use of exporter, My team is maintaining the exporter and other teams are reading data from exporter via exposed port. In some cases(say, bad Prometheus server scrape interval ) can cause such unexpected load. In order to avoid such issues, I wanted to have rate limiting at exporter layer itself. Can you please also point me to hard-coded threads count limit in jmx-exporter code? On Thursday, December 10, 2020 at 4:04:56 PM UTC [email protected] wrote: > On Thu, 10 Dec 2020 at 15:42, [email protected] <[email protected]> > wrote: > >> >> *Background:* >> When JMX exporter is overloaded (say 20-30 qps), we have observed that >> some of the requests take more than 20 sec to serve which was higher than >> the client-side request timeout. As a result, the agent tried to send >> response on connections already closed, which has two consequences: (a) it >> resulted errors and (b) it sometimes led to the socket channel being >> blocked indefinitely on the write syscall. Eventually, all threads of the >> HTTP server in the Prometheus agent get stuck and no more requests can be >> accepted. However, the thread accepting connections is still active and new >> connections are created but never actually used, and since all request >> threads of the HTTP server are stuck, the connections are never closed by >> the server, resulting in a long backlog of CLOSE_WAIT sockets waiting to >> be closed. >> >> *Proposed Solution* >> >> *1. Limit connections* >> We want to limit the number of connection to the exporter agents. There >> is no native way for jmx exporter to put such restrictions(To over come >> this we will be adding ip table rules for jmx port). >> > > There is already a hardcoded limit of 5 threads in the exporter. > > It sounds like the real problem here is excessive load on the exporter, > 20-30qps is far from typical usage for any exporter before even considering > the inefficiencies of JMX. I'd suggest reconsidering how you are using the > exporter. > > Brian > > >> >> *2. Adding timeouts to requestes* >> This could be easily achieved by JVM settings. But it would be nice to >> add these in jmx-exporter's documentation. >> >> *-Dsun.net.httpserver.maxReqTime=20 -Dsun.net.httpserver.maxRspTime=20 * >> >> *Please let me know if this idea makes sense to community. I can work on >> design.* >> >> >> Thanks >> Brajesh Kumar >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/d3d0c669-0794-47e9-b0db-96cf2cb2d8efn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-developers/d3d0c669-0794-47e9-b0db-96cf2cb2d8efn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/c8c7dc40-7f35-4bb4-8f56-b19f6d284168n%40googlegroups.com.
