Hi Mateusz When you run ab against your ATS to create high enough artificial load, do all requests succeed and how quickly? Using some tool like wireshark during ab testing to dump and analyze TCP traffic could then give you hint where exactly the failure happens. To exclude any network bottlenecks, run ab locally on server.
Veiko 2017-12-28 22:58 GMT+02:00 Mateusz Zajakala <[email protected]>: > Hi, > > I'm trying to optimize the throughput of ATS 6.2.0 running on 16G / 8 > cores server. ATS handles up to 7 Gbps of traffic (circa 500 requests / > second) serving up to 80% of traffic from ram-disk-based cache. > > The problem I'm seeing is that from time to time my http clients can't > connect to the server reasonably fast (I define this as 1 second to > establish TCP conn). Unfortunately, http keep alive is not used by clients, > so those 500 request / second are all made over new TCP connections. > Clients connect, retrieve the file and disconnect. I do realize the > overheads, but this is not something I can easily change (client-side)... > > I'm wondering what I can do to improve the performance and eliminate those > failed connection attempts. Some ideas I have tried > - 30000 connection throttle in records.config (afaik this also sets the > max no of open files for ATS) > - tcp_fin_timeout is set to 1 - I'm not running out of ports because of > sockets stuck in TIME_WAIT, I have checked. At any given time I have no > more than 1k TCP connections open > > Unfortunately, I'm not sure where these incomng connections are > dropped/stuck and I'm not sure which TCP stats would help understanding > this. I have also not tweaked around default Centos 7 TCP settings as I > don't feel competent enough. > > One thing that caught my attention is proxy.config.accept_threads value > set to 1 (default). This seems really low given the traffic, but I read > somewhere that it's best left at that. Can you please comment on that? > Shouldn't this value be adjusted (e.g. 4 or more)? Or even move the accepts > to worker threads? > > I'm not seeing any meaningful errors in ATS logs, but there are no debug > tsgs enabled. Any suggestion on how to debug / improve much appreciated. > > Thanks > Mateusz > > >
