RE: [users@httpd] Apache in under attack. [EXT]
The first place to look in this case is the size of the apache processes. Once the OP has got on top of this - then other issues can be investigated. So process would be: 1) Reduce number of modules in Apache (>100 at the moment) should be around 15-25 region; 2) Look at memory usage; 3) If high would also look to see which PHP packages have been installed; 4) Once past these I would start looking at the actual attack and the particular requests; -Original Message- From: @lbutlr Sent: 15 January 2021 06:37 To: users@httpd.apache.org Subject: Re: [users@httpd] Apache in under attack. [EXT] On 14 Jan 2021, at 04:48, Jason Long wrote: > Server have 4 CPU cores and 6GB of RAM. > I pasted Apache configuration. In your opinion, which parts of servers must > be examine? Throwing more resources at the problem is not likely to fix the problem. You need to figure out what is going on with your server and WHY it is taking so much time it is bogging down and WEHRE the slowdown is happening. This is not something that someone can just say "Oh, it's this" because the problem is unique to your machine, your content, and your users. I would start with those very suspicious (to me) looking URL requests containing dozens of digits of hex. Do those look like they are legitimate links to your server's web content? Also, please stop top-posting and quoting the entire message thread below. -- We are born naked, wet and hungry; then it's all downhill. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache in under attack.
On 14 Jan 2021, at 04:48, Jason Long wrote: > Server have 4 CPU cores and 6GB of RAM. > I pasted Apache configuration. In your opinion, which parts of servers must > be examine? Throwing more resources at the problem is not likely to fix the problem. You need to figure out what is going on with your server and WHY it is taking so much time it is bogging down and WEHRE the slowdown is happening. This is not something that someone can just say "Oh, it's this" because the problem is unique to your machine, your content, and your users. I would start with those very suspicious (to me) looking URL requests containing dozens of digits of hex. Do those look like they are legitimate links to your server's web content? Also, please stop top-posting and quoting the entire message thread below. -- We are born naked, wet and hungry; then it's all downhill. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache in under attack.
You should look at adding the %D and %T format strings to your httpd access log configuration so that you can capture the amount of time spent in delivery of a resource. > Date: Thursday, January 14, 2021 11:48:55 + > From: Jason Long > > Server have 4 CPU cores and 6GB of RAM. > I pasted Apache configuration. In your opinion, which parts of > servers must be examine? > > > On Wednesday, January 13, 2021, 08:30:58 PM GMT+3:30, @lbutlr > wrote: > > >> On 12 Jan 2021, at 01:52, Jason Long >> wrote: >> >> It show me: >> >> 13180 X.X.X.X >> 1127 X.X.X.X >> 346 X.X.X.X >> 294 X.X.X.X >> 241 X.X.X.X >> 169 X.X.X.X >> 168 X.X.X.X >> 157 X.X.X.X >> 155 X.X.X.X >> 153 X.X.X.X > > Your server would not be getting bogged down by that few > connections unless your hardware is very weak or you are hosting > something insane. > > I have a very lightly used web server that gets more than 40K hits > a day running on a Celeron machine with a whole 4GB of RAM and my > load average is in the 1.2 range consistently. > > I wonder if there is not some configuration error. > > Also, the URLs shown in your logs starting with /tag/ followed by a > long series of hex digits, do those look like valid URLs for your > server? > > Do a dig -x on the IP that is hitting you 13,000 times and see > where it is. You can try firewalling it, but if it's not some > misconfigured server, the DOS will simply move to another IP. > >> https://paste.ubuntu.com/p/PsxM8yPXPQ/ > > I haven't run F2B in quite a while, but is that a list of IPs that > you are whitelisiing or does [Protect] mean "Protect FROM"? > > But if 13,000 queries are crippling your web server, I think your > real problem lies elsewhere than the 13,000 hits. > > (You are loading almost double the modules that I am, by the way. > It seems like an lot. Do you know why each of those modules is > enabled?) End Original Message - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
On Thu, Jan 14, 2021 at 2:14 PM Yann Ylavic wrote: > > You could use a "ProxyPass ... ttl=[timeout]" on the proxy with > [timeout] < KeepAliveTimeout on the backend (say the KeepAliveTimeout > on the backend's vhost is 5, try ttl=3 in the proxy's ProxyPass > directive). > By doing this, you avoid the case where the proxy is reusing a > connection that is being closed by the backend at the same time > (because its KeepAliveTimeout expires), the more connections the > greater the risk of this happening.. > > But this is unlikely to fix the errors raised by the backend, because > those seem to happen on well established connections. > I may be missing something though, so this first step could help already.. As a first first-step and before fine tuning like described above though, I'd suggest completely disabling keepalive to see if it helps. So either "ProxyPass ... disablereuse=on" on the proxy or "KeepAlive off" on the backend. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
Hi, On Thu, Jan 14, 2021 at 11:15 AM Bram Mertens wrote: > > Our setup: > Subversion 1.10 on RHEL8 served by httpd 2.4 > Reverse proxy httpd 2.4 on RHEL8 [snip backend errors] > Am I correct to assume that it is the client (or something between the client > and the proxy server) that is breaking the connection? Or is this a problem > on either of the apache servers? The errors on the backend suggest that the connection with the client was aborted (downstream). Do determine whether it happens before the reverse proxy or between the proxy and the backend you'd have to capture traffic there (tcpdump/wireshark to look for tcp connections reset) or use "LogLevel trace1" on the proxy (that's the level where client network errors are logged there). > > In the error_log of the proxy server I see errors like: > > [Thu Jan 14 06:42:01.647633 2021] [proxy_http:error] [pid 17204:tid > 140412413015808] (104)Connection reset by peer: [client 192.168.x.y:38940] > AH01102: error reading status line from remote server :443 This may be the symptom of a KeepAliveTimeout mismatch between the proxy and the backend, which can be addressed by using a slightly lower value on the proxy. You could use a "ProxyPass ... ttl=[timeout]" on the proxy with [timeout] < KeepAliveTimeout on the backend (say the KeepAliveTimeout on the backend's vhost is 5, try ttl=3 in the proxy's ProxyPass directive). By doing this, you avoid the case where the proxy is reusing a connection that is being closed by the backend at the same time (because its KeepAliveTimeout expires), the more connections the greater the risk of this happening.. But this is unlikely to fix the errors raised by the backend, because those seem to happen on well established connections. I may be missing something though, so this first step could help already.. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache in under attack.
Server have 4 CPU cores and 6GB of RAM. I pasted Apache configuration. In your opinion, which parts of servers must be examine? On Wednesday, January 13, 2021, 08:30:58 PM GMT+3:30, @lbutlr wrote: > On 12 Jan 2021, at 01:52, Jason Long wrote: > > It show me: > > 13180 X.X.X.X > 1127 X.X.X.X > 346 X.X.X.X > 294 X.X.X.X > 241 X.X.X.X > 169 X.X.X.X > 168 X.X.X.X > 157 X.X.X.X > 155 X.X.X.X > 153 X.X.X.X Your server would not be getting bogged down by that few connections unless your hardware is very weak or you are hosting something insane. I have a very lightly used web server that gets more than 40K hits a day running on a Celeron machine with a whole 4GB of RAM and my load average is in the 1.2 range consistently. I wonder if there is not some configuration error. Also, the URLs shown in your logs starting with /tag/ followed by a long series of hex digits, do those look like valid URLs for your server? Do a dig -x on the IP that is hitting you 13,000 times and see where it is. You can try firewalling it, but if it's not some misconfigured server, the DOS will simply move to another IP. > https://paste.ubuntu.com/p/PsxM8yPXPQ/ I haven't run F2B in quite a while, but is that a list of IPs that you are whitelisiing or does [Protect] mean "Protect FROM"? But if 13,000 queries are crippling your web server, I think your real problem lies elsewhere than the 13,000 hits. (You are loading almost double the modules that I am, by the way. It seems like an lot. Do you know why each of those modules is enabled?) -- They say whisky'll kill you, but I don't think it will I'm ridin' with you to the top of the hill - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
Hi, Our setup: Subversion 1.10 on RHEL8 served by httpd 2.4 Reverse proxy httpd 2.4 on RHEL8 We're seeing one or more failures/day during SVN checkout/updates. There appear to be some variations on the error: These are the logs on the SVN (backend) server First type: [dav:error] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] A failure occurred while driving the update report editor [500, #104] [dav:error] Error writing base64 data: Connection reset by peer [500, #104] Second type: [dav:error] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] A failure occurred while driving the update report editor [500, #32] [dav:error] Error writing base64 data: Broken pipe [500, #32] Third: [dav:error] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] Broken pipe [400, #32] Fourth: [dav:error] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] Connection reset by peer [400, #104] Clients are both TortoiseSVN on Windows and Jenkins SVNkit on Windows and Linux. Am I correct to assume that it is the client (or something between the client and the proxy server) that is breaking the connection? Or is this a problem on either of the apache servers? In the error_log of the proxy server I see errors like: [Thu Jan 14 06:42:01.647633 2021] [proxy_http:error] [pid 17204:tid 140412413015808] (104)Connection reset by peer: [client 192.168.x.y:38940] AH01102: error reading status line from remote server :443 For this error the following SO post (https://stackoverflow.com/a/64365796/1313616) suggests to add ` SetEnv proxy-initial-not-pooled 1` but https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html warns that this may downgrade performance. But the timings of these errors do not match the dav:error error messages. Clocks on both servers are synchronized via NTP. How can I fix/troubleshoot this further? I cannot reliably reproduce this on the dev server using a single large checkout. I have found that when a lot of checkouts are triggered simultaneously the error happened. Whether that was because of the high load or simply because more checkouts increase the chance of running into the problem is not clear to me. Thanks in advance Bram
