Jaaz, On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote: > On 27.11.2016 19:03, Jaaz Portal wrote: >> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >> >>> On 27.11.2016 14:26, Jaaz Portal wrote: >>> >>>> hi, >>>> everything i know so far is just this single log line that appeared in >>>> apache error.log >>>> >>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid >>>> 1397934896385 >>>> 92] AH00484: server reached MaxRequestWorkers setting, consider raising >>>> the >>>> MaxR >>>> equestWorkers setting >>>> >>>> there was nothing else, just this strange line >>>> >>> >>> This is not a "strange" line. It is telling you something. >>> One problem is that you seem convinced in advance, without serious >>> proof, >>> that there is a "bug" or a vulnerability in httpd or tomcat. >>> Read the explanation of the httpd parameter, here : >>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers >>> and try to understand it. >>> >> >> I understand it very well. We are serving no more that 500 clients per >> day >> so there was no other option that some kind of attack. >> >> >> About the "bug" or "vulnerability" : a webserver is supposed to serve >> HTTP >>> requests from clients. That is what you expect of it. It has no >>> choice but >>> to accept any client connection and request, up to the maximum it can >>> handle considering its configuration and the capacity of the system on >>> which it runs. That is not a bug, it is a feature. >>> >>> >> We have some weeks ago come under attack from some Polish group. It was >> first bind that was DoS'ed, we was running on stable Debian so i updated >> bind to latest version. It did not helped. They has dos'ed it so we >> switched to other dns provider. That has helped. >> >> Then they exploited some well know vulnerability in mod_proxy. We have >> updated apache to the latest but again they has exploited it, so we have >> switched to mod_jk. And then guess what. They exploited it too so i >> decided >> to write to this list looking for help before trying jetty. >> >> >>> >>> The normal Apache httpd access log, will log a request only when it is >>> finished. If the request never finishes, it will not get logged. >>> That may be why you do not see these requests in the log. >>> But have a look at this Apache httpd module : >>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html >>> >> >> ok, thanks, will take care >> >> Note : that is also why I was telling you to enable the mod_jk log, >> and to >>> examine it. >>> Because mod_jk will also log information before the request produces a >>> response. >>> >>> >>> and server hanged. >>> >>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ? >>> >> >> Apache was accepting connection but not processed it. After restart of >> tomcat server it worked again. >> >> >>> Also in >>> >>>> access logs there are no clues that it was under any heavy load. >>>> >>>> after around hour after discovering that our server hanged-out we have >>>> restarted tomcat server >>>> and it worked again >>>> >>> >>> Yes, because that will close all connections between Apache httpd and >>> tomcat, and abort all requests that are in the process of being >>> processed >>> by tomcat. So mod_jk will get an error from tomcat, and will report an >>> error to httpd, and httpd will communicate that error to the clients, >>> and >>> close their connection. >>> It still does not tell you what the problem was. >>> The only thing that it suggests, is that the "bad" requests actually >>> make >>> it all the way to tomcat. >>> >> >> correct >> >> i will enable logs that you has pointed out and we will look what i will >> catch >> however i think we have only one chance, as if the solution we has found >> out (connection_timeout + mod_bn) >> will work they will stop exploiting it >> >> thank you very much for all the help and explanations >> i will report to the list new facts once they will attack us again >> >> best regards, >> artur >> > > Ok, but also read this e.g. : > https://www.corero.com/blog/695-going-after-the-people-behind-ddos-attacks.html > > Attempts to bring down a site by DoS attacks is a crime, in most places. > You can report it, while at the same time trying to defend yourself > against it. > > It is also relatively easy, and quite inexpensive in terms of system > resources, to run a small shell script which takes a list every few > seconds of the connections to the port of your webserver, and which IPs > they are coming *from*. > E.g. > First try the netstat command, to see what it lists, like : > # netstat -n --tcp | more > > Then you will want to filter this a bit, to only consider established > connections to your webserver, for example : > # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" > > Then you will want to send this to a logfile, regularly, like this : > > # date >> some_file.log > # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> some_file.log > (repeat every 3 seconds) > > This will not generate GB of logfiles, and it will tell you, when the > problem happens, how many connections there are exactly to your > webserver, and where they are coming from. > Then later you can further analyse this information.. > >> >> >>>> i think that setting connection-timeout and limiting the number of >>>> clients >>>> by mod_bd i will >>>> have effect that next time somebody will try this exploit it will block >>>> his >>>> access to the site >>>> for minute or two, pretty good holistic solution i would say >>>> >>>> still, it seems that there is serious vulnerability somewhere in >>>> apache, >>>> mod_jk or tomcat >>>> i would like to help find it out but need some hints which debug >>>> options >>>> enable to catch the bad guys >>>> when they will try next time >>>> >>>> best regards, >>>> artur >>>> >>>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>> >>>> On 27.11.2016 13:23, Jaaz Portal wrote: >>>>> >>>>> hi Andre, >>>>>> thank you very much this was very educative but in my case it is >>>>>> little >>>>>> bit >>>>>> different. >>>>>> The server is no flooded, there is maybe dozen of very sophisticated >>>>>> connections that somehow hangs apache workers threads >>>>>> >>>>>> >>>>> Can you be a bit more specific ? >>>>> When you say "apache workers threads", do you mean threads in Apache >>>>> httpd, or threads in Apache Tomcat ? (both are Apache webservers, >>>>> so it >>>>> is >>>>> difficult to tell what you are talking about, unless you are very >>>>> precise). >>>>> >>>>> Let me give you some additional explanations, and maybe you can figure >>>>> out >>>>> exactly where it "hangs". >>>>> >>>>> From the Apache httpd front-end point of view, mod_jk (the >>>>> connector to >>>>> Apache Tomcat) is basically one among other "response generators". >>>>> Apache >>>>> httpd does not "know" that behind mod_jk, there is one or more Tomcat >>>>> servers. Apache httpd receives the original client request, and >>>>> depending >>>>> on the URL of the request, it will pass it to mod_jk or to another >>>>> response >>>>> generator, to generate the response to the request. >>>>> That mod_jk in the background is using a Tomcat server to actually >>>>> generate the response, is none of the business of Apache httpd, and it >>>>> does >>>>> not care. All it cares about, is to actually receive the response from >>>>> mod_jk, and pass it back to the client. >>>>> >>>>> If httpd passes a request to mod_jk, it is because you have >>>>> specified in >>>>> the Apache configuration, the type of URL that it should pass to >>>>> mod_jk.. >>>>> That happens via your "JkMount (URL pattern)" directives in Apache >>>>> httpd. >>>>> >>>>> Of course Apache httpd will not pass a request to mod_jk, before it >>>>> has >>>>> received at least the URL of the request, and analysed it to determine >>>>> *if* >>>>> it should pass it to mod_jk (*). >>>>> >>>>> If the mod_jk logging is enabled, you can see in it, exactly *which* >>>>> requests are passed to mod_jk and to Tomcat. >>>>> Do you know *which* requests, from which clients, cause this "thread >>>>> hanging" symptom ? >>>>> Once you would know this, maybe you can design a strategy to block >>>>> specifically these requests. >>>>> >>>>> and the effect is permanent. Quickly the pool is exhausted >>>>> >>>>>> >>>>>> >>>>> which pool exactly ? >>>>> >>>>> and the only >>>>> >>>>> solution that works is to restart tomcat. >>>>>> >>>>>> I think it is a bug similar to this one from mod_proxy: >>>>>> https://tools.cisco.com/security/center/viewAlert.x?alertId=34971 >>>>>> >>>>>> >>>>>> Maybe, maybe not. As long as we do not know what the requests are, >>>>>> that >>>>> block things, we do not know this. >>>>> >>>>> >>>>> I think also that your solution with setting connectionTimeout will >>>>> solve >>>>> >>>>>> the problem, at least partially. THANK YOU. >>>>>> >>>>>> >>>>> Same thing, we do not know this yet. It was only giving this >>>>> explanation, >>>>> to help you think about where the problem may be. >>>>> >>>>> >>>>> I would like to help you further investigate this issue, as our server >>>>>> comes under such attack once or twice in a week. >>>>>> >>>>>> >>>>>> Other than giving you hints, there is not much I or anyone else >>>>>> can do >>>>> to >>>>> help. You are the one with control of your servers and logfiles, so >>>>> you >>>>> have to investigate and provide more precise information. >>>>> >>>>> >>>>> (*) actually, to be precise, Apache httpd passes *all* requests to >>>>> mod_jk, >>>>> to ask it "if it wants that request". mod_jk then accepts or declines, >>>>> depending on the JkMount instructions. If mod_jk declines, then Apache >>>>> httpd will ask the next response generator if it wants this request, >>>>> etc... >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> best regards, >>>>> >>>>>> artur >>>>>> >>>>>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>>>> >>>>>> Hi. >>>>>> >>>>>>> >>>>>>> Have a look that the indicated parameters in the two pages below. >>>>>>> You may be the target of such a variant of DDoS attack : many >>>>>>> clients >>>>>>> open >>>>>>> a TCP connection to your server (front-end), but then never sends a >>>>>>> HTTP >>>>>>> request on that connection. In the meantime, the server accepts the >>>>>>> TCP >>>>>>> connection, and passes it on to a "child" process or thread for >>>>>>> processing. The child then waits for the HTTP request line to >>>>>>> arrive >>>>>>> on >>>>>>> the connection (during a certain time), but it never arrives. >>>>>>> After a >>>>>>> while, this triggers a timeout (see below), but the standard >>>>>>> value of >>>>>>> that >>>>>>> timeout may be such that in the meantime, a lot of other connections >>>>>>> have >>>>>>> been established by other such nefarious clients, so a lot of >>>>>>> resources >>>>>>> of >>>>>>> the webserver are tied up, waiting for something that will never >>>>>>> come.. >>>>>>> Since there is never any real request sent on the connection, you >>>>>>> would >>>>>>> (probably) not see this in the logs either. >>>>>>> >>>>>>> The above is the basic mechanism of such an attack. There may be >>>>>>> variations, such as the client not "not sending" a request line, but >>>>>>> sending it extremely slowly, thus achieving perhaps similar kinds of >>>>>>> effects. >>>>>>> >>>>>>> As someone pointed out, it is quite difficult to do something about >>>>>>> this >>>>>>> at the level of the webserver itself, because by the time you >>>>>>> would do >>>>>>> something about it, the resources have already been consumed and >>>>>>> your >>>>>>> server is probably already overloaded. >>>>>>> There are specialised front-end devices and software available, to >>>>>>> detect >>>>>>> and protect against this kind of attack. >>>>>>> >>>>>>> You may want to have a look at the following parameters, but make >>>>>>> sure >>>>>>> to >>>>>>> read the caveats (side-effects, interlocking timeouts etc.), >>>>>>> otherwise >>>>>>> you >>>>>>> may do more harm than good. >>>>>>> >>>>>>> Another thing : the settings below are for Apache Tomcat, which >>>>>>> in your >>>>>>> case is the back-end. It would of course be much better to detect >>>>>>> and >>>>>>> eliminate this at the front-end, or even before. I had a look at >>>>>>> the >>>>>>> Apache httpd documentation, and could not find a corresponding >>>>>>> parameter. >>>>>>> But I am sure that it must exist. You may want to post this same >>>>>>> question >>>>>>> on the Apache httpd user's list for a better response. >>>>>>> >>>>>>> Tomcat configuration settings : >>>>>>> >>>>>>> AJP Connector : (http://tomcat.apache.org/tomc >>>>>>> at-8.5-doc/config/ajp.html# >>>>>>> Standard_Implementations) >>>>>>> >>>>>>> connectionTimeout >>>>>>> >>>>>>> The number of milliseconds this Connector will wait, after >>>>>>> accepting a >>>>>>> connection, for the request URI line to be presented. The default >>>>>>> value >>>>>>> for >>>>>>> AJP protocol connectors is -1 (i.e. infinite). >>>>>>> >>>>>>> (You could for example try to set this to 3000 (milliseconds) or >>>>>>> even >>>>>>> lower. That should be more than enough for any legitimate client so >>>>>>> send >>>>>>> the HTTP request line. Note however that by doing this at the >>>>>>> Tomcat >>>>>>> level, you will probably move the problem to the Apache httpd/mod_jk >>>>>>> level. But at least it might confirm that this is the problem >>>>>>> that you >>>>>>> are >>>>>>> seeing. The mod_jk logfile at the httpd level may give you some >>>>>>> hints >>>>>>> there.) >>>>>>> >>>>>>> >>>>>>> HTTP Connector : (http://tomcat.apache.org/tomc >>>>>>> at-8.5-doc/config/http.html#Standard_Implementation) >>>>>>> >>>>>>> connectionTimeout >>>>>>> >>>>>>> The number of milliseconds this Connector will wait, after >>>>>>> accepting a >>>>>>> connection, for the request URI line to be presented. Use a value >>>>>>> of -1 >>>>>>> to >>>>>>> indicate no (i.e. infinite) timeout. The default value is 60000 >>>>>>> (i.e. >>>>>>> 60 >>>>>>> seconds) but note that the standard server.xml that ships with >>>>>>> Tomcat >>>>>>> sets >>>>>>> this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is >>>>>>> set to >>>>>>> false, this timeout will also be used when reading the request >>>>>>> body (if >>>>>>> any). >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 26.11.2016 09:57, Jaaz Portal wrote: >>>>>>> >>>>>>> hi, >>>>>>> >>>>>>>> sorry, its mod_jk no jk2, my typo. All at latest versions. We tried >>>>>>>> with >>>>>>>> mod proxy too. >>>>>>>> >>>>>>>> There is no flood of the server. Nobody is flooding us, they use >>>>>>>> some >>>>>>>> specific connections after which pool of apache workers is >>>>>>>> exhausted >>>>>>>> and >>>>>>>> blocked >>>>>>>> and we need to restart tomcat server. >>>>>>>> It is some kind of exploit but do not know how to log it to obtain >>>>>>>> details. >>>>>>>> >>>>>>>> i had put a limit on connections per client with hope that this >>>>>>>> will >>>>>>>> help >>>>>>>> but once again, it is not a flood. >>>>>>>> They open several connections that are not dropped by apache >>>>>>>> when they >>>>>>>> disconnect. This way whole pool is quickly exhausted and the server >>>>>>>> broken. >>>>>>>> >>>>>>>> i would like to help you to figure details of this attack but >>>>>>>> this is >>>>>>>> production server so it is impossible to much debugging options >>>>>>>> >>>>>>>> best, >>>>>>>> artur >>>>>>>> >>>>>>>> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu < >>>>>>>> niranjan.bo...@gmail.com >>>>>>>> >>>>>>>> : >>>>>>>>> >>>>>>>>> >>>>>>>> you can find who is flooding site in apache access.log and block >>>>>>>> them >>>>>>>> in >>>>>>>> >>>>>>>> firewall. >>>>>>>>> >>>>>>>>> ex to find the IP: >>>>>>>>> >>>>>>>>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq -c|sort >>>>>>>>> -gr >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal >>>>>>>>> <jaazpor...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> hi, >>>>>>>>> >>>>>>>>> we are from some weeks struggling with some Polish hackers that >>>>>>>>> are >>>>>>>>>> bringing our server down. After updating apache to latest version >>>>>>>>>> >>>>>>>>>> (2.4.23) >>>>>>>>>> >>>>>>>>> >>>>>>>>> and tomcat (8.0.38) available for debian systems we still cannot >>>>>>>>> secure >>>>>>>>> >>>>>>>>>> >>>>>>>>>> our >>>>>>>>>> >>>>>>>>> >>>>>>>>> server. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Today it has stopped to respond again and we needed to restart >>>>>>>>>> tomcat >>>>>>>>>> process to get it back alive. >>>>>>>>>> >>>>>>>>>> There is no too much clues in the logs. The apache error.log >>>>>>>>>> gives >>>>>>>>>> just >>>>>>>>>> this line: >>>>>>>>>> >>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid >>>>>>>>>> 13385:tid >>>>>>>>>> 1397934896385 >>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, consider >>>>>>>>>> raising >>>>>>>>>> >>>>>>>>>> the >>>>>>>>>> >>>>>>>>> >>>>>>>>> MaxR >>>>>>>>> >>>>>>>>>> equestWorkers setting >>>>>>>>>> >>>>>>>>>> seems that somehow tomcat, mod-jk2 or even apache is >>>>>>>>>> vulnerable to >>>>>>>>>> some >>>>>>>>>> >>>>>>>>>> new >>>>>>>>>> >>>>>>>>> >>>>>>>>> exploit, as we certainly does not have such traffic that would >>>>>>>>> block >>>>>>>>> >>>>>>>>>> our >>>>>>>>>> server otherwise >>>>>>>>>> >>>>>>>>>> for now we have increased MaxRequestWorkers and we have limited >>>>>>>>>> number >>>>>>>>>> of >>>>>>>>>> connections from one client to 5 by mod_bw and limited number of >>>>>>>>>> simultaneous connections from one ip by iptables but does not >>>>>>>>>> know >>>>>>>>>> if >>>>>>>>>> >>>>>>>>>> this >>>>>>>>>> >>>>>>>>> >>>>>>>>> will help >>>>>>>>> >>>>>>>>>> >>>>>>>>>> best regards, >>>>>>>>>> artur >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Thanks* >>>>>>>>> *Niranjan* >>>>>>>>>
This sounds like a variant of the slowloris attack. This type of attack doesn't take a large number of clients or consume a large amount of bandwidth. Basically, the maximum number of connections are made to the server, and just enough data is sent to each connection in order to not trigger the timeout. André has explained this in more detail earlier in the thread. Search for "slowloris attack" for more information. There are several ways of mitigating this type of attack. As André has mentioned, placing a dedicated device in front of your systems is often the best way. Lots of benefits (platform neutral, no stress on your current servers), and some issues (cost, placement / access may be an issue with hosted solutions). However, there are Apache HTTPD modules that can help mitigate these types of attacks. Some of them are: mod_reqtimeout (should be included by default in your Apache HTRPD 2.4) mod_qos (quality of service module) mod_security (application firewall with lots of security rules) Do a quick search on "slowloris attack apache httpd 2.4" to get some ideas. All of them will probably place additional load on your Apache HTTPD server, so make sure that the platform is robust enough to manage the additional load. There is also a beta version of the mod_security module written as a servlet filter. It should be possible to build this and configure the filter in Tomcat's default web.xml ($CATALINA_BASE/conf/web.xml). I've not tried this. Also, the code base hasn't seen any activity for 3 years. Do a quick search on "modsecurity servlet filter" to find out more about the servlet filter version of mod_security. In short, there appear to be some ways to mitigate the attack. . . . just my two cents /mde/
signature.asc
Description: OpenPGP digital signature
