Artur, On 11/30/2016 8:36 AM, Jaaz Portal wrote: > hi, > they has tried again with success despite setting connection_timeout and > limiting number of clients by mod_bw > the tomcat has frozen again. > > netstat does not showed any connections on port 80 but plenty of > connections from apache to localhost:8009 > so it was not an attack that you has described (no slowlaris) > > im looking into debug files of mod_jk and forensic for some hints. If you > want i can share them (they have 4mb compressed) > > best wishes > artur
This is beginning to look like an application or a configuration issue and not a DOS (or DDOS) attack. One the issues that may cause this is a mismatched timeout value between connection_pool_timeout in workers.properties (mod_jk) and the connectionTimeout in server.xml (Tomcat) for the AJP connector. Also, at least for the mod_jk version that I'm running, there is no limit for reply_timeout (mod_jk) by default. Can you post your workers.properties file and the AJP connector portion of your server.xml? In the conf directory of the mod_jk source code, there is a very nice workers.properties file that has sensible defaults. If you've not done so, I suggest that you start with the values specified in that file, and make sure that the timeout values match (see my comment above). Also, when you used mod-proxy, did you use mod-proxy-ajp or mod-proxy-http? If you used mod-proxy-ajp, then again there could be a timeout mismatch (or no timeout specified at all). . . . just my two cents /mde/ > > 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: > >> On 28.11.2016 22:04, Jaaz Portal wrote: >> >>> hi Andre, >>> you are wrong. This vulnerability is not only causing memory leaks, it >>> makes also apache workers to hang >>> >> >> Maybe for the last time here : >> >> - what do you call "apache workers" ? >> >> , making it easy to exhaust the pool. >> >> - what do you call "the pool" ? >> >> what i have in my log files. But it is true also that such exhaustion can >>> be made by other forms of dos attacks described in this thread. >>> >>> regarding you suggestion on our application, it does not dos bind server >>> nether does not scan for various vulnerabilities in apache, what i have >>> also in the logs >>> >> >> For your information : I run about 25 Internet-facing Apache webservers >> (some with a back-end tomcat, some not). >> On every single one of those webservers, there are *hundreds* of such >> "scans" every day, as shown by the Apache access logs. That is just a fact >> of life on the Internet. >> They are annoying, but most of them are harmless (from an "attack" point >> of view), because they are scanning for things that we do not run >> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and >> thus are responded to by Apache as "404 Not found". >> What is annoying with those scans, is >> a) that they fill the logfile, and thus make it more difficult to find >> really significant things >> b) that each of those requires some bandwidth and system resource, if only >> to return a "404 Not found" (or a "401 Unauthorised"), and that we pay for >> that bandwidth and resources. >> >> If I could find a way to charge 0.1 cent per access to my servers, from >> the people who wrote or run the programs who are doing this, I could retire >> in luxury. >> >> But they are not a real problem, because they are caught as "invalid" by >> Apache, and rejected quickly, so they cannot do anything really nasty >> (except if they were sending several thousand such requests per second to >> one of my servers for a long time). >> >> The ones that are worrying, are the ones >> - a) which do /not/ end up as a "404 Not found", because they have found >> an application which responds, and they are not coming from our legitimate >> customers >> - b) /the ones which we do not see/, because they either do not send a >> valid HTTP request, or they have found a way to trigger one of our >> applications, in such a way that the application misbehaves and, perhaps, >> even if they do not crash our servers, they may provide the attacker with >> some entry point to do other things which we do not know and do not control >> >> What I am trying to say here, is /do not jump to premature conclusions/. >> Such "scans" as you mention, happen to everyone, all the time, from >> ever-changing IP addresses located all over the world. Some of those >> "scans" may come from the infected PC of your grandmother, and she does not >> even know about it. >> >> There is no guarantee, and no indication or proof so far, that /these/ >> scans are even related to "the other thing" which happens on your >> webserver, which looked much more focused. >> >> So do not just bundle them together as being the same thing, until you >> have some real data that shows for example that these different things all >> come from the same IP addresses. >> >> And one more thing, also finally until you come back with some real data : >> I am not saying that your application "scans your server". What I am >> saying is that, maybe, by chance or by design, the attackers have found a >> URL which goes to your application, and which causes your application to >> keep tomcat and/or Apache busy for a long time. >> And that maybe /that/ is the problem you should be looking for, and not >> some hypothetical bug in Apache httpd or tomcat. >> >> >> >>> kindly regards, >>> artur >>> >>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>> >>> On 28.11.2016 20:34, Jaaz Portal wrote: >>>> >>>> hi mark, >>>>> yes, i understand now what slowloris attack is. >>>>> maybe it was this maybe *this one based on * * mod_proxy denial of >>>>> service * >>>>> CVE-2014-0117 <http://cve.mitre.org/cgi-bin/ >>>>> cvename.cgi?name=CVE-2014-0117> >>>>> >>>>> >>>> You keep on saying this, but the description of that vulnerability of >>>> *Apache httpd*, and the symptoms which you described, *do not match*. >>>> You described the problem as ocurring in Apache tomcat, which in your >>>> case >>>> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat >>>> cured the problem. >>>> >>>> The CVE above applies to Apache httpd, and describes how an attacker >>>> could >>>> attack Apache httpd and cause *its children* processes to crash (the >>>> children processes of Apache httpd), by leading them to consume a lot of >>>> memory and crash with an out-of-memory error. >>>> Granted, the problem occurred in the mod_proxy module of Apache httpd; >>>> but >>>> it was httpd which crashed, not tomcat. >>>> And tomcat processes are not "Apache httpd children processes" in any >>>> understanding of the term. >>>> >>>> As far as I remember, you never mentioned Apache httpd crashing. You >>>> mentioned "the pool" getting full or satured or something like that, >>>> without ever describing properly what you meant by "the pool". >>>> >>>> As far as I am concerned, according to all the relatively unspecific >>>> information which you have previously provided : >>>> 1) the attack - if that is what it is/was - is definitely NOT related to >>>> the CVE which you have repeatedly mentioned >>>> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed at >>>> your front-end Apache. Instead, it seems that the requests are properly >>>> received by Apache, properly decoded by Apache, and then whatever Apache >>>> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) is >>>> properly forwarding these requests to a back-end tomcat; and it is at the >>>> level of that back-end tomcat that the requests never seem to end, and in >>>> the end paralyse your tomcat server (and later on maybe your Apache httpd >>>> server too, because it is also waiting for tomcat to respond). >>>> >>>> So your very way of describing the problem, in terms of "first we used >>>> this proxy module, and then they exploited the vulnerability so and so; >>>> then we changed the proxy module, and they exploited that too; etc.." >>>> seems to not have anything to do with the problem per se, and (I believe) >>>> confuses everyone, including yourself. >>>> >>>> It is not that "they" exploited different vulnerabilities of various >>>> httpd >>>> proxy modules, one after the other. Each of these proxy modules was doing >>>> its job properly, and forwarding valid HTTP requests properly to tomcat. >>>> When you changed from one proxy module to another, you did not really >>>> change anything in that respect, because any proxy module would do the >>>> same. >>>> >>>> But in all cases, what did not change, was the tomcat behind the >>>> front-end, and the application running on that tomcat. So the presumed >>>> attackers did not have to change anything, they just kept on sending the >>>> same requests, because they were really targeting your back-end tomcat or >>>> the tomcat application in it, no matter /how/ you were forwarding >>>> requests >>>> from Apache httpd to tomcat. >>>> >>>> So either it is tomcat itself, which has a problem with some request URLs >>>> which do not bother Apache httpd (possible, but statistically unlikely), >>>> or >>>> it is the application which runs in tomcat that has such a problem >>>> (statistically, much more likely). >>>> >>>> we do not know yet >>>> >>>>> >>>>> we have setup more logging and are waiting for them to attack once again >>>>> >>>>> >>>> Yes, that is the right thing to do. Before deciding what the problem may >>>> be, and what you can do about it, the first thing you need is *data*. >>>> You >>>> need to know >>>> - which request URL(s?) cause that problem >>>> - which IPs these requests come from (always the same ? multiple IPs that >>>> change all the time ? how many ? can these IPs be valid/expected clients >>>> or >>>> not ? do these IPs look like some "coordinated group" ?) >>>> - how many such requests there may be during some period of time (10, >>>> 100, >>>> 1000, more ?) >>>> - if these URLs result in passing the request to tomcat >>>> - what tomcat application (if any) they are directed to >>>> - if so, when that application receives such a request, what is it >>>> supposed to do ? does it do it properly ? how long does it need, to >>>> respond >>>> to such a request ? >>>> >>>> You also need to ask yourself a question : is the application which you >>>> run inside tomcat something that you designed yourself (and which hackers >>>> are unlikely to know well-enough to find such a URL which paralyses your >>>> server) ? or is it some well-known third-party java application which you >>>> are running (and for which would-be attackers would be much more likely >>>> to >>>> know exactly such a bug) ? >>>> >>>> >>>> anyway, thank you for all informations, it was very useful and >>>>> educational >>>>> reading for all of us >>>>> >>>>> best wishes, >>>>> artur >>>>> >>>>> 2016-11-28 19:46 GMT+01:00 Mark Eggers <its_toas...@yahoo.com.invalid>: >>>>> >>>>> Jaaz, >>>>> >>>>>> >>>>>> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote: >>>>>> >>>>>> On 27.11.2016 19:03, Jaaz Portal wrote: >>>>>>> >>>>>>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>>>>>> >>>>>>>> On 27.11.2016 14:26, Jaaz Portal wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> hi, >>>>>>>>> >>>>>>>>>> everything i know so far is just this single log line that appeared >>>>>>>>>> in >>>>>>>>>> apache error.log >>>>>>>>>> >>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid >>>>>>>>>> 1397934896385 >>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, consider >>>>>>>>>> >>>>>>>>>> raising >>>>>>>>> >>>>>>>> >>>>>> the >>>>>>> >>>>>>>> MaxR >>>>>>>>>> equestWorkers setting >>>>>>>>>> >>>>>>>>>> there was nothing else, just this strange line >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is not a "strange" line. It is telling you something. >>>>>>>>> One problem is that you seem convinced in advance, without serious >>>>>>>>> proof, >>>>>>>>> that there is a "bug" or a vulnerability in httpd or tomcat. >>>>>>>>> Read the explanation of the httpd parameter, here : >>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequ >>>>>>>>> estworkers >>>>>>>>> and try to understand it. >>>>>>>>> >>>>>>>>> >>>>>>>>> I understand it very well. We are serving no more that 500 clients >>>>>>>> per >>>>>>>> day >>>>>>>> so there was no other option that some kind of attack. >>>>>>>> >>>>>>>> >>>>>>>> About the "bug" or "vulnerability" : a webserver is supposed to serve >>>>>>>> HTTP >>>>>>>> >>>>>>>> requests from clients. That is what you expect of it. It has no >>>>>>>>> choice but >>>>>>>>> to accept any client connection and request, up to the maximum it >>>>>>>>> can >>>>>>>>> handle considering its configuration and the capacity of the system >>>>>>>>> on >>>>>>>>> which it runs. That is not a bug, it is a feature. >>>>>>>>> >>>>>>>>> >>>>>>>>> We have some weeks ago come under attack from some Polish group. It >>>>>>>>> >>>>>>>> was >>>>>>>> first bind that was DoS'ed, we was running on stable Debian so i >>>>>>>> updated >>>>>>>> bind to latest version. It did not helped. They has dos'ed it so we >>>>>>>> switched to other dns provider. That has helped. >>>>>>>> >>>>>>>> Then they exploited some well know vulnerability in mod_proxy. We >>>>>>>> have >>>>>>>> updated apache to the latest but again they has exploited it, so we >>>>>>>> have >>>>>>>> switched to mod_jk. And then guess what. They exploited it too so i >>>>>>>> decided >>>>>>>> to write to this list looking for help before trying jetty. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> The normal Apache httpd access log, will log a request only when it >>>>>>>>> is >>>>>>>>> finished. If the request never finishes, it will not get logged. >>>>>>>>> That may be why you do not see these requests in the log. >>>>>>>>> But have a look at this Apache httpd module : >>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html >>>>>>>>> >>>>>>>>> >>>>>>>>> ok, thanks, will take care >>>>>>>> >>>>>>>> Note : that is also why I was telling you to enable the mod_jk log, >>>>>>>> and to >>>>>>>> >>>>>>>> examine it. >>>>>>>>> Because mod_jk will also log information before the request >>>>>>>>> produces a >>>>>>>>> response. >>>>>>>>> >>>>>>>>> >>>>>>>>> and server hanged. >>>>>>>>> >>>>>>>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ? >>>>>>>>> >>>>>>>>> >>>>>>>>> Apache was accepting connection but not processed it. After restart >>>>>>>> of >>>>>>>> tomcat server it worked again. >>>>>>>> >>>>>>>> >>>>>>>> Also in >>>>>>>> >>>>>>>>> >>>>>>>>> access logs there are no clues that it was under any heavy load. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> after around hour after discovering that our server hanged-out we >>>>>>>>>> have >>>>>>>>>> restarted tomcat server >>>>>>>>>> and it worked again >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Yes, because that will close all connections between Apache httpd >>>>>>>>> and >>>>>>>>> tomcat, and abort all requests that are in the process of being >>>>>>>>> processed >>>>>>>>> by tomcat. So mod_jk will get an error from tomcat, and will report >>>>>>>>> an >>>>>>>>> error to httpd, and httpd will communicate that error to the >>>>>>>>> clients, >>>>>>>>> and >>>>>>>>> close their connection. >>>>>>>>> It still does not tell you what the problem was. >>>>>>>>> The only thing that it suggests, is that the "bad" requests actually >>>>>>>>> make >>>>>>>>> it all the way to tomcat. >>>>>>>>> >>>>>>>>> >>>>>>>>> correct >>>>>>>> >>>>>>>> i will enable logs that you has pointed out and we will look what i >>>>>>>> will >>>>>>>> catch >>>>>>>> however i think we have only one chance, as if the solution we has >>>>>>>> found >>>>>>>> out (connection_timeout + mod_bn) >>>>>>>> will work they will stop exploiting it >>>>>>>> >>>>>>>> thank you very much for all the help and explanations >>>>>>>> i will report to the list new facts once they will attack us again >>>>>>>> >>>>>>>> best regards, >>>>>>>> artur >>>>>>>> >>>>>>>> >>>>>>>> Ok, but also read this e.g. : >>>>>>> https://www.corero.com/blog/695-going-after-the-people- >>>>>>> >>>>>>> behind-ddos-attacks.html >>>>>> >>>>>> >>>>>>> Attempts to bring down a site by DoS attacks is a crime, in most >>>>>>> places.. >>>>>>> >>>>>>> You can report it, while at the same time trying to defend yourself >>>>>>> against it. >>>>>>> >>>>>>> It is also relatively easy, and quite inexpensive in terms of system >>>>>>> resources, to run a small shell script which takes a list every few >>>>>>> seconds of the connections to the port of your webserver, and which >>>>>>> IPs >>>>>>> they are coming *from*. >>>>>>> E.g. >>>>>>> First try the netstat command, to see what it lists, like : >>>>>>> # netstat -n --tcp | more >>>>>>> >>>>>>> Then you will want to filter this a bit, to only consider established >>>>>>> connections to your webserver, for example : >>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >>>>>>> >>>>>>> Then you will want to send this to a logfile, regularly, like this : >>>>>>> >>>>>>> # date >> some_file.log >>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> some_file.log >>>>>>> (repeat every 3 seconds) >>>>>>> >>>>>>> This will not generate GB of logfiles, and it will tell you, when the >>>>>>> problem happens, how many connections there are exactly to your >>>>>>> webserver, and where they are coming from. >>>>>>> Then later you can further analyse this information.. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> i think that setting connection-timeout and limiting the number of >>>>>>>> >>>>>>>>> clients >>>>>>>>>> by mod_bd i will >>>>>>>>>> have effect that next time somebody will try this exploit it will >>>>>>>>>> >>>>>>>>>> block >>>>>>>>> >>>>>>>> >>>>>> his >>>>>>> >>>>>>>> access to the site >>>>>>>>>> for minute or two, pretty good holistic solution i would say >>>>>>>>>> >>>>>>>>>> still, it seems that there is serious vulnerability somewhere in >>>>>>>>>> apache, >>>>>>>>>> mod_jk or tomcat >>>>>>>>>> i would like to help find it out but need some hints which debug >>>>>>>>>> options >>>>>>>>>> enable to catch the bad guys >>>>>>>>>> when they will try next time >>>>>>>>>> >>>>>>>>>> best regards, >>>>>>>>>> artur >>>>>>>>>> >>>>>>>>>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>>>>>>>> >>>>>>>>>> On 27.11.2016 13:23, Jaaz Portal wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> hi Andre, >>>>>>>>>>> >>>>>>>>>>> thank you very much this was very educative but in my case it is >>>>>>>>>>>> little >>>>>>>>>>>> bit >>>>>>>>>>>> different. >>>>>>>>>>>> The server is no flooded, there is maybe dozen of very >>>>>>>>>>>> sophisticated >>>>>>>>>>>> connections that somehow hangs apache workers threads >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Can you be a bit more specific ? >>>>>>>>>>>> >>>>>>>>>>> When you say "apache workers threads", do you mean threads in >>>>>>>>>>> Apache >>>>>>>>>>> httpd, or threads in Apache Tomcat ? (both are Apache webservers, >>>>>>>>>>> so it >>>>>>>>>>> is >>>>>>>>>>> difficult to tell what you are talking about, unless you are very >>>>>>>>>>> precise). >>>>>>>>>>> >>>>>>>>>>> Let me give you some additional explanations, and maybe you can >>>>>>>>>>> >>>>>>>>>>> figure >>>>>>>>>> >>>>>>>>> >>>>>> out >>>>>>> >>>>>>>> exactly where it "hangs". >>>>>>>>>>> >>>>>>>>>>> From the Apache httpd front-end point of view, mod_jk (the >>>>>>>>>>> connector to >>>>>>>>>>> Apache Tomcat) is basically one among other "response generators". >>>>>>>>>>> Apache >>>>>>>>>>> httpd does not "know" that behind mod_jk, there is one or more >>>>>>>>>>> Tomcat >>>>>>>>>>> servers. Apache httpd receives the original client request, and >>>>>>>>>>> depending >>>>>>>>>>> on the URL of the request, it will pass it to mod_jk or to another >>>>>>>>>>> response >>>>>>>>>>> generator, to generate the response to the request. >>>>>>>>>>> That mod_jk in the background is using a Tomcat server to actually >>>>>>>>>>> generate the response, is none of the business of Apache httpd, >>>>>>>>>>> and >>>>>>>>>>> >>>>>>>>>>> it >>>>>>>>>> >>>>>>>>> >>>>>> does >>>>>>> >>>>>>>> not care. All it cares about, is to actually receive the response >>>>>>>>>>> >>>>>>>>>>> from >>>>>>>>>> >>>>>>>>> >>>>>> mod_jk, and pass it back to the client. >>>>>>> >>>>>>>> >>>>>>>>>>> If httpd passes a request to mod_jk, it is because you have >>>>>>>>>>> specified in >>>>>>>>>>> the Apache configuration, the type of URL that it should pass to >>>>>>>>>>> mod_jk.. >>>>>>>>>>> That happens via your "JkMount (URL pattern)" directives in Apache >>>>>>>>>>> httpd. >>>>>>>>>>> >>>>>>>>>>> Of course Apache httpd will not pass a request to mod_jk, before >>>>>>>>>>> it >>>>>>>>>>> has >>>>>>>>>>> received at least the URL of the request, and analysed it to >>>>>>>>>>> >>>>>>>>>>> determine >>>>>>>>>> >>>>>>>>> >>>>>> *if* >>>>>>> >>>>>>>> it should pass it to mod_jk (*). >>>>>>>>>>> >>>>>>>>>>> If the mod_jk logging is enabled, you can see in it, exactly >>>>>>>>>>> *which* >>>>>>>>>>> requests are passed to mod_jk and to Tomcat. >>>>>>>>>>> Do you know *which* requests, from which clients, cause this >>>>>>>>>>> "thread >>>>>>>>>>> hanging" symptom ? >>>>>>>>>>> Once you would know this, maybe you can design a strategy to block >>>>>>>>>>> specifically these requests. >>>>>>>>>>> >>>>>>>>>>> and the effect is permanent. Quickly the pool is exhausted >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> which pool exactly ? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> and the only >>>>>>>>>>> >>>>>>>>>>> solution that works is to restart tomcat. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> I think it is a bug similar to this one from mod_proxy: >>>>>>>>>>>> https://tools.cisco.com/security/center/viewAlert.x?alertId= >>>>>>>>>>>> 34971 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Maybe, maybe not. As long as we do not know what the requests >>>>>>>>>>>> are, >>>>>>>>>>>> that >>>>>>>>>>>> >>>>>>>>>>>> block things, we do not know this. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I think also that your solution with setting connectionTimeout >>>>>>>>>>> will >>>>>>>>>>> solve >>>>>>>>>>> >>>>>>>>>>> the problem, at least partially. THANK YOU. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Same thing, we do not know this yet. It was only giving this >>>>>>>>>>>> >>>>>>>>>>> explanation, >>>>>>>>>>> to help you think about where the problem may be. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I would like to help you further investigate this issue, as our >>>>>>>>>>> >>>>>>>>>>> server >>>>>>>>>> >>>>>>>>> >>>>>> comes under such attack once or twice in a week. >>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Other than giving you hints, there is not much I or anyone else >>>>>>>>>>>> can do >>>>>>>>>>>> >>>>>>>>>>>> to >>>>>>>>>>> help. You are the one with control of your servers and logfiles, >>>>>>>>>>> so >>>>>>>>>>> you >>>>>>>>>>> have to investigate and provide more precise information. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> (*) actually, to be precise, Apache httpd passes *all* requests to >>>>>>>>>>> mod_jk, >>>>>>>>>>> to ask it "if it wants that request". mod_jk then accepts or >>>>>>>>>>> >>>>>>>>>>> declines, >>>>>>>>>> >>>>>>>>> >>>>>> depending on the JkMount instructions. If mod_jk declines, then >>>>>>> >>>>>>>> >>>>>>>>>>> Apache >>>>>>>>>> >>>>>>>>> >>>>>> httpd will ask the next response generator if it wants this request, >>>>>>> >>>>>>>> etc... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> best regards, >>>>>>>>>>> >>>>>>>>>>> artur >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com >>>>>>>>>>>>> : >>>>>>>>>>>> >>>>>>>>>>>> Hi. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Have a look that the indicated parameters in the two pages >>>>>>>>>>>>> below.. >>>>>>>>>>>>> >>>>>>>>>>>>> You may be the target of such a variant of DDoS attack : many >>>>>>>>>>>>> clients >>>>>>>>>>>>> open >>>>>>>>>>>>> a TCP connection to your server (front-end), but then never >>>>>>>>>>>>> sends >>>>>>>>>>>>> a >>>>>>>>>>>>> HTTP >>>>>>>>>>>>> request on that connection. In the meantime, the server accepts >>>>>>>>>>>>> >>>>>>>>>>>>> the >>>>>>>>>>>> >>>>>>>>>>> >>>>>> TCP >>>>>>> >>>>>>>> connection, and passes it on to a "child" process or thread for >>>>>>>>>>>>> processing. The child then waits for the HTTP request line to >>>>>>>>>>>>> arrive >>>>>>>>>>>>> on >>>>>>>>>>>>> the connection (during a certain time), but it never arrives. >>>>>>>>>>>>> After a >>>>>>>>>>>>> while, this triggers a timeout (see below), but the standard >>>>>>>>>>>>> value of >>>>>>>>>>>>> that >>>>>>>>>>>>> timeout may be such that in the meantime, a lot of other >>>>>>>>>>>>> >>>>>>>>>>>>> connections >>>>>>>>>>>> >>>>>>>>>>> >>>>>> have >>>>>>> >>>>>>>> been established by other such nefarious clients, so a lot of >>>>>>>>>>>>> resources >>>>>>>>>>>>> of >>>>>>>>>>>>> the webserver are tied up, waiting for something that will never >>>>>>>>>>>>> come.. >>>>>>>>>>>>> Since there is never any real request sent on the connection, >>>>>>>>>>>>> you >>>>>>>>>>>>> would >>>>>>>>>>>>> (probably) not see this in the logs either. >>>>>>>>>>>>> >>>>>>>>>>>>> The above is the basic mechanism of such an attack. There may >>>>>>>>>>>>> be >>>>>>>>>>>>> variations, such as the client not "not sending" a request line, >>>>>>>>>>>>> >>>>>>>>>>>>> but >>>>>>>>>>>> >>>>>>>>>>> >>>>>> sending it extremely slowly, thus achieving perhaps similar kinds >>>>>>> >>>>>>>> >>>>>>>>>>>>> of >>>>>>>>>>>> >>>>>>>>>>> >>>>>> effects. >>>>>>> >>>>>>>> >>>>>>>>>>>>> As someone pointed out, it is quite difficult to do something >>>>>>>>>>>>> about >>>>>>>>>>>>> this >>>>>>>>>>>>> at the level of the webserver itself, because by the time you >>>>>>>>>>>>> would do >>>>>>>>>>>>> something about it, the resources have already been consumed and >>>>>>>>>>>>> your >>>>>>>>>>>>> server is probably already overloaded. >>>>>>>>>>>>> There are specialised front-end devices and software available, >>>>>>>>>>>>> to >>>>>>>>>>>>> detect >>>>>>>>>>>>> and protect against this kind of attack. >>>>>>>>>>>>> >>>>>>>>>>>>> You may want to have a look at the following parameters, but >>>>>>>>>>>>> make >>>>>>>>>>>>> sure >>>>>>>>>>>>> to >>>>>>>>>>>>> read the caveats (side-effects, interlocking timeouts etc.), >>>>>>>>>>>>> otherwise >>>>>>>>>>>>> you >>>>>>>>>>>>> may do more harm than good. >>>>>>>>>>>>> >>>>>>>>>>>>> Another thing : the settings below are for Apache Tomcat, which >>>>>>>>>>>>> in your >>>>>>>>>>>>> case is the back-end. It would of course be much better to >>>>>>>>>>>>> detect >>>>>>>>>>>>> and >>>>>>>>>>>>> eliminate this at the front-end, or even before. I had a look >>>>>>>>>>>>> at >>>>>>>>>>>>> the >>>>>>>>>>>>> Apache httpd documentation, and could not find a corresponding >>>>>>>>>>>>> parameter. >>>>>>>>>>>>> But I am sure that it must exist. You may want to post this same >>>>>>>>>>>>> question >>>>>>>>>>>>> on the Apache httpd user's list for a better response. >>>>>>>>>>>>> >>>>>>>>>>>>> Tomcat configuration settings : >>>>>>>>>>>>> >>>>>>>>>>>>> AJP Connector : (http://tomcat.apache.org/tomc >>>>>>>>>>>>> at-8.5-doc/config/ajp.html# >>>>>>>>>>>>> Standard_Implementations) >>>>>>>>>>>>> >>>>>>>>>>>>> connectionTimeout >>>>>>>>>>>>> >>>>>>>>>>>>> The number of milliseconds this Connector will wait, after >>>>>>>>>>>>> accepting a >>>>>>>>>>>>> connection, for the request URI line to be presented. The >>>>>>>>>>>>> default >>>>>>>>>>>>> value >>>>>>>>>>>>> for >>>>>>>>>>>>> AJP protocol connectors is -1 (i.e. infinite). >>>>>>>>>>>>> >>>>>>>>>>>>> (You could for example try to set this to 3000 (milliseconds) or >>>>>>>>>>>>> even >>>>>>>>>>>>> lower. That should be more than enough for any legitimate client >>>>>>>>>>>>> so >>>>>>>>>>>>> send >>>>>>>>>>>>> the HTTP request line. Note however that by doing this at the >>>>>>>>>>>>> Tomcat >>>>>>>>>>>>> level, you will probably move the problem to the Apache >>>>>>>>>>>>> >>>>>>>>>>>>> httpd/mod_jk >>>>>>>>>>>> >>>>>>>>>>> >>>>>> level. But at least it might confirm that this is the problem >>>>>>> >>>>>>>> that you >>>>>>>>>>>>> are >>>>>>>>>>>>> seeing. The mod_jk logfile at the httpd level may give you some >>>>>>>>>>>>> hints >>>>>>>>>>>>> there.) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> HTTP Connector : (http://tomcat.apache.org/tomc >>>>>>>>>>>>> at-8.5-doc/config/http.html#Standard_Implementation) >>>>>>>>>>>>> >>>>>>>>>>>>> connectionTimeout >>>>>>>>>>>>> >>>>>>>>>>>>> The number of milliseconds this Connector will wait, after >>>>>>>>>>>>> accepting a >>>>>>>>>>>>> connection, for the request URI line to be presented. Use a >>>>>>>>>>>>> value >>>>>>>>>>>>> of -1 >>>>>>>>>>>>> to >>>>>>>>>>>>> indicate no (i.e. infinite) timeout. The default value is 60000 >>>>>>>>>>>>> (i.e. >>>>>>>>>>>>> 60 >>>>>>>>>>>>> seconds) but note that the standard server.xml that ships with >>>>>>>>>>>>> Tomcat >>>>>>>>>>>>> sets >>>>>>>>>>>>> this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is >>>>>>>>>>>>> set to >>>>>>>>>>>>> false, this timeout will also be used when reading the request >>>>>>>>>>>>> body (if >>>>>>>>>>>>> any). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 26.11.2016 09:57, Jaaz Portal wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> hi, >>>>>>>>>>>>> >>>>>>>>>>>>> sorry, its mod_jk no jk2, my typo. All at latest versions. We >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> tried >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>> with >>>>>>> >>>>>>>> mod proxy too. >>>>>>>>>>>>>> >>>>>>>>>>>>>> There is no flood of the server. Nobody is flooding us, they >>>>>>>>>>>>>> use >>>>>>>>>>>>>> some >>>>>>>>>>>>>> specific connections after which pool of apache workers is >>>>>>>>>>>>>> exhausted >>>>>>>>>>>>>> and >>>>>>>>>>>>>> blocked >>>>>>>>>>>>>> and we need to restart tomcat server. >>>>>>>>>>>>>> It is some kind of exploit but do not know how to log it to >>>>>>>>>>>>>> obtain >>>>>>>>>>>>>> details. >>>>>>>>>>>>>> >>>>>>>>>>>>>> i had put a limit on connections per client with hope that this >>>>>>>>>>>>>> will >>>>>>>>>>>>>> help >>>>>>>>>>>>>> but once again, it is not a flood. >>>>>>>>>>>>>> They open several connections that are not dropped by apache >>>>>>>>>>>>>> when they >>>>>>>>>>>>>> disconnect. This way whole pool is quickly exhausted and the >>>>>>>>>>>>>> >>>>>>>>>>>>>> server >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>> broken. >>>>>>> >>>>>>>> >>>>>>>>>>>>>> i would like to help you to figure details of this attack but >>>>>>>>>>>>>> this is >>>>>>>>>>>>>> production server so it is impossible to much debugging options >>>>>>>>>>>>>> >>>>>>>>>>>>>> best, >>>>>>>>>>>>>> artur >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu < >>>>>>>>>>>>>> niranjan.bo...@gmail.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> : >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> you can find who is flooding site in apache access.log and >>>>>>>>>>>>>>> block >>>>>>>>>>>>>>> >>>>>>>>>>>>>> them >>>>>>>>>>>>>> in >>>>>>>>>>>>>> >>>>>>>>>>>>>> firewall. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> ex to find the IP: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -c|sort >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>> -gr >>>>>>> >>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal >>>>>>>>>>>>>>> <jaazpor...@gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> we are from some weeks struggling with some Polish hackers >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>> are >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> bringing our server down. After updating apache to latest >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> version >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>> (2.4.23) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> and tomcat (8.0.38) available for debian systems we still >>>>>>>>>>>>>>> cannot >>>>>>>>>>>>>>> secure >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> server. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Today it has stopped to respond again and we needed to restart >>>>>>>>>>>>>>>> tomcat >>>>>>>>>>>>>>>> process to get it back alive. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There is no too much clues in the logs. The apache error.log >>>>>>>>>>>>>>>> gives >>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>> this line: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid >>>>>>>>>>>>>>>> 13385:tid >>>>>>>>>>>>>>>> 1397934896385 >>>>>>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, >>>>>>>>>>>>>>>> consider >>>>>>>>>>>>>>>> raising >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> MaxR >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> equestWorkers setting >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> seems that somehow tomcat, mod-jk2 or even apache is >>>>>>>>>>>>>>>> vulnerable to >>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> exploit, as we certainly does not have such traffic that >>>>>>>>>>>>>>> would >>>>>>>>>>>>>>> block >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> our >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> server otherwise >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> for now we have increased MaxRequestWorkers and we have >>>>>>>>>>>>>>>> limited >>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> connections from one client to 5 by mod_bw and limited number >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> simultaneous connections from one ip by iptables but does not >>>>>>>>>>>>>>>> know >>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> will help >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> best regards, >>>>>>>>>>>>>>>> artur >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Thanks* >>>>>>>>>>>>>>> *Niranjan* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This sounds like a variant of the slowloris attack. >>>>>> >>>>>> This type of attack doesn't take a large number of clients or consume a >>>>>> large amount of bandwidth. >>>>>> >>>>>> Basically, the maximum number of connections are made to the server, >>>>>> and >>>>>> just enough data is sent to each connection in order to not trigger the >>>>>> timeout. André has explained this in more detail earlier in the thread. >>>>>> Search for "slowloris attack" for more information. >>>>>> >>>>>> There are several ways of mitigating this type of attack. >>>>>> >>>>>> As André has mentioned, placing a dedicated device in front of your >>>>>> systems is often the best way. Lots of benefits (platform neutral, no >>>>>> stress on your current servers), and some issues (cost, placement / >>>>>> access may be an issue with hosted solutions). >>>>>> >>>>>> However, there are Apache HTTPD modules that can help mitigate these >>>>>> types of attacks. Some of them are: >>>>>> >>>>>> mod_reqtimeout (should be included by default in your Apache HTRPD 2.4) >>>>>> mod_qos (quality of service module) >>>>>> mod_security (application firewall with lots of security rules) >>>>>> >>>>>> Do a quick search on "slowloris attack apache httpd 2.4" to get some >>>>>> ideas. >>>>>> >>>>>> All of them will probably place additional load on your Apache HTTPD >>>>>> server, so make sure that the platform is robust enough to manage the >>>>>> additional load. >>>>>> >>>>>> There is also a beta version of the mod_security module written as a >>>>>> servlet filter. It should be possible to build this and configure the >>>>>> filter in Tomcat's default web.xml ($CATALINA_BASE/conf/web.xml). I've >>>>>> not tried this. Also, the code base hasn't seen any activity for 3 >>>>>> years. >>>>>> >>>>>> Do a quick search on "modsecurity servlet filter" to find out more >>>>>> about >>>>>> the servlet filter version of mod_security. >>>>>> >>>>>> In short, there appear to be some ways to mitigate the attack. >>>>>> >>>>>> . . . just my two cents >>>>>> /mde/
signature.asc
Description: OpenPGP digital signature
