Artur, On 11/30/2016 9:02 AM, Jaaz Portal wrote: > hi, > sorry, there was two open connection on port 80 > from 194.135.88.32 that is somwhere on epix.net.pl > an association of internet traffick exchange (some pirate hub) > > best, > artur
194.135.88.32 appears to be your web site, no? . . . just my two cents /mde/ > > 2016-11-30 17:52 GMT+01:00 Jaaz Portal <jaazpor...@gmail.com>: > >> hi, >> i looked at the logs but there are no strange things, >> traffic as usual, no errors despite this one: >> [Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid >> 139906329666752] AH00484: server reached MaxRequestWorkers setting, >> consider raising the MaxRequestWorkers setting >> >> any idea what to do next? >> >> best, >> artur >> >> 2016-11-30 17:36 GMT+01:00 Jaaz Portal <jaazpor...@gmail.com>: >> >>> hi, >>> they has tried again with success despite setting connection_timeout and >>> limiting number of clients by mod_bw >>> the tomcat has frozen again. >>> >>> netstat does not showed any connections on port 80 but plenty of >>> connections from apache to localhost:8009 >>> so it was not an attack that you has described (no slowlaris) >>> >>> im looking into debug files of mod_jk and forensic for some hints. If you >>> want i can share them (they have 4mb compressed) >>> >>> best wishes >>> artur >>> >>> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>> >>>> On 28.11.2016 22:04, Jaaz Portal wrote: >>>> >>>>> hi Andre, >>>>> you are wrong. This vulnerability is not only causing memory leaks, it >>>>> makes also apache workers to hang >>>>> >>>> >>>> Maybe for the last time here : >>>> >>>> - what do you call "apache workers" ? >>>> >>>> , making it easy to exhaust the pool. >>>> >>>> - what do you call "the pool" ? >>>> >>>> what i have in my log files. But it is true also that such exhaustion can >>>>> be made by other forms of dos attacks described in this thread. >>>>> >>>>> regarding you suggestion on our application, it does not dos bind server >>>>> nether does not scan for various vulnerabilities in apache, what i have >>>>> also in the logs >>>>> >>>> >>>> For your information : I run about 25 Internet-facing Apache webservers >>>> (some with a back-end tomcat, some not). >>>> On every single one of those webservers, there are *hundreds* of such >>>> "scans" every day, as shown by the Apache access logs. That is just a fact >>>> of life on the Internet. >>>> They are annoying, but most of them are harmless (from an "attack" point >>>> of view), because they are scanning for things that we do not run >>>> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and >>>> thus are responded to by Apache as "404 Not found". >>>> What is annoying with those scans, is >>>> a) that they fill the logfile, and thus make it more difficult to find >>>> really significant things >>>> b) that each of those requires some bandwidth and system resource, if >>>> only to return a "404 Not found" (or a "401 Unauthorised"), and that we pay >>>> for that bandwidth and resources. >>>> >>>> If I could find a way to charge 0.1 cent per access to my servers, from >>>> the people who wrote or run the programs who are doing this, I could retire >>>> in luxury. >>>> >>>> But they are not a real problem, because they are caught as "invalid" by >>>> Apache, and rejected quickly, so they cannot do anything really nasty >>>> (except if they were sending several thousand such requests per second to >>>> one of my servers for a long time). >>>> >>>> The ones that are worrying, are the ones >>>> - a) which do /not/ end up as a "404 Not found", because they have found >>>> an application which responds, and they are not coming from our legitimate >>>> customers >>>> - b) /the ones which we do not see/, because they either do not send a >>>> valid HTTP request, or they have found a way to trigger one of our >>>> applications, in such a way that the application misbehaves and, perhaps, >>>> even if they do not crash our servers, they may provide the attacker with >>>> some entry point to do other things which we do not know and do not control >>>> >>>> What I am trying to say here, is /do not jump to premature conclusions/. >>>> Such "scans" as you mention, happen to everyone, all the time, from >>>> ever-changing IP addresses located all over the world. Some of those >>>> "scans" may come from the infected PC of your grandmother, and she does not >>>> even know about it. >>>> >>>> There is no guarantee, and no indication or proof so far, that /these/ >>>> scans are even related to "the other thing" which happens on your >>>> webserver, which looked much more focused. >>>> >>>> So do not just bundle them together as being the same thing, until you >>>> have some real data that shows for example that these different things all >>>> come from the same IP addresses. >>>> >>>> And one more thing, also finally until you come back with some real data >>>> : I am not saying that your application "scans your server". What I am >>>> saying is that, maybe, by chance or by design, the attackers have found a >>>> URL which goes to your application, and which causes your application to >>>> keep tomcat and/or Apache busy for a long time. >>>> And that maybe /that/ is the problem you should be looking for, and not >>>> some hypothetical bug in Apache httpd or tomcat. >>>> >>>> >>>> >>>>> kindly regards, >>>>> artur >>>>> >>>>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>>> >>>>> On 28.11.2016 20:34, Jaaz Portal wrote: >>>>>> >>>>>> hi mark, >>>>>>> yes, i understand now what slowloris attack is. >>>>>>> maybe it was this maybe *this one based on * * mod_proxy denial of >>>>>>> service * >>>>>>> CVE-2014-0117 <http://cve.mitre.org/cgi-bin/ >>>>>>> cvename.cgi?name=CVE-2014-0117> >>>>>>> >>>>>>> >>>>>> You keep on saying this, but the description of that vulnerability of >>>>>> *Apache httpd*, and the symptoms which you described, *do not match*. >>>>>> You described the problem as ocurring in Apache tomcat, which in your >>>>>> case >>>>>> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat >>>>>> cured the problem. >>>>>> >>>>>> The CVE above applies to Apache httpd, and describes how an attacker >>>>>> could >>>>>> attack Apache httpd and cause *its children* processes to crash (the >>>>>> children processes of Apache httpd), by leading them to consume a lot >>>>>> of >>>>>> memory and crash with an out-of-memory error. >>>>>> Granted, the problem occurred in the mod_proxy module of Apache httpd; >>>>>> but >>>>>> it was httpd which crashed, not tomcat. >>>>>> And tomcat processes are not "Apache httpd children processes" in any >>>>>> understanding of the term. >>>>>> >>>>>> As far as I remember, you never mentioned Apache httpd crashing. You >>>>>> mentioned "the pool" getting full or satured or something like that, >>>>>> without ever describing properly what you meant by "the pool". >>>>>> >>>>>> As far as I am concerned, according to all the relatively unspecific >>>>>> information which you have previously provided : >>>>>> 1) the attack - if that is what it is/was - is definitely NOT related >>>>>> to >>>>>> the CVE which you have repeatedly mentioned >>>>>> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed >>>>>> at >>>>>> your front-end Apache. Instead, it seems that the requests are properly >>>>>> received by Apache, properly decoded by Apache, and then whatever >>>>>> Apache >>>>>> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) is >>>>>> properly forwarding these requests to a back-end tomcat; and it is at >>>>>> the >>>>>> level of that back-end tomcat that the requests never seem to end, and >>>>>> in >>>>>> the end paralyse your tomcat server (and later on maybe your Apache >>>>>> httpd >>>>>> server too, because it is also waiting for tomcat to respond). >>>>>> >>>>>> So your very way of describing the problem, in terms of "first we used >>>>>> this proxy module, and then they exploited the vulnerability so and so; >>>>>> then we changed the proxy module, and they exploited that too; etc.." >>>>>> seems to not have anything to do with the problem per se, and (I >>>>>> believe) >>>>>> confuses everyone, including yourself. >>>>>> >>>>>> It is not that "they" exploited different vulnerabilities of various >>>>>> httpd >>>>>> proxy modules, one after the other. Each of these proxy modules was >>>>>> doing >>>>>> its job properly, and forwarding valid HTTP requests properly to >>>>>> tomcat. >>>>>> When you changed from one proxy module to another, you did not really >>>>>> change anything in that respect, because any proxy module would do the >>>>>> same. >>>>>> >>>>>> But in all cases, what did not change, was the tomcat behind the >>>>>> front-end, and the application running on that tomcat. So the presumed >>>>>> attackers did not have to change anything, they just kept on sending >>>>>> the >>>>>> same requests, because they were really targeting your back-end tomcat >>>>>> or >>>>>> the tomcat application in it, no matter /how/ you were forwarding >>>>>> requests >>>>>> from Apache httpd to tomcat. >>>>>> >>>>>> So either it is tomcat itself, which has a problem with some request >>>>>> URLs >>>>>> which do not bother Apache httpd (possible, but statistically >>>>>> unlikely), or >>>>>> it is the application which runs in tomcat that has such a problem >>>>>> (statistically, much more likely). >>>>>> >>>>>> we do not know yet >>>>>> >>>>>>> >>>>>>> we have setup more logging and are waiting for them to attack once >>>>>>> again >>>>>>> >>>>>>> >>>>>> Yes, that is the right thing to do. Before deciding what the problem >>>>>> may >>>>>> be, and what you can do about it, the first thing you need is *data*. >>>>>> You >>>>>> need to know >>>>>> - which request URL(s?) cause that problem >>>>>> - which IPs these requests come from (always the same ? multiple IPs >>>>>> that >>>>>> change all the time ? how many ? can these IPs be valid/expected >>>>>> clients or >>>>>> not ? do these IPs look like some "coordinated group" ?) >>>>>> - how many such requests there may be during some period of time (10, >>>>>> 100, >>>>>> 1000, more ?) >>>>>> - if these URLs result in passing the request to tomcat >>>>>> - what tomcat application (if any) they are directed to >>>>>> - if so, when that application receives such a request, what is it >>>>>> supposed to do ? does it do it properly ? how long does it need, to >>>>>> respond >>>>>> to such a request ? >>>>>> >>>>>> You also need to ask yourself a question : is the application which you >>>>>> run inside tomcat something that you designed yourself (and which >>>>>> hackers >>>>>> are unlikely to know well-enough to find such a URL which paralyses >>>>>> your >>>>>> server) ? or is it some well-known third-party java application which >>>>>> you >>>>>> are running (and for which would-be attackers would be much more >>>>>> likely to >>>>>> know exactly such a bug) ? >>>>>> >>>>>> >>>>>> anyway, thank you for all informations, it was very useful and >>>>>>> educational >>>>>>> reading for all of us >>>>>>> >>>>>>> best wishes, >>>>>>> artur >>>>>>> >>>>>>> 2016-11-28 19:46 GMT+01:00 Mark Eggers <its_toas...@yahoo.com.invalid >>>>>>>> : >>>>>>> >>>>>>> Jaaz, >>>>>>> >>>>>>>> >>>>>>>> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote: >>>>>>>> >>>>>>>> On 27.11.2016 19:03, Jaaz Portal wrote: >>>>>>>>> >>>>>>>>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>>>>>>>> >>>>>>>>>> On 27.11.2016 14:26, Jaaz Portal wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> hi, >>>>>>>>>>> >>>>>>>>>>>> everything i know so far is just this single log line that >>>>>>>>>>>> appeared >>>>>>>>>>>> in >>>>>>>>>>>> apache error.log >>>>>>>>>>>> >>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid >>>>>>>>>>>> 13385:tid >>>>>>>>>>>> 1397934896385 >>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, consider >>>>>>>>>>>> >>>>>>>>>>>> raising >>>>>>>>>>> >>>>>>>>>> >>>>>>>> the >>>>>>>>> >>>>>>>>>> MaxR >>>>>>>>>>>> equestWorkers setting >>>>>>>>>>>> >>>>>>>>>>>> there was nothing else, just this strange line >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This is not a "strange" line. It is telling you something. >>>>>>>>>>> One problem is that you seem convinced in advance, without serious >>>>>>>>>>> proof, >>>>>>>>>>> that there is a "bug" or a vulnerability in httpd or tomcat. >>>>>>>>>>> Read the explanation of the httpd parameter, here : >>>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequ >>>>>>>>>>> estworkers >>>>>>>>>>> and try to understand it. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I understand it very well. We are serving no more that 500 >>>>>>>>>> clients per >>>>>>>>>> day >>>>>>>>>> so there was no other option that some kind of attack. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> About the "bug" or "vulnerability" : a webserver is supposed to >>>>>>>>>> serve >>>>>>>>>> HTTP >>>>>>>>>> >>>>>>>>>> requests from clients. That is what you expect of it. It has no >>>>>>>>>>> choice but >>>>>>>>>>> to accept any client connection and request, up to the maximum it >>>>>>>>>>> can >>>>>>>>>>> handle considering its configuration and the capacity of the >>>>>>>>>>> system on >>>>>>>>>>> which it runs. That is not a bug, it is a feature. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> We have some weeks ago come under attack from some Polish group. >>>>>>>>>>> It >>>>>>>>>>> >>>>>>>>>> was >>>>>>>>>> first bind that was DoS'ed, we was running on stable Debian so i >>>>>>>>>> updated >>>>>>>>>> bind to latest version. It did not helped. They has dos'ed it so we >>>>>>>>>> switched to other dns provider. That has helped. >>>>>>>>>> >>>>>>>>>> Then they exploited some well know vulnerability in mod_proxy. We >>>>>>>>>> have >>>>>>>>>> updated apache to the latest but again they has exploited it, so we >>>>>>>>>> have >>>>>>>>>> switched to mod_jk. And then guess what. They exploited it too so i >>>>>>>>>> decided >>>>>>>>>> to write to this list looking for help before trying jetty. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The normal Apache httpd access log, will log a request only when >>>>>>>>>>> it is >>>>>>>>>>> finished. If the request never finishes, it will not get logged. >>>>>>>>>>> That may be why you do not see these requests in the log. >>>>>>>>>>> But have a look at this Apache httpd module : >>>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ok, thanks, will take care >>>>>>>>>> >>>>>>>>>> Note : that is also why I was telling you to enable the mod_jk log, >>>>>>>>>> and to >>>>>>>>>> >>>>>>>>>> examine it. >>>>>>>>>>> Because mod_jk will also log information before the request >>>>>>>>>>> produces a >>>>>>>>>>> response. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> and server hanged. >>>>>>>>>>> >>>>>>>>>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Apache was accepting connection but not processed it. After >>>>>>>>>> restart of >>>>>>>>>> tomcat server it worked again. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Also in >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> access logs there are no clues that it was under any heavy load. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> after around hour after discovering that our server hanged-out we >>>>>>>>>>>> have >>>>>>>>>>>> restarted tomcat server >>>>>>>>>>>> and it worked again >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Yes, because that will close all connections between Apache >>>>>>>>>>> httpd and >>>>>>>>>>> tomcat, and abort all requests that are in the process of being >>>>>>>>>>> processed >>>>>>>>>>> by tomcat. So mod_jk will get an error from tomcat, and will >>>>>>>>>>> report an >>>>>>>>>>> error to httpd, and httpd will communicate that error to the >>>>>>>>>>> clients, >>>>>>>>>>> and >>>>>>>>>>> close their connection. >>>>>>>>>>> It still does not tell you what the problem was. >>>>>>>>>>> The only thing that it suggests, is that the "bad" requests >>>>>>>>>>> actually >>>>>>>>>>> make >>>>>>>>>>> it all the way to tomcat. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> correct >>>>>>>>>> >>>>>>>>>> i will enable logs that you has pointed out and we will look what i >>>>>>>>>> will >>>>>>>>>> catch >>>>>>>>>> however i think we have only one chance, as if the solution we has >>>>>>>>>> found >>>>>>>>>> out (connection_timeout + mod_bn) >>>>>>>>>> will work they will stop exploiting it >>>>>>>>>> >>>>>>>>>> thank you very much for all the help and explanations >>>>>>>>>> i will report to the list new facts once they will attack us again >>>>>>>>>> >>>>>>>>>> best regards, >>>>>>>>>> artur >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Ok, but also read this e.g. : >>>>>>>>> https://www.corero.com/blog/695-going-after-the-people- >>>>>>>>> >>>>>>>>> behind-ddos-attacks.html >>>>>>>> >>>>>>>> >>>>>>>>> Attempts to bring down a site by DoS attacks is a crime, in most >>>>>>>>> places.. >>>>>>>>> >>>>>>>>> You can report it, while at the same time trying to defend yourself >>>>>>>>> against it. >>>>>>>>> >>>>>>>>> It is also relatively easy, and quite inexpensive in terms of system >>>>>>>>> resources, to run a small shell script which takes a list every few >>>>>>>>> seconds of the connections to the port of your webserver, and which >>>>>>>>> IPs >>>>>>>>> they are coming *from*. >>>>>>>>> E.g. >>>>>>>>> First try the netstat command, to see what it lists, like : >>>>>>>>> # netstat -n --tcp | more >>>>>>>>> >>>>>>>>> Then you will want to filter this a bit, to only consider >>>>>>>>> established >>>>>>>>> connections to your webserver, for example : >>>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >>>>>>>>> >>>>>>>>> Then you will want to send this to a logfile, regularly, like this : >>>>>>>>> >>>>>>>>> # date >> some_file.log >>>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> >>>>>>>>> some_file.log >>>>>>>>> (repeat every 3 seconds) >>>>>>>>> >>>>>>>>> This will not generate GB of logfiles, and it will tell you, when >>>>>>>>> the >>>>>>>>> problem happens, how many connections there are exactly to your >>>>>>>>> webserver, and where they are coming from. >>>>>>>>> Then later you can further analyse this information.. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> i think that setting connection-timeout and limiting the number of >>>>>>>>>> >>>>>>>>>>> clients >>>>>>>>>>>> by mod_bd i will >>>>>>>>>>>> have effect that next time somebody will try this exploit it will >>>>>>>>>>>> >>>>>>>>>>>> block >>>>>>>>>>> >>>>>>>>>> >>>>>>>> his >>>>>>>>> >>>>>>>>>> access to the site >>>>>>>>>>>> for minute or two, pretty good holistic solution i would say >>>>>>>>>>>> >>>>>>>>>>>> still, it seems that there is serious vulnerability somewhere in >>>>>>>>>>>> apache, >>>>>>>>>>>> mod_jk or tomcat >>>>>>>>>>>> i would like to help find it out but need some hints which debug >>>>>>>>>>>> options >>>>>>>>>>>> enable to catch the bad guys >>>>>>>>>>>> when they will try next time >>>>>>>>>>>> >>>>>>>>>>>> best regards, >>>>>>>>>>>> artur >>>>>>>>>>>> >>>>>>>>>>>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com >>>>>>>>>>>>> : >>>>>>>>>>>> >>>>>>>>>>>> On 27.11.2016 13:23, Jaaz Portal wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> hi Andre, >>>>>>>>>>>>> >>>>>>>>>>>>> thank you very much this was very educative but in my case it is >>>>>>>>>>>>>> little >>>>>>>>>>>>>> bit >>>>>>>>>>>>>> different. >>>>>>>>>>>>>> The server is no flooded, there is maybe dozen of very >>>>>>>>>>>>>> sophisticated >>>>>>>>>>>>>> connections that somehow hangs apache workers threads >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you be a bit more specific ? >>>>>>>>>>>>>> >>>>>>>>>>>>> When you say "apache workers threads", do you mean threads in >>>>>>>>>>>>> Apache >>>>>>>>>>>>> httpd, or threads in Apache Tomcat ? (both are Apache >>>>>>>>>>>>> webservers, >>>>>>>>>>>>> so it >>>>>>>>>>>>> is >>>>>>>>>>>>> difficult to tell what you are talking about, unless you are >>>>>>>>>>>>> very >>>>>>>>>>>>> precise). >>>>>>>>>>>>> >>>>>>>>>>>>> Let me give you some additional explanations, and maybe you can >>>>>>>>>>>>> >>>>>>>>>>>>> figure >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> out >>>>>>>>> >>>>>>>>>> exactly where it "hangs". >>>>>>>>>>>>> >>>>>>>>>>>>> From the Apache httpd front-end point of view, mod_jk (the >>>>>>>>>>>>> connector to >>>>>>>>>>>>> Apache Tomcat) is basically one among other "response >>>>>>>>>>>>> generators". >>>>>>>>>>>>> Apache >>>>>>>>>>>>> httpd does not "know" that behind mod_jk, there is one or more >>>>>>>>>>>>> Tomcat >>>>>>>>>>>>> servers. Apache httpd receives the original client request, and >>>>>>>>>>>>> depending >>>>>>>>>>>>> on the URL of the request, it will pass it to mod_jk or to >>>>>>>>>>>>> another >>>>>>>>>>>>> response >>>>>>>>>>>>> generator, to generate the response to the request. >>>>>>>>>>>>> That mod_jk in the background is using a Tomcat server to >>>>>>>>>>>>> actually >>>>>>>>>>>>> generate the response, is none of the business of Apache httpd, >>>>>>>>>>>>> and >>>>>>>>>>>>> >>>>>>>>>>>>> it >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> does >>>>>>>>> >>>>>>>>>> not care. All it cares about, is to actually receive the response >>>>>>>>>>>>> >>>>>>>>>>>>> from >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> mod_jk, and pass it back to the client. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> If httpd passes a request to mod_jk, it is because you have >>>>>>>>>>>>> specified in >>>>>>>>>>>>> the Apache configuration, the type of URL that it should pass to >>>>>>>>>>>>> mod_jk.. >>>>>>>>>>>>> That happens via your "JkMount (URL pattern)" directives in >>>>>>>>>>>>> Apache >>>>>>>>>>>>> httpd. >>>>>>>>>>>>> >>>>>>>>>>>>> Of course Apache httpd will not pass a request to mod_jk, >>>>>>>>>>>>> before it >>>>>>>>>>>>> has >>>>>>>>>>>>> received at least the URL of the request, and analysed it to >>>>>>>>>>>>> >>>>>>>>>>>>> determine >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> *if* >>>>>>>>> >>>>>>>>>> it should pass it to mod_jk (*). >>>>>>>>>>>>> >>>>>>>>>>>>> If the mod_jk logging is enabled, you can see in it, exactly >>>>>>>>>>>>> *which* >>>>>>>>>>>>> requests are passed to mod_jk and to Tomcat. >>>>>>>>>>>>> Do you know *which* requests, from which clients, cause this >>>>>>>>>>>>> "thread >>>>>>>>>>>>> hanging" symptom ? >>>>>>>>>>>>> Once you would know this, maybe you can design a strategy to >>>>>>>>>>>>> block >>>>>>>>>>>>> specifically these requests. >>>>>>>>>>>>> >>>>>>>>>>>>> and the effect is permanent. Quickly the pool is exhausted >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> which pool exactly ? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> and the only >>>>>>>>>>>>> >>>>>>>>>>>>> solution that works is to restart tomcat. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> I think it is a bug similar to this one from mod_proxy: >>>>>>>>>>>>>> https://tools.cisco.com/security/center/viewAlert.x?alertId= >>>>>>>>>>>>>> 34971 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Maybe, maybe not. As long as we do not know what the requests >>>>>>>>>>>>>> are, >>>>>>>>>>>>>> that >>>>>>>>>>>>>> >>>>>>>>>>>>>> block things, we do not know this. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I think also that your solution with setting connectionTimeout >>>>>>>>>>>>> will >>>>>>>>>>>>> solve >>>>>>>>>>>>> >>>>>>>>>>>>> the problem, at least partially. THANK YOU. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Same thing, we do not know this yet. It was only giving this >>>>>>>>>>>>>> >>>>>>>>>>>>> explanation, >>>>>>>>>>>>> to help you think about where the problem may be. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I would like to help you further investigate this issue, as our >>>>>>>>>>>>> >>>>>>>>>>>>> server >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> comes under such attack once or twice in a week. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Other than giving you hints, there is not much I or anyone else >>>>>>>>>>>>>> can do >>>>>>>>>>>>>> >>>>>>>>>>>>>> to >>>>>>>>>>>>> help. You are the one with control of your servers and >>>>>>>>>>>>> logfiles, so >>>>>>>>>>>>> you >>>>>>>>>>>>> have to investigate and provide more precise information. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> (*) actually, to be precise, Apache httpd passes *all* requests >>>>>>>>>>>>> to >>>>>>>>>>>>> mod_jk, >>>>>>>>>>>>> to ask it "if it wants that request". mod_jk then accepts or >>>>>>>>>>>>> >>>>>>>>>>>>> declines, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> depending on the JkMount instructions. If mod_jk declines, then >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> Apache >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> httpd will ask the next response generator if it wants this request, >>>>>>>>> >>>>>>>>>> etc... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> best regards, >>>>>>>>>>>>> >>>>>>>>>>>>> artur >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) < >>>>>>>>>>>>>> a...@ice-sa.com>: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Have a look that the indicated parameters in the two pages >>>>>>>>>>>>>>> below.. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You may be the target of such a variant of DDoS attack : many >>>>>>>>>>>>>>> clients >>>>>>>>>>>>>>> open >>>>>>>>>>>>>>> a TCP connection to your server (front-end), but then never >>>>>>>>>>>>>>> sends >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>> HTTP >>>>>>>>>>>>>>> request on that connection. In the meantime, the server >>>>>>>>>>>>>>> accepts >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> the >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> TCP >>>>>>>>> >>>>>>>>>> connection, and passes it on to a "child" process or thread for >>>>>>>>>>>>>>> processing. The child then waits for the HTTP request line to >>>>>>>>>>>>>>> arrive >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>> the connection (during a certain time), but it never arrives. >>>>>>>>>>>>>>> After a >>>>>>>>>>>>>>> while, this triggers a timeout (see below), but the standard >>>>>>>>>>>>>>> value of >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>> timeout may be such that in the meantime, a lot of other >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> connections >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> have >>>>>>>>> >>>>>>>>>> been established by other such nefarious clients, so a lot of >>>>>>>>>>>>>>> resources >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>> the webserver are tied up, waiting for something that will >>>>>>>>>>>>>>> never >>>>>>>>>>>>>>> come.. >>>>>>>>>>>>>>> Since there is never any real request sent on the connection, >>>>>>>>>>>>>>> you >>>>>>>>>>>>>>> would >>>>>>>>>>>>>>> (probably) not see this in the logs either. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The above is the basic mechanism of such an attack. There >>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>> variations, such as the client not "not sending" a request >>>>>>>>>>>>>>> line, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> but >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> sending it extremely slowly, thus achieving perhaps similar kinds >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> of >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> effects. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> As someone pointed out, it is quite difficult to do something >>>>>>>>>>>>>>> about >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> at the level of the webserver itself, because by the time you >>>>>>>>>>>>>>> would do >>>>>>>>>>>>>>> something about it, the resources have already been consumed >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> server is probably already overloaded. >>>>>>>>>>>>>>> There are specialised front-end devices and software >>>>>>>>>>>>>>> available, to >>>>>>>>>>>>>>> detect >>>>>>>>>>>>>>> and protect against this kind of attack. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You may want to have a look at the following parameters, but >>>>>>>>>>>>>>> make >>>>>>>>>>>>>>> sure >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> read the caveats (side-effects, interlocking timeouts etc.), >>>>>>>>>>>>>>> otherwise >>>>>>>>>>>>>>> you >>>>>>>>>>>>>>> may do more harm than good. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Another thing : the settings below are for Apache Tomcat, >>>>>>>>>>>>>>> which >>>>>>>>>>>>>>> in your >>>>>>>>>>>>>>> case is the back-end. It would of course be much better to >>>>>>>>>>>>>>> detect >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> eliminate this at the front-end, or even before. I had a >>>>>>>>>>>>>>> look at >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> Apache httpd documentation, and could not find a corresponding >>>>>>>>>>>>>>> parameter. >>>>>>>>>>>>>>> But I am sure that it must exist. You may want to post this >>>>>>>>>>>>>>> same >>>>>>>>>>>>>>> question >>>>>>>>>>>>>>> on the Apache httpd user's list for a better response. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tomcat configuration settings : >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> AJP Connector : (http://tomcat.apache.org/tomc >>>>>>>>>>>>>>> at-8.5-doc/config/ajp.html# >>>>>>>>>>>>>>> Standard_Implementations) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> connectionTimeout >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after >>>>>>>>>>>>>>> accepting a >>>>>>>>>>>>>>> connection, for the request URI line to be presented. The >>>>>>>>>>>>>>> default >>>>>>>>>>>>>>> value >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>> AJP protocol connectors is -1 (i.e. infinite). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (You could for example try to set this to 3000 (milliseconds) >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>> even >>>>>>>>>>>>>>> lower. That should be more than enough for any legitimate >>>>>>>>>>>>>>> client >>>>>>>>>>>>>>> so >>>>>>>>>>>>>>> send >>>>>>>>>>>>>>> the HTTP request line. Note however that by doing this at the >>>>>>>>>>>>>>> Tomcat >>>>>>>>>>>>>>> level, you will probably move the problem to the Apache >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> httpd/mod_jk >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> level. But at least it might confirm that this is the problem >>>>>>>>> >>>>>>>>>> that you >>>>>>>>>>>>>>> are >>>>>>>>>>>>>>> seeing. The mod_jk logfile at the httpd level may give you >>>>>>>>>>>>>>> some >>>>>>>>>>>>>>> hints >>>>>>>>>>>>>>> there.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> HTTP Connector : (http://tomcat.apache.org/tomc >>>>>>>>>>>>>>> at-8.5-doc/config/http.html#Standard_Implementation) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> connectionTimeout >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after >>>>>>>>>>>>>>> accepting a >>>>>>>>>>>>>>> connection, for the request URI line to be presented. Use a >>>>>>>>>>>>>>> value >>>>>>>>>>>>>>> of -1 >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> indicate no (i.e. infinite) timeout. The default value is >>>>>>>>>>>>>>> 60000 >>>>>>>>>>>>>>> (i.e. >>>>>>>>>>>>>>> 60 >>>>>>>>>>>>>>> seconds) but note that the standard server.xml that ships with >>>>>>>>>>>>>>> Tomcat >>>>>>>>>>>>>>> sets >>>>>>>>>>>>>>> this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>> set to >>>>>>>>>>>>>>> false, this timeout will also be used when reading the request >>>>>>>>>>>>>>> body (if >>>>>>>>>>>>>>> any). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 26.11.2016 09:57, Jaaz Portal wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> sorry, its mod_jk no jk2, my typo. All at latest versions. We >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> tried >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> with >>>>>>>>> >>>>>>>>>> mod proxy too. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There is no flood of the server. Nobody is flooding us, they >>>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>> specific connections after which pool of apache workers is >>>>>>>>>>>>>>>> exhausted >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>> blocked >>>>>>>>>>>>>>>> and we need to restart tomcat server. >>>>>>>>>>>>>>>> It is some kind of exploit but do not know how to log it to >>>>>>>>>>>>>>>> obtain >>>>>>>>>>>>>>>> details. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> i had put a limit on connections per client with hope that >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>> help >>>>>>>>>>>>>>>> but once again, it is not a flood. >>>>>>>>>>>>>>>> They open several connections that are not dropped by apache >>>>>>>>>>>>>>>> when they >>>>>>>>>>>>>>>> disconnect. This way whole pool is quickly exhausted and the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> server >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> broken. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> i would like to help you to figure details of this attack but >>>>>>>>>>>>>>>> this is >>>>>>>>>>>>>>>> production server so it is impossible to much debugging >>>>>>>>>>>>>>>> options >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> best, >>>>>>>>>>>>>>>> artur >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu < >>>>>>>>>>>>>>>> niranjan.bo...@gmail.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> you can find who is flooding site in apache access.log and >>>>>>>>>>>>>>>>> block >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> firewall. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ex to find the IP: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -c|sort >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>> -gr >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal >>>>>>>>>>>>>>>>> <jaazpor...@gmail.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> we are from some weeks struggling with some Polish hackers >>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> bringing our server down. After updating apache to latest >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> version >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>> (2.4.23) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> and tomcat (8.0.38) available for debian systems we still >>>>>>>>>>>>>>>>> cannot >>>>>>>>>>>>>>>>> secure >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> server. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Today it has stopped to respond again and we needed to >>>>>>>>>>>>>>>>>> restart >>>>>>>>>>>>>>>>>> tomcat >>>>>>>>>>>>>>>>>> process to get it back alive. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> There is no too much clues in the logs. The apache >>>>>>>>>>>>>>>>>> error.log >>>>>>>>>>>>>>>>>> gives >>>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>> this line: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid >>>>>>>>>>>>>>>>>> 13385:tid >>>>>>>>>>>>>>>>>> 1397934896385 >>>>>>>>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, >>>>>>>>>>>>>>>>>> consider >>>>>>>>>>>>>>>>>> raising >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> MaxR >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> equestWorkers setting >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> seems that somehow tomcat, mod-jk2 or even apache is >>>>>>>>>>>>>>>>>> vulnerable to >>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> exploit, as we certainly does not have such traffic that >>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>> block >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> server otherwise >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> for now we have increased MaxRequestWorkers and we have >>>>>>>>>>>>>>>>>> limited >>>>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> connections from one client to 5 by mod_bw and limited >>>>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> simultaneous connections from one ip by iptables but does >>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>> know >>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> will help >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> best regards, >>>>>>>>>>>>>>>>>> artur >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Thanks* >>>>>>>>>>>>>>>>> *Niranjan* >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This sounds like a variant of the slowloris attack. >>>>>>>> >>>>>>>> This type of attack doesn't take a large number of clients or >>>>>>>> consume a >>>>>>>> large amount of bandwidth. >>>>>>>> >>>>>>>> Basically, the maximum number of connections are made to the server, >>>>>>>> and >>>>>>>> just enough data is sent to each connection in order to not trigger >>>>>>>> the >>>>>>>> timeout. André has explained this in more detail earlier in the >>>>>>>> thread. >>>>>>>> Search for "slowloris attack" for more information. >>>>>>>> >>>>>>>> There are several ways of mitigating this type of attack. >>>>>>>> >>>>>>>> As André has mentioned, placing a dedicated device in front of your >>>>>>>> systems is often the best way. Lots of benefits (platform neutral, no >>>>>>>> stress on your current servers), and some issues (cost, placement / >>>>>>>> access may be an issue with hosted solutions). >>>>>>>> >>>>>>>> However, there are Apache HTTPD modules that can help mitigate these >>>>>>>> types of attacks. Some of them are: >>>>>>>> >>>>>>>> mod_reqtimeout (should be included by default in your Apache HTRPD >>>>>>>> 2.4) >>>>>>>> mod_qos (quality of service module) >>>>>>>> mod_security (application firewall with lots of security rules) >>>>>>>> >>>>>>>> Do a quick search on "slowloris attack apache httpd 2.4" to get some >>>>>>>> ideas. >>>>>>>> >>>>>>>> All of them will probably place additional load on your Apache HTTPD >>>>>>>> server, so make sure that the platform is robust enough to manage the >>>>>>>> additional load. >>>>>>>> >>>>>>>> There is also a beta version of the mod_security module written as a >>>>>>>> servlet filter. It should be possible to build this and configure the >>>>>>>> filter in Tomcat's default web.xml ($CATALINA_BASE/conf/web.xml). >>>>>>>> I've >>>>>>>> not tried this. Also, the code base hasn't seen any activity for 3 >>>>>>>> years. >>>>>>>> >>>>>>>> Do a quick search on "modsecurity servlet filter" to find out more >>>>>>>> about >>>>>>>> the servlet filter version of mod_security. >>>>>>>> >>>>>>>> In short, there appear to be some ways to mitigate the attack. >>>>>>>> >>>>>>>> . . . just my two cents >>>>>>>> /mde/
signature.asc
Description: OpenPGP digital signature
