Just FYI, we run Koha behind the open source version of nginx, and it has lots of flexibility for rate-limiting by client ip and doing pretty sophisticated cache handling.
On Thu, Oct 27, 2016 at 7:50 AM, Pedro Amorim <pjamori...@gmail.com> wrote: > Hello all, > > As I said before, the Koha I'm currently working on is behind a proxy > server, namely HAProxy (http://www.haproxy.org/). > My solution for this was adding the following lines in the frontend of my > haproxy.cfg: > > # Table definition > stick-table type ip size 100k expire 30s store conn_cur > > # Shut the new connection as long as the client has already 10 opened > tcp-request connection reject if { src_conn_cur ge 10 } > tcp-request connection track-sc1 src > > I won't get into too many technical details, but what this does is check if > the current connections is higher than the normal behaviour. More on this > can be read at > http://www.loadbalancer.org/uk/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2 > > For me, this is working great. > > After some testing with several requests and stressing the server at the > same time as keeping F5 pressed in the opac-search, only a couple processes > reach the Koha apache server before HAproxy blocks any subsequent requests. > And this rejection of requests is up for as long as F5 is pressed. As soon > as F5 is released, and normal navigation behaviour ensues, the application > works as normal and the server is healthy. > > I know this is solved at the HAProxy and not in Koha, so the problem > persists in Koha (and other applications as well). > > Me, personally, I agree that this is a Koha problem *and* I also agree that > this is a sys admin responsability. With that said, the Koha community and > the application itself should be aware of this, and by aware I mean at > least some piece of documentation in the Koha performance tuning or server > configuration pages should mention this, even if a prevention > implementation never comes or isn't in the scope of the application. > > Let me know your thoughts on the solution I implemented and if you have any > ideas of how this particular solution could be improved, or any other > alternative solution you might have found more adaptable for your > environment. > > Cheers, > > Pedro Amorim > > > 2016-10-26 20:32 GMT+00:00 clint.deckard <clint.deck...@frontiers.co.nz>: > >> Thank you to everyone who replied, I appreciate you taking the time. >> >> I understand that this is not a Koha specific problem, I understand the >> solution is a sys admin responsibility and I understand that there is >> probably not a 'one size fits all' solution given the many different system >> configurations. I also accept my limitations when it comes to system >> administration, though I am always learning, and I have engaged >> professional help to assist with this issue. >> >> I just thought that since this would be a relatively common issue faced by >> the Koha community and I suspect many Koha instances are run by folk >> without in-depth sysadmin knowledge it might be helpful to have some advice >> on possible ways to deal with the issue. >> >> I appreciate that the Koha community give their time and knowledge >> generously and I really value that. I don't want to take advantage of the >> 'free advice' and I am most appreciative of it when it is offered. >> >> I will be happy to share my experience when I do get a solution. >> >> Again, thank you for your time and assistance, >> Clint. >> >> >> Harry@DHD wrote: >> >>> >>> Even though "sharing" is not obligatory, I had the impression that this >>> is a somewhat "sharing community". >>> If someone is willing he/she could share his solution-experience-howto or >>> what ever. >>> >>> Probably this is the wrong place to ask for this kind of help ... >>> >>> Look into places like SHOREWALL firewall ... etc. But don't get me wrong >>> Firewalls are not easy to master! >>> >>> Regards! >>> >>> >>> On 26/10/2016 08:07 μμ, Huck wrote: >>> >>>> Koha isn't offered 'out of the box'... there is extensive setup required >>>> and the most basic portion(a webserver) is a requirement, and that would be >>>> the responsibility of the server's administrator, not a function of a Koha >>>> developer. >>>> >>>> >>>> On 10/26/2016 5:52 AM, Philippe Blouin wrote: >>>> >>>>> I disagree. If Koha is offered out of the box, and we take time to fix >>>>> security issues, then it's normal for users to expect "basic" attacks to >>>>> be >>>>> taken care of. >>>>> >>>>> More so, blocking IP is not a possibility if genuine users are involved >>>>> using a station from within the library. >>>>> >>>>> I'm not saying you're wrong that it's mostly sysadmin work and not >>>>> Koha, but it doesn't mean nothing can be done. From the apache's threads, >>>>> I found nothing useful (mostly derisive comments). But we could at least >>>>> talk about it. >>>>> >>>>> What about having a javascript preventing refresh on the page withing 5 >>>>> sec of each other? Needs to be done in a way that the refresh doesn't >>>>> restart the timer. >>>>> >>>>> What about having the OPAC search be code where the refresh will >>>>> basically send nothing ? The checkbox are filled, the request is sent to >>>>> the backend, but the frontend keeps nothing... I'm just smoking here, but >>>>> I'm trying to induce some brainstorming in this interesting topic. >>>>> >>>>> Philippe Blouin, >>>>> Responsable du développement informatique >>>>> >>>>> Tél. : (888) 604-2627 >>>>> philippe.blo...@inlibro.com <mailto:philippe.blo...@inlibro.com> >>>>> >>>>> inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> >>>>> On 10/26/2016 07:13 AM, Jonathan Druart wrote: >>>>> >>>>>> Hi, >>>>>> I don't think this can/must be fixed on Koha side. >>>>>> It's a sysadmin duty to take care of that. >>>>>> I would take a look at fail2ban to parse the web server access logs. >>>>>> But >>>>>> make sure not to block your X librarians using the same ip ;) >>>>>> >>>>>> On Wed, 26 Oct 2016 at 12:28 Pedro Amorim <pjamori...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> I have tested this and the stress caused on the server is very severe. >>>>>>> It >>>>>>> seems that for every request, a new zebra process is created and the >>>>>>> server >>>>>>> will only respond when the last one is finished. This ofc will result >>>>>>> in >>>>>>> time outs and eventually a crash in the server. >>>>>>> >>>>>>> This is a major critical issue IMO, anyone who knows about this has >>>>>>> the >>>>>>> power to deny the service of any Koha online without using any >>>>>>> additional >>>>>>> hacking/attacking software. >>>>>>> >>>>>>> The Koha I'm working on right now - still in development - is accessed >>>>>>> behind a proxy server, and I will attempt to solve the problem through >>>>>>> that, by limiting the requests from the same origin with very little >>>>>>> time >>>>>>> between them. Still, even if I'm successful with this, the problem >>>>>>> will >>>>>>> still lie in Koha. >>>>>>> >>>>>>> Anyone with some sort of insight is very welcome. >>>>>>> >>>>>>> Pedro Amorim >>>>>>> >>>>>>> 2016-10-26 8:24 GMT+00:00 clint.deckard < >>>>>>> clint.deck...@frontiers.co.nz>: >>>>>>> >>>>>>> I have had this issue appear today. I have attempted to set up >>>>>>>> >>>>>>> mod_evasive >>>>>>> >>>>>>>> for apache but it doesn't seem to have solved the problem. >>>>>>>> I would really appreciate some advice. >>>>>>>> Clint. >>>>>>>> >>>>>>>> >>>>>>>> rfblanchard wrote: >>>>>>>> >>>>>>>> Assume a basic opac search: >>>>>>>>> http://..../cgi-bin/koha/opac-search.pl?q=dog&branch_group_l >>>>>>>>> imit=branch%3A349 >>>>>>>>> >>>>>>>>> This would take about 10 seconds to return the first time. >>>>>>>>> >>>>>>>>> Assume the user refreshes the results using f5 and keep there finger >>>>>>>>> there a >>>>>>>>> moment to long (3s): >>>>>>>>> This would kill my server for about 1 minute. >>>>>>>>> >>>>>>>>> Any attacker could easily make the server unresponsive indefinitely >>>>>>>>> by >>>>>>>>> simply holding f5 on an opac search. >>>>>>>>> >>>>>>>>> Any recommendations on how to deal with this problem? >>>>>>>>> >>>>>>>>> here is a sample from top: >>>>>>>>> >>>>>>>>> Tasks: 313 total, 3 running, 309 sleeping, 0 stopped, 1 zombie >>>>>>>>> %Cpu(s): 93.7 us, 5.2 sy, 0.0 ni, 1.0 id, 0.2 wa, 0.0 hi, 0.0 >>>>>>>>> si, >>>>>>>>> 0.0 >>>>>>>>> st >>>>>>>>> KiB Mem: 16465036 total, 1532492 used, 14932544 free, 63180 >>>>>>>>> buffers >>>>>>>>> KiB Swap: 8526844 total, 0 used, 8526844 free. 505124 >>>>>>>>> cached >>>>>>>>> Mem >>>>>>>>> >>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>>>> COMMAND >>>>>>>>> 7027 peischo+ 20 0 416164 162924 12756 S 58.8 1.0 0:26.43 >>>>>>>>> /usr/share/koha >>>>>>>>> 7009 peischo+ 20 0 416800 163524 12756 S 56.5 1.0 0:33.77 >>>>>>>>> /usr/share/koha >>>>>>>>> 7444 peischo+ 20 0 129832 15216 5900 R 37.2 0.1 0:01.12 >>>>>>>>> zebrasrv >>>>>>>>> 7445 peischo+ 20 0 129832 15216 5900 R 35.6 0.1 0:01.07 >>>>>>>>> zebrasrv >>>>>>>>> 1151 mysql 20 0 886564 181096 10808 S 8.6 1.1 1:27.57 >>>>>>>>> >>>>>>>> mysqld >>>>>>> >>>>>>>> 7435 koha 20 0 25892 3272 2528 R 0.3 0.0 0:00.03 >>>>>>>>> top >>>>>>>>> 1 root 20 0 176144 5044 3096 S 0.0 0.0 0:01.43 >>>>>>>>> systemd >>>>>>>>> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 >>>>>>>>> kthreadd >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: http://koha.1045719.n5.nabble. >>>>>>>>> com/F5-Attacks-tp5906098.html >>>>>>>>> Sent from the Koha-general mailing list archive at Nabble.com. >>>>>>>>> _______________________________________________ >>>>>>>>> Koha mailing list http://koha-community.org >>>>>>>>> Koha@lists.katipo.co.nz >>>>>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>> Koha mailing list http://koha-community.org >>>>>>>> Koha@lists.katipo.co.nz >>>>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>> Koha mailing list http://koha-community.org >>>>>>> Koha@lists.katipo.co.nz >>>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha >>>>>>> >>>>>>> _______________________________________________ >>>>>> Koha mailing list http://koha-community.org >>>>>> Koha@lists.katipo.co.nz >>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Koha mailing list http://koha-community.org >>>>> Koha@lists.katipo.co.nz >>>>> https://lists.katipo.co.nz/mailman/listinfo/koha >>>>> >>>> >>>> _______________________________________________ >>>> Koha mailing list http://koha-community.org >>>> Koha@lists.katipo.co.nz >>>> https://lists.katipo.co.nz/mailman/listinfo/koha >>>> >>> >>> _______________________________________________ >>> Koha mailing list http://koha-community.org >>> Koha@lists.katipo.co.nz >>> https://lists.katipo.co.nz/mailman/listinfo/koha >>> >> _______________________________________________ >> Koha mailing list http://koha-community.org >> Koha@lists.katipo.co.nz >> https://lists.katipo.co.nz/mailman/listinfo/koha >> > _______________________________________________ > Koha mailing list http://koha-community.org > Koha@lists.katipo.co.nz > https://lists.katipo.co.nz/mailman/listinfo/koha -- Jason _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha