Re: [Koha] F5 Attacks

[Jonathan Druart]

Hi,
I don't think this can/must be fixed on Koha side.
It's a sysadmin duty to take care of that.
I would take a look at fail2ban to parse the web server access logs. But
make sure not to block your X librarians using the same ip ;)

On Wed, 26 Oct 2016 at 12:28 Pedro Amorim  wrote:

> I have tested this and the stress caused on the server is very severe. It
> seems that for every request, a new zebra process is created and the server
> will only respond when the last one is finished. This ofc will result in
> time outs and eventually a crash in the server.
>
> This is a major critical issue IMO, anyone who knows about this has the
> power to deny the service of any Koha online without using any additional
> hacking/attacking software.
>
> The Koha I'm working on right now - still in development - is accessed
> behind a proxy server, and I will attempt to solve the problem through
> that, by limiting the requests from the same origin with very little time
> between them. Still, even if I'm successful with this, the problem will
> still lie in Koha.
>
> Anyone with some sort of insight is very welcome.
>
> Pedro Amorim
>
> 2016-10-26 8:24 GMT+00:00 clint.deckard :
>
> > I have had this issue appear today. I have attempted to set up
> mod_evasive
> > for apache but it doesn't seem to have solved the problem.
> > I would really appreciate some advice.
> > Clint.
> >
> >
> > rfblanchard wrote:
> >
> >> Assume a basic opac search:
> >> http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
> >> imit=branch%3A349
> >>
> >> This would take about 10 seconds to return the first time.
> >>
> >> Assume the user refreshes the results using f5 and keep there finger
> >> there a
> >> moment to long (3s):
> >> This would kill my server for about 1 minute.
> >>
> >> Any attacker could easily make the server unresponsive indefinitely by
> >> simply holding f5 on an opac search.
> >>
> >> Any recommendations on how to deal with this problem?
> >>
> >> here is a sample from top:
> >>
> >> Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
> >> %Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa,  0.0 hi,  0.0 si,
> >> 0.0
> >> st
> >> KiB Mem:  16465036 total,  1532492 used, 14932544 free,63180 buffers
> >> KiB Swap:  8526844 total,0 used,  8526844 free.   505124 cached
> >> Mem
> >>
> >>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
> >> COMMAND
> >>  7027 peischo+  20   0  416164 162924  12756 S  58.8  1.0   0:26.43
> >> /usr/share/koha
> >>  7009 peischo+  20   0  416800 163524  12756 S  56.5  1.0   0:33.77
> >> /usr/share/koha
> >>  7444 peischo+  20   0  129832  15216   5900 R  37.2  0.1   0:01.12
> >> zebrasrv
> >>  7445 peischo+  20   0  129832  15216   5900 R  35.6  0.1   0:01.07
> >> zebrasrv
> >>  1151 mysql 20   0  886564 181096  10808 S   8.6  1.1   1:27.57
> mysqld
> >>  7435 koha  20   0   25892   3272   2528 R   0.3  0.0   0:00.03 top
> >> 1 root  20   0  176144   5044   3096 S   0.0  0.0   0:01.43
> >> systemd
> >> 2 root  20   0   0  0  0 S   0.0  0.0   0:00.00
> >> kthreadd
> >>
> >>
> >>
> >> --
> >> View this message in context: http://koha.1045719.n5.nabble.
> >> com/F5-Attacks-tp5906098.html
> >> Sent from the Koha-general mailing list archive at Nabble.com.
> >> ___
> >> Koha mailing list  http://koha-community.org
> >> Koha@lists.katipo.co.nz
> >> https://lists.katipo.co.nz/mailman/listinfo/koha
> >>
> > ___
> > Koha mailing list  http://koha-community.org
> > Koha@lists.katipo.co.nz
> > https://lists.katipo.co.nz/mailman/listinfo/koha
> >
> ___
> Koha mailing list  http://koha-community.org
> Koha@lists.katipo.co.nz
> https://lists.katipo.co.nz/mailman/listinfo/koha
>
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha

Re: [Koha] F5 Attacks

[Viktor.Sarge]

Hi! 

If this is actually related to Zebra I wonder if the problem will persist when 
the switch to Elasticsearch is made? And if so: is there anything that could be 
done about it now while work is done on Elastic anyway? It’s always easier to 
get it right from the start. 

If I understand the description of the problem correctly what we would want is 
a way to drop previous queries when the search engine is flooded with rerequest 
of queries from the same session id? Dropping the traffic in the firewall seems 
like the best solution since it’s as upstream as possible for a attacker with 
intent. It seems possible to implement something in JS as well but that will 
only protect from clumsy refreshers without intent. 

IF we are feeling helpful towards inexperienced system librarians who forgot or 
misconfigured the firewall
AND it’s really only Zebra that can’t handle the load while apache and Koha are 
doing fine
AND there is already some part of Koha or Zebra that’s already aware of active 
queries and their session ids 
AND there is no caching in Zebra to try first activate 
AND it’s not a passing thing with Elastic coming
AND someone have the funds or time and knowledge to fix it
THEN it seems reasonable to see if we could also drop traffic inside Koha to 
decrease the load

Kind regards
Viktor Sarge

Senior regional library development officer
Regional development council of Halland, Sweden




26 okt 2016 kl. 12:28 skrev Pedro Amorim :

> I have tested this and the stress caused on the server is very severe. It
> seems that for every request, a new zebra process is created and the server
> will only respond when the last one is finished. This ofc will result in
> time outs and eventually a crash in the server.
> 
> This is a major critical issue IMO, anyone who knows about this has the
> power to deny the service of any Koha online without using any additional
> hacking/attacking software.
> 
> The Koha I'm working on right now - still in development - is accessed
> behind a proxy server, and I will attempt to solve the problem through
> that, by limiting the requests from the same origin with very little time
> between them. Still, even if I'm successful with this, the problem will
> still lie in Koha.
> 
> Anyone with some sort of insight is very welcome.
> 
> Pedro Amorim
> 
> 2016-10-26 8:24 GMT+00:00 clint.deckard :
> 
>> I have had this issue appear today. I have attempted to set up mod_evasive
>> for apache but it doesn't seem to have solved the problem.
>> I would really appreciate some advice.
>> Clint.
>> 
>> 
>> rfblanchard wrote:
>> 
>>> Assume a basic opac search:
>>> http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
>>> imit=branch%3A349
>>> 
>>> This would take about 10 seconds to return the first time.
>>> 
>>> Assume the user refreshes the results using f5 and keep there finger
>>> there a
>>> moment to long (3s):
>>> This would kill my server for about 1 minute.
>>> 
>>> Any attacker could easily make the server unresponsive indefinitely by
>>> simply holding f5 on an opac search.
>>> 
>>> Any recommendations on how to deal with this problem?
>>> 
>>> here is a sample from top:
>>> 
>>> Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
>>> %Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa,  0.0 hi,  0.0 si,
>>> 0.0
>>> st
>>> KiB Mem:  16465036 total,  1532492 used, 14932544 free,63180 buffers
>>> KiB Swap:  8526844 total,0 used,  8526844 free.   505124 cached
>>> Mem
>>> 
>>>  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
>>> COMMAND
>>> 7027 peischo+  20   0  416164 162924  12756 S  58.8  1.0   0:26.43
>>> /usr/share/koha
>>> 7009 peischo+  20   0  416800 163524  12756 S  56.5  1.0   0:33.77
>>> /usr/share/koha
>>> 7444 peischo+  20   0  129832  15216   5900 R  37.2  0.1   0:01.12
>>> zebrasrv
>>> 7445 peischo+  20   0  129832  15216   5900 R  35.6  0.1   0:01.07
>>> zebrasrv
>>> 1151 mysql 20   0  886564 181096  10808 S   8.6  1.1   1:27.57 mysqld
>>> 7435 koha  20   0   25892   3272   2528 R   0.3  0.0   0:00.03 top
>>>1 root  20   0  176144   5044   3096 S   0.0  0.0   0:01.43
>>> systemd
>>>2 root  20   0   0  0  0 S   0.0  0.0   0:00.00
>>> kthreadd
>>> 
>>> 
>>> 
>>> --
>>> View this message in context: http://koha.1045719.n5.nabble.
>>> com/F5-Attacks-tp5906098.html
>>> Sent from the Koha-general mailing list archive at Nabble.com.
>>> ___
>>> Koha mailing list  http://koha-community.org
>>> Koha@lists.katipo.co.nz
>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>> 
>> ___
>> Koha mailing list  http://koha-community.org
>> Koha@lists.katipo.co.nz
>> https://lists.katipo.co.nz/mailman/listinfo/koha
>> 
> ___
> Koha mailing list  http://koha-community.org
> Koha@lists.katipo.co.nz
> 

Re: [Koha] F5 Attacks

[Jason Sherman]

Just FYI, we run Koha behind the open source version of nginx, and it
has lots of flexibility for rate-limiting by client ip and doing
pretty sophisticated cache handling.

On Thu, Oct 27, 2016 at 7:50 AM, Pedro Amorim  wrote:
> Hello all,
>
> As I said before, the Koha I'm currently working on is behind a proxy
> server, namely HAProxy (http://www.haproxy.org/).
> My solution for this was adding the following lines in the frontend of my
> haproxy.cfg:
>
> # Table definition
> stick-table type ip size 100k expire 30s store conn_cur
>
> # Shut the new connection as long as the client has already 10 opened
> tcp-request connection reject if { src_conn_cur ge 10 }
> tcp-request connection track-sc1 src
>
> I won't get into too many technical details, but what this does is check if
> the current connections is higher than the normal behaviour. More on this
> can be read at
> http://www.loadbalancer.org/uk/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2
>
> For me, this is working great.
>
> After some testing with several requests and stressing the server at the
> same time as keeping F5 pressed in the opac-search, only a couple processes
> reach the Koha apache server before HAproxy blocks any subsequent requests.
> And this rejection of requests is up for as long as F5 is pressed. As soon
> as F5 is released, and normal navigation behaviour ensues, the application
> works as normal and the server is healthy.
>
> I know this is solved at the HAProxy and not in Koha, so the problem
> persists in Koha (and other applications as well).
>
> Me, personally, I agree that this is a Koha problem *and* I also agree that
> this is a sys admin responsability. With that said, the Koha community and
> the application itself should be aware of this, and by aware I mean at
> least some piece of documentation in the Koha performance tuning or server
> configuration pages should mention this, even if a prevention
> implementation never comes or isn't in the scope of the application.
>
> Let me know your thoughts on the solution I implemented and if you have any
> ideas of how this particular solution could be improved, or any other
> alternative solution you might have found more adaptable for your
> environment.
>
> Cheers,
>
> Pedro Amorim
>
>
> 2016-10-26 20:32 GMT+00:00 clint.deckard :
>
>> Thank you to everyone who replied, I appreciate you taking the time.
>>
>> I understand that this is not a Koha specific problem, I understand the
>> solution is a sys admin responsibility and I understand that there is
>> probably not a 'one size fits all' solution given the many different system
>> configurations. I also accept my limitations when it comes to system
>> administration, though I am always learning, and I have engaged
>> professional help to assist with this issue.
>>
>> I just thought that since this would be a relatively common issue faced by
>> the Koha community and I suspect many Koha instances are run by folk
>> without in-depth sysadmin knowledge it might be helpful to have some advice
>> on possible ways to deal with the issue.
>>
>> I appreciate that the Koha community give their time and knowledge
>> generously and I really value that. I don't want to take advantage of the
>> 'free advice' and I am most appreciative of it when it is offered.
>>
>> I will be happy to share my experience when I do get a solution.
>>
>> Again, thank you for your time and assistance,
>> Clint.
>>
>>
>> Harry@DHD wrote:
>>
>>>
>>> Even though "sharing" is not obligatory, I had the impression that this
>>> is a somewhat "sharing community".
>>> If someone is willing he/she could share his solution-experience-howto or
>>> what ever.
>>>
>>> Probably this is the wrong place to ask for this kind of help ...
>>>
>>> Look into places like SHOREWALL firewall  ... etc. But don't get me wrong
>>> Firewalls are not easy to master!
>>>
>>> Regards!
>>>
>>>
>>> On 26/10/2016 08:07 μμ, Huck wrote:
>>>
 Koha isn't offered 'out of the box'... there is extensive setup required
 and the most basic portion(a webserver) is a requirement, and that would be
 the responsibility of the server's administrator, not a function of a Koha
 developer.


 On 10/26/2016 5:52 AM, Philippe Blouin wrote:

> I disagree.  If Koha is offered out of the box, and we take time to fix
> security issues, then it's normal for users to expect "basic" attacks to 
> be
> taken care of.
>
> More so, blocking IP is not a possibility if genuine users are involved
> using a station from within the library.
>
> I'm not saying you're wrong that it's mostly sysadmin work and not
> Koha, but it doesn't mean nothing can be done.  From the apache's threads,
> I found nothing useful (mostly derisive comments).  But we could at least
> talk about it.
>
> What about having a javascript preventing refresh on the 

Re: [Koha] F5 Attacks

[Pedro Amorim]

Hello all,

As I said before, the Koha I'm currently working on is behind a proxy
server, namely HAProxy (http://www.haproxy.org/).
My solution for this was adding the following lines in the frontend of my
haproxy.cfg:

# Table definition
stick-table type ip size 100k expire 30s store conn_cur

# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src

I won't get into too many technical details, but what this does is check if
the current connections is higher than the normal behaviour. More on this
can be read at
http://www.loadbalancer.org/uk/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2

For me, this is working great.

After some testing with several requests and stressing the server at the
same time as keeping F5 pressed in the opac-search, only a couple processes
reach the Koha apache server before HAproxy blocks any subsequent requests.
And this rejection of requests is up for as long as F5 is pressed. As soon
as F5 is released, and normal navigation behaviour ensues, the application
works as normal and the server is healthy.

I know this is solved at the HAProxy and not in Koha, so the problem
persists in Koha (and other applications as well).

Me, personally, I agree that this is a Koha problem *and* I also agree that
this is a sys admin responsability. With that said, the Koha community and
the application itself should be aware of this, and by aware I mean at
least some piece of documentation in the Koha performance tuning or server
configuration pages should mention this, even if a prevention
implementation never comes or isn't in the scope of the application.

Let me know your thoughts on the solution I implemented and if you have any
ideas of how this particular solution could be improved, or any other
alternative solution you might have found more adaptable for your
environment.

Cheers,

Pedro Amorim


2016-10-26 20:32 GMT+00:00 clint.deckard :

> Thank you to everyone who replied, I appreciate you taking the time.
>
> I understand that this is not a Koha specific problem, I understand the
> solution is a sys admin responsibility and I understand that there is
> probably not a 'one size fits all' solution given the many different system
> configurations. I also accept my limitations when it comes to system
> administration, though I am always learning, and I have engaged
> professional help to assist with this issue.
>
> I just thought that since this would be a relatively common issue faced by
> the Koha community and I suspect many Koha instances are run by folk
> without in-depth sysadmin knowledge it might be helpful to have some advice
> on possible ways to deal with the issue.
>
> I appreciate that the Koha community give their time and knowledge
> generously and I really value that. I don't want to take advantage of the
> 'free advice' and I am most appreciative of it when it is offered.
>
> I will be happy to share my experience when I do get a solution.
>
> Again, thank you for your time and assistance,
> Clint.
>
>
> Harry@DHD wrote:
>
>>
>> Even though "sharing" is not obligatory, I had the impression that this
>> is a somewhat "sharing community".
>> If someone is willing he/she could share his solution-experience-howto or
>> what ever.
>>
>> Probably this is the wrong place to ask for this kind of help ...
>>
>> Look into places like SHOREWALL firewall  ... etc. But don't get me wrong
>> Firewalls are not easy to master!
>>
>> Regards!
>>
>>
>> On 26/10/2016 08:07 μμ, Huck wrote:
>>
>>> Koha isn't offered 'out of the box'... there is extensive setup required
>>> and the most basic portion(a webserver) is a requirement, and that would be
>>> the responsibility of the server's administrator, not a function of a Koha
>>> developer.
>>>
>>>
>>> On 10/26/2016 5:52 AM, Philippe Blouin wrote:
>>>
 I disagree.  If Koha is offered out of the box, and we take time to fix
 security issues, then it's normal for users to expect "basic" attacks to be
 taken care of.

 More so, blocking IP is not a possibility if genuine users are involved
 using a station from within the library.

 I'm not saying you're wrong that it's mostly sysadmin work and not
 Koha, but it doesn't mean nothing can be done.  From the apache's threads,
 I found nothing useful (mostly derisive comments).  But we could at least
 talk about it.

 What about having a javascript preventing refresh on the page withing 5
 sec of each other?  Needs to be done in a way that the refresh doesn't
 restart the timer.

 What about having the OPAC search be code where the refresh will
 basically send nothing ?  The checkbox are filled, the request is sent to
 the backend, but the frontend keeps nothing...  I'm just smoking here, but
 I'm trying to induce some brainstorming in this 

Re: [Koha] F5 Attacks

[clint.deckard]

Thank you to everyone who replied, I appreciate you taking the time.

I understand that this is not a Koha specific problem, I understand the 
solution is a sys admin responsibility and I understand that there is 
probably not a 'one size fits all' solution given the many different 
system configurations. I also accept my limitations when it comes to 
system administration, though I am always learning, and I have engaged 
professional help to assist with this issue.


I just thought that since this would be a relatively common issue faced 
by the Koha community and I suspect many Koha instances are run by folk 
without in-depth sysadmin knowledge it might be helpful to have some 
advice on possible ways to deal with the issue.


I appreciate that the Koha community give their time and knowledge 
generously and I really value that. I don't want to take advantage of 
the 'free advice' and I am most appreciative of it when it is offered.


I will be happy to share my experience when I do get a solution.

Again, thank you for your time and assistance,
Clint.

Harry@DHD wrote:


Even though "sharing" is not obligatory, I had the impression that this 
is a somewhat "sharing community".
If someone is willing he/she could share his solution-experience-howto 
or what ever.


Probably this is the wrong place to ask for this kind of help ...

Look into places like SHOREWALL firewall  ... etc. But don't get me 
wrong Firewalls are not easy to master!


Regards!


On 26/10/2016 08:07 μμ, Huck wrote:
Koha isn't offered 'out of the box'... there is extensive setup 
required and the most basic portion(a webserver) is a requirement, and 
that would be the responsibility of the server's administrator, not a 
function of a Koha developer.



On 10/26/2016 5:52 AM, Philippe Blouin wrote:
I disagree.  If Koha is offered out of the box, and we take time to 
fix security issues, then it's normal for users to expect "basic" 
attacks to be taken care of.


More so, blocking IP is not a possibility if genuine users are 
involved using a station from within the library.


I'm not saying you're wrong that it's mostly sysadmin work and not 
Koha, but it doesn't mean nothing can be done.  From the apache's 
threads, I found nothing useful (mostly derisive comments).  But we 
could at least talk about it.


What about having a javascript preventing refresh on the page withing 
5 sec of each other?  Needs to be done in a way that the refresh 
doesn't restart the timer.


What about having the OPAC search be code where the refresh will 
basically send nothing ?  The checkbox are filled, the request is 
sent to the backend, but the frontend keeps nothing...  I'm just 
smoking here, but I'm trying to induce some brainstorming in this 
interesting topic.


Philippe Blouin,
Responsable du développement informatique

Tél.  : (888) 604-2627
philippe.blo...@inlibro.com 

inLibro | pour esprit libre | www.inLibro.com 
On 10/26/2016 07:13 AM, Jonathan Druart wrote:

Hi,
I don't think this can/must be fixed on Koha side.
It's a sysadmin duty to take care of that.
I would take a look at fail2ban to parse the web server access logs. 
But

make sure not to block your X librarians using the same ip ;)

On Wed, 26 Oct 2016 at 12:28 Pedro Amorim  wrote:

I have tested this and the stress caused on the server is very 
severe. It
seems that for every request, a new zebra process is created and 
the server
will only respond when the last one is finished. This ofc will 
result in

time outs and eventually a crash in the server.

This is a major critical issue IMO, anyone who knows about this has 
the
power to deny the service of any Koha online without using any 
additional

hacking/attacking software.

The Koha I'm working on right now - still in development - is accessed
behind a proxy server, and I will attempt to solve the problem through
that, by limiting the requests from the same origin with very 
little time
between them. Still, even if I'm successful with this, the problem 
will

still lie in Koha.

Anyone with some sort of insight is very welcome.

Pedro Amorim

2016-10-26 8:24 GMT+00:00 clint.deckard 
:



I have had this issue appear today. I have attempted to set up

mod_evasive

for apache but it doesn't seem to have solved the problem.
I would really appreciate some advice.
Clint.


rfblanchard wrote:


Assume a basic opac search:
http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
imit=branch%3A349

This would take about 10 seconds to return the first time.

Assume the user refreshes the results using f5 and keep there finger
there a
moment to long (3s):
This would kill my server for about 1 minute.

Any attacker could easily make the server unresponsive 
indefinitely by

simply holding f5 on an opac search.

Any recommendations on how to deal with this problem?

here is a sample from top:

Tasks: 313 total,   3 running, 309 

Re: [Koha] F5 Attacks

[Harry]

Even though "sharing" is not obligatory, I had the impression that this 
is a somewhat "sharing community".
If someone is willing he/she could share his solution-experience-howto 
or what ever.


Probably this is the wrong place to ask for this kind of help ...

Look into places like SHOREWALL firewall  ... etc. But don't get me 
wrong Firewalls are not easy to master!


Regards!


On 26/10/2016 08:07 μμ, Huck wrote:
Koha isn't offered 'out of the box'... there is extensive setup 
required and the most basic portion(a webserver) is a requirement, and 
that would be the responsibility of the server's administrator, not a 
function of a Koha developer.



On 10/26/2016 5:52 AM, Philippe Blouin wrote:
I disagree.  If Koha is offered out of the box, and we take time to 
fix security issues, then it's normal for users to expect "basic" 
attacks to be taken care of.


More so, blocking IP is not a possibility if genuine users are 
involved using a station from within the library.


I'm not saying you're wrong that it's mostly sysadmin work and not 
Koha, but it doesn't mean nothing can be done.  From the apache's 
threads, I found nothing useful (mostly derisive comments).  But we 
could at least talk about it.


What about having a javascript preventing refresh on the page withing 
5 sec of each other?  Needs to be done in a way that the refresh 
doesn't restart the timer.


What about having the OPAC search be code where the refresh will 
basically send nothing ?  The checkbox are filled, the request is 
sent to the backend, but the frontend keeps nothing...  I'm just 
smoking here, but I'm trying to induce some brainstorming in this 
interesting topic.


Philippe Blouin,
Responsable du développement informatique

Tél.  : (888) 604-2627
philippe.blo...@inlibro.com 

inLibro | pour esprit libre | www.inLibro.com 
On 10/26/2016 07:13 AM, Jonathan Druart wrote:

Hi,
I don't think this can/must be fixed on Koha side.
It's a sysadmin duty to take care of that.
I would take a look at fail2ban to parse the web server access logs. 
But

make sure not to block your X librarians using the same ip ;)

On Wed, 26 Oct 2016 at 12:28 Pedro Amorim  wrote:

I have tested this and the stress caused on the server is very 
severe. It
seems that for every request, a new zebra process is created and 
the server
will only respond when the last one is finished. This ofc will 
result in

time outs and eventually a crash in the server.

This is a major critical issue IMO, anyone who knows about this has 
the
power to deny the service of any Koha online without using any 
additional

hacking/attacking software.

The Koha I'm working on right now - still in development - is accessed
behind a proxy server, and I will attempt to solve the problem through
that, by limiting the requests from the same origin with very 
little time
between them. Still, even if I'm successful with this, the problem 
will

still lie in Koha.

Anyone with some sort of insight is very welcome.

Pedro Amorim

2016-10-26 8:24 GMT+00:00 clint.deckard 
:



I have had this issue appear today. I have attempted to set up

mod_evasive

for apache but it doesn't seem to have solved the problem.
I would really appreciate some advice.
Clint.


rfblanchard wrote:


Assume a basic opac search:
http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
imit=branch%3A349

This would take about 10 seconds to return the first time.

Assume the user refreshes the results using f5 and keep there finger
there a
moment to long (3s):
This would kill my server for about 1 minute.

Any attacker could easily make the server unresponsive 
indefinitely by

simply holding f5 on an opac search.

Any recommendations on how to deal with this problem?

here is a sample from top:

Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
%Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa, 0.0 hi,  
0.0 si,

0.0
st
KiB Mem:  16465036 total,  1532492 used, 14932544 free, 63180 
buffers
KiB Swap:  8526844 total,0 used,  8526844 free. 505124 
cached

Mem

   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
COMMAND
  7027 peischo+  20   0  416164 162924  12756 S  58.8 1.0   0:26.43
/usr/share/koha
  7009 peischo+  20   0  416800 163524  12756 S  56.5 1.0   0:33.77
/usr/share/koha
  7444 peischo+  20   0  129832  15216   5900 R  37.2 0.1   0:01.12
zebrasrv
  7445 peischo+  20   0  129832  15216   5900 R  35.6 0.1   0:01.07
zebrasrv
  1151 mysql 20   0  886564 181096  10808 S   8.6 1.1   1:27.57

mysqld
  7435 koha  20   0   25892 3272   2528 R   0.3  0.0   
0:00.03 top

 1 root  20   0  176144   5044   3096 S   0.0 0.0   0:01.43
systemd
 2 root  20   0   0  0  0 S   0.0 0.0   0:00.00
kthreadd



--
View this message in context: http://koha.1045719.n5.nabble.
com/F5-Attacks-tp5906098.html
Sent from the Koha-general 

Re: [Koha] F5 Attacks

[Huck]

Koha isn't offered 'out of the box'... there is extensive setup required 
and the most basic portion(a webserver) is a requirement, and that would 
be the responsibility of the server's administrator, not a function of a 
Koha developer.



On 10/26/2016 5:52 AM, Philippe Blouin wrote:
I disagree.  If Koha is offered out of the box, and we take time to 
fix security issues, then it's normal for users to expect "basic" 
attacks to be taken care of.


More so, blocking IP is not a possibility if genuine users are 
involved using a station from within the library.


I'm not saying you're wrong that it's mostly sysadmin work and not 
Koha, but it doesn't mean nothing can be done.  From the apache's 
threads, I found nothing useful (mostly derisive comments).  But we 
could at least talk about it.


What about having a javascript preventing refresh on the page withing 
5 sec of each other?  Needs to be done in a way that the refresh 
doesn't restart the timer.


What about having the OPAC search be code where the refresh will 
basically send nothing ?  The checkbox are filled, the request is sent 
to the backend, but the frontend keeps nothing...  I'm just smoking 
here, but I'm trying to induce some brainstorming in this interesting 
topic.


Philippe Blouin,
Responsable du développement informatique

Tél.  : (888) 604-2627
philippe.blo...@inlibro.com 

inLibro | pour esprit libre | www.inLibro.com 
On 10/26/2016 07:13 AM, Jonathan Druart wrote:

Hi,
I don't think this can/must be fixed on Koha side.
It's a sysadmin duty to take care of that.
I would take a look at fail2ban to parse the web server access logs. But
make sure not to block your X librarians using the same ip ;)

On Wed, 26 Oct 2016 at 12:28 Pedro Amorim  wrote:

I have tested this and the stress caused on the server is very 
severe. It
seems that for every request, a new zebra process is created and the 
server
will only respond when the last one is finished. This ofc will 
result in

time outs and eventually a crash in the server.

This is a major critical issue IMO, anyone who knows about this has the
power to deny the service of any Koha online without using any 
additional

hacking/attacking software.

The Koha I'm working on right now - still in development - is accessed
behind a proxy server, and I will attempt to solve the problem through
that, by limiting the requests from the same origin with very little 
time

between them. Still, even if I'm successful with this, the problem will
still lie in Koha.

Anyone with some sort of insight is very welcome.

Pedro Amorim

2016-10-26 8:24 GMT+00:00 clint.deckard 
:



I have had this issue appear today. I have attempted to set up

mod_evasive

for apache but it doesn't seem to have solved the problem.
I would really appreciate some advice.
Clint.


rfblanchard wrote:


Assume a basic opac search:
http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
imit=branch%3A349

This would take about 10 seconds to return the first time.

Assume the user refreshes the results using f5 and keep there finger
there a
moment to long (3s):
This would kill my server for about 1 minute.

Any attacker could easily make the server unresponsive 
indefinitely by

simply holding f5 on an opac search.

Any recommendations on how to deal with this problem?

here is a sample from top:

Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
%Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa,  0.0 hi,  
0.0 si,

0.0
st
KiB Mem:  16465036 total,  1532492 used, 14932544 free, 63180 buffers
KiB Swap:  8526844 total,0 used,  8526844 free. 505124 cached
Mem

   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
COMMAND
  7027 peischo+  20   0  416164 162924  12756 S  58.8 1.0   0:26.43
/usr/share/koha
  7009 peischo+  20   0  416800 163524  12756 S  56.5 1.0   0:33.77
/usr/share/koha
  7444 peischo+  20   0  129832  15216   5900 R  37.2 0.1   0:01.12
zebrasrv
  7445 peischo+  20   0  129832  15216   5900 R  35.6 0.1   0:01.07
zebrasrv
  1151 mysql 20   0  886564 181096  10808 S   8.6 1.1   1:27.57

mysqld
  7435 koha  20   0   25892 3272   2528 R   0.3  0.0   0:00.03 
top

 1 root  20   0  176144   5044   3096 S   0.0 0.0   0:01.43
systemd
 2 root  20   0   0  0  0 S   0.0 0.0   0:00.00
kthreadd



--
View this message in context: http://koha.1045719.n5.nabble.
com/F5-Attacks-tp5906098.html
Sent from the Koha-general mailing list archive at Nabble.com.
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha


___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha


___
Koha mailing 

Re: [Koha] F5 Attacks

[Philippe Blouin]

I disagree.  If Koha is offered out of the box, and we take time to fix 
security issues, then it's normal for users to expect "basic" attacks to 
be taken care of.


More so, blocking IP is not a possibility if genuine users are involved 
using a station from within the library.


I'm not saying you're wrong that it's mostly sysadmin work and not Koha, 
but it doesn't mean nothing can be done.  From the apache's threads, I 
found nothing useful (mostly derisive comments).  But we could at least 
talk about it.


What about having a javascript preventing refresh on the page withing 5 
sec of each other?  Needs to be done in a way that the refresh doesn't 
restart the timer.


What about having the OPAC search be code where the refresh will 
basically send nothing ?  The checkbox are filled, the request is sent 
to the backend, but the frontend keeps nothing...  I'm just smoking 
here, but I'm trying to induce some brainstorming in this interesting topic.


Philippe Blouin,
Responsable du développement informatique

Tél.  : (888) 604-2627
philippe.blo...@inlibro.com 

inLibro | pour esprit libre | www.inLibro.com 
On 10/26/2016 07:13 AM, Jonathan Druart wrote:

Hi,
I don't think this can/must be fixed on Koha side.
It's a sysadmin duty to take care of that.
I would take a look at fail2ban to parse the web server access logs. But
make sure not to block your X librarians using the same ip ;)

On Wed, 26 Oct 2016 at 12:28 Pedro Amorim  wrote:


I have tested this and the stress caused on the server is very severe. It
seems that for every request, a new zebra process is created and the server
will only respond when the last one is finished. This ofc will result in
time outs and eventually a crash in the server.

This is a major critical issue IMO, anyone who knows about this has the
power to deny the service of any Koha online without using any additional
hacking/attacking software.

The Koha I'm working on right now - still in development - is accessed
behind a proxy server, and I will attempt to solve the problem through
that, by limiting the requests from the same origin with very little time
between them. Still, even if I'm successful with this, the problem will
still lie in Koha.

Anyone with some sort of insight is very welcome.

Pedro Amorim

2016-10-26 8:24 GMT+00:00 clint.deckard :


I have had this issue appear today. I have attempted to set up

mod_evasive

for apache but it doesn't seem to have solved the problem.
I would really appreciate some advice.
Clint.


rfblanchard wrote:


Assume a basic opac search:
http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
imit=branch%3A349

This would take about 10 seconds to return the first time.

Assume the user refreshes the results using f5 and keep there finger
there a
moment to long (3s):
This would kill my server for about 1 minute.

Any attacker could easily make the server unresponsive indefinitely by
simply holding f5 on an opac search.

Any recommendations on how to deal with this problem?

here is a sample from top:

Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
%Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa,  0.0 hi,  0.0 si,
0.0
st
KiB Mem:  16465036 total,  1532492 used, 14932544 free,63180 buffers
KiB Swap:  8526844 total,0 used,  8526844 free.   505124 cached
Mem

   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
COMMAND
  7027 peischo+  20   0  416164 162924  12756 S  58.8  1.0   0:26.43
/usr/share/koha
  7009 peischo+  20   0  416800 163524  12756 S  56.5  1.0   0:33.77
/usr/share/koha
  7444 peischo+  20   0  129832  15216   5900 R  37.2  0.1   0:01.12
zebrasrv
  7445 peischo+  20   0  129832  15216   5900 R  35.6  0.1   0:01.07
zebrasrv
  1151 mysql 20   0  886564 181096  10808 S   8.6  1.1   1:27.57

mysqld

  7435 koha  20   0   25892   3272   2528 R   0.3  0.0   0:00.03 top
 1 root  20   0  176144   5044   3096 S   0.0  0.0   0:01.43
systemd
 2 root  20   0   0  0  0 S   0.0  0.0   0:00.00
kthreadd



--
View this message in context: http://koha.1045719.n5.nabble.
com/F5-Attacks-tp5906098.html
Sent from the Koha-general mailing list archive at Nabble.com.
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha


___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha


___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha


___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha

Re: [Koha] F5 Attacks

[Jonathan Druart]

Hi,
I don't think this can/must be fixed on Koha side.
It's a sysadmin duty to take care of that.
I would take a look at fail2ban to parse the web server access logs. But
make sure not to block your X librarians using the same ip ;)

On Wed, 26 Oct 2016 at 12:28 Pedro Amorim  wrote:

> I have tested this and the stress caused on the server is very severe. It
> seems that for every request, a new zebra process is created and the server
> will only respond when the last one is finished. This ofc will result in
> time outs and eventually a crash in the server.
>
> This is a major critical issue IMO, anyone who knows about this has the
> power to deny the service of any Koha online without using any additional
> hacking/attacking software.
>
> The Koha I'm working on right now - still in development - is accessed
> behind a proxy server, and I will attempt to solve the problem through
> that, by limiting the requests from the same origin with very little time
> between them. Still, even if I'm successful with this, the problem will
> still lie in Koha.
>
> Anyone with some sort of insight is very welcome.
>
> Pedro Amorim
>
> 2016-10-26 8:24 GMT+00:00 clint.deckard :
>
> > I have had this issue appear today. I have attempted to set up
> mod_evasive
> > for apache but it doesn't seem to have solved the problem.
> > I would really appreciate some advice.
> > Clint.
> >
> >
> > rfblanchard wrote:
> >
> >> Assume a basic opac search:
> >> http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
> >> imit=branch%3A349
> >>
> >> This would take about 10 seconds to return the first time.
> >>
> >> Assume the user refreshes the results using f5 and keep there finger
> >> there a
> >> moment to long (3s):
> >> This would kill my server for about 1 minute.
> >>
> >> Any attacker could easily make the server unresponsive indefinitely by
> >> simply holding f5 on an opac search.
> >>
> >> Any recommendations on how to deal with this problem?
> >>
> >> here is a sample from top:
> >>
> >> Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
> >> %Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa,  0.0 hi,  0.0 si,
> >> 0.0
> >> st
> >> KiB Mem:  16465036 total,  1532492 used, 14932544 free,63180 buffers
> >> KiB Swap:  8526844 total,0 used,  8526844 free.   505124 cached
> >> Mem
> >>
> >>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
> >> COMMAND
> >>  7027 peischo+  20   0  416164 162924  12756 S  58.8  1.0   0:26.43
> >> /usr/share/koha
> >>  7009 peischo+  20   0  416800 163524  12756 S  56.5  1.0   0:33.77
> >> /usr/share/koha
> >>  7444 peischo+  20   0  129832  15216   5900 R  37.2  0.1   0:01.12
> >> zebrasrv
> >>  7445 peischo+  20   0  129832  15216   5900 R  35.6  0.1   0:01.07
> >> zebrasrv
> >>  1151 mysql 20   0  886564 181096  10808 S   8.6  1.1   1:27.57
> mysqld
> >>  7435 koha  20   0   25892   3272   2528 R   0.3  0.0   0:00.03 top
> >> 1 root  20   0  176144   5044   3096 S   0.0  0.0   0:01.43
> >> systemd
> >> 2 root  20   0   0  0  0 S   0.0  0.0   0:00.00
> >> kthreadd
> >>
> >>
> >>
> >> --
> >> View this message in context: http://koha.1045719.n5.nabble.
> >> com/F5-Attacks-tp5906098.html
> >> Sent from the Koha-general mailing list archive at Nabble.com.
> >> ___
> >> Koha mailing list  http://koha-community.org
> >> Koha@lists.katipo.co.nz
> >> https://lists.katipo.co.nz/mailman/listinfo/koha
> >>
> > ___
> > Koha mailing list  http://koha-community.org
> > Koha@lists.katipo.co.nz
> > https://lists.katipo.co.nz/mailman/listinfo/koha
> >
> ___
> Koha mailing list  http://koha-community.org
> Koha@lists.katipo.co.nz
> https://lists.katipo.co.nz/mailman/listinfo/koha
>
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha

Re: [Koha] F5 Attacks

[Pedro Amorim]

I have tested this and the stress caused on the server is very severe. It
seems that for every request, a new zebra process is created and the server
will only respond when the last one is finished. This ofc will result in
time outs and eventually a crash in the server.

This is a major critical issue IMO, anyone who knows about this has the
power to deny the service of any Koha online without using any additional
hacking/attacking software.

The Koha I'm working on right now - still in development - is accessed
behind a proxy server, and I will attempt to solve the problem through
that, by limiting the requests from the same origin with very little time
between them. Still, even if I'm successful with this, the problem will
still lie in Koha.

Anyone with some sort of insight is very welcome.

Pedro Amorim

2016-10-26 8:24 GMT+00:00 clint.deckard :

> I have had this issue appear today. I have attempted to set up mod_evasive
> for apache but it doesn't seem to have solved the problem.
> I would really appreciate some advice.
> Clint.
>
>
> rfblanchard wrote:
>
>> Assume a basic opac search:
>> http:///cgi-bin/koha/opac-search.pl?q=dog_group_l
>> imit=branch%3A349
>>
>> This would take about 10 seconds to return the first time.
>>
>> Assume the user refreshes the results using f5 and keep there finger
>> there a
>> moment to long (3s):
>> This would kill my server for about 1 minute.
>>
>> Any attacker could easily make the server unresponsive indefinitely by
>> simply holding f5 on an opac search.
>>
>> Any recommendations on how to deal with this problem?
>>
>> here is a sample from top:
>>
>> Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
>> %Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa,  0.0 hi,  0.0 si,
>> 0.0
>> st
>> KiB Mem:  16465036 total,  1532492 used, 14932544 free,63180 buffers
>> KiB Swap:  8526844 total,0 used,  8526844 free.   505124 cached
>> Mem
>>
>>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
>> COMMAND
>>  7027 peischo+  20   0  416164 162924  12756 S  58.8  1.0   0:26.43
>> /usr/share/koha
>>  7009 peischo+  20   0  416800 163524  12756 S  56.5  1.0   0:33.77
>> /usr/share/koha
>>  7444 peischo+  20   0  129832  15216   5900 R  37.2  0.1   0:01.12
>> zebrasrv
>>  7445 peischo+  20   0  129832  15216   5900 R  35.6  0.1   0:01.07
>> zebrasrv
>>  1151 mysql 20   0  886564 181096  10808 S   8.6  1.1   1:27.57 mysqld
>>  7435 koha  20   0   25892   3272   2528 R   0.3  0.0   0:00.03 top
>> 1 root  20   0  176144   5044   3096 S   0.0  0.0   0:01.43
>> systemd
>> 2 root  20   0   0  0  0 S   0.0  0.0   0:00.00
>> kthreadd
>>
>>
>>
>> --
>> View this message in context: http://koha.1045719.n5.nabble.
>> com/F5-Attacks-tp5906098.html
>> Sent from the Koha-general mailing list archive at Nabble.com.
>> ___
>> Koha mailing list  http://koha-community.org
>> Koha@lists.katipo.co.nz
>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>
> ___
> Koha mailing list  http://koha-community.org
> Koha@lists.katipo.co.nz
> https://lists.katipo.co.nz/mailman/listinfo/koha
>
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha

Re: [Koha] F5 Attacks

[clint.deckard]

I have had this issue appear today. I have attempted to set up 
mod_evasive for apache but it doesn't seem to have solved the problem.

I would really appreciate some advice.
Clint.

rfblanchard wrote:

Assume a basic opac search:
http:///cgi-bin/koha/opac-search.pl?q=dog_group_limit=branch%3A349

This would take about 10 seconds to return the first time.

Assume the user refreshes the results using f5 and keep there finger there a
moment to long (3s):
This would kill my server for about 1 minute.

Any attacker could easily make the server unresponsive indefinitely by
simply holding f5 on an opac search.

Any recommendations on how to deal with this problem?

here is a sample from top:

Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
%Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0
st
KiB Mem:  16465036 total,  1532492 used, 14932544 free,63180 buffers
KiB Swap:  8526844 total,0 used,  8526844 free.   505124 cached Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
 7027 peischo+  20   0  416164 162924  12756 S  58.8  1.0   0:26.43
/usr/share/koha
 7009 peischo+  20   0  416800 163524  12756 S  56.5  1.0   0:33.77
/usr/share/koha
 7444 peischo+  20   0  129832  15216   5900 R  37.2  0.1   0:01.12 zebrasrv
 7445 peischo+  20   0  129832  15216   5900 R  35.6  0.1   0:01.07 zebrasrv
 1151 mysql 20   0  886564 181096  10808 S   8.6  1.1   1:27.57 mysqld
 7435 koha  20   0   25892   3272   2528 R   0.3  0.0   0:00.03 top
1 root  20   0  176144   5044   3096 S   0.0  0.0   0:01.43 systemd
2 root  20   0   0  0  0 S   0.0  0.0   0:00.00 kthreadd



--
View this message in context: 
http://koha.1045719.n5.nabble.com/F5-Attacks-tp5906098.html
Sent from the Koha-general mailing list archive at Nabble.com.
___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha

___
Koha mailing list  http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha