Ah! I see in my logs:
Semrushbot ...
petalbot ...
AhrefsBot/7.0 ...
Although it appears to be hitting us just once a second or so. Maybe
that is enough on a 1 core system.
Once I started searching Koha docs for robots.txt I found:
José Anjos 2016-12-22 10:25:27 UTC
I added file robots.txt to /usr/share/koha/opac/htdocs, but makes no
difference.
I've try it too on /usr/share/koha/opac/htdocs/opac-tmpl but problem
persists.
I have non stop requests from SemrushBot, AhrefsBot and Google.
Server is getting high CPU usage and out of memory every 2/3 days
Ref: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=4042#c3
I'll look at firewall solutions tomorrow and see if for a while we can
set login required on OPAC searches.
I'll get back if this fixes the problem.
Thanks
---
Mike Lake
On 2021-04-21 23:23, Coehoorn, Joel wrote:
Check the apache logs and find the web server log corresponding to
those
requests. That should either give you a bot name you can use for a
robots.txt entry or an IP you can block at your firewall.
Joel Coehoorn
Director of Information Technology
York College of Nebraska
On Wed, Apr 21, 2021 at 8:12 AM Mike Lake <[email protected]>
wrote:
Hi
Lately our library instance is peaking at 100% CPU usage. When I login
and use "htop" I can see the opac search and export is, every 30
seconds, starting and using 100% CPU for 15 seconds or so. It then
stops
and 30 seconds later does it again. I'm averaging 80% CPU usage just
from this.
Logging in when the load is 100% I see:
root@asf3:/etc/init.d# ps ax | grep opac
16412 ? R 0:00 /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/opac-export.pl
for several seconds and then
root@asf3:/etc/init.d# ps ax | grep opac
19098 ? R 0:02 /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/opac-search.pl
19100 ? R 0:01 /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/opac-search.pl
This problem started a few weeks ago. I'm using a dedicated virtual
machine with 1 core and 2 GB RAM, 512 MB swap. The VM control panel
shows its averaging 800 MB used out of 2GB but swap is fully used.
This system had previously been adequate so something seems to have
changed.
What advice for narrowing down the problem? It's not a large
installation.
Any links to hardware/cpu/mem/swap requirements?
--
Mike Lake
_______________________________________________
Koha mailing list http://koha-community.org
[email protected]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________
Koha mailing list http://koha-community.org
[email protected]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________
Koha mailing list http://koha-community.org
[email protected]
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
