Hi Mark and Hugo
Many thanks for your hints! I have now done the following.
1. I created a file "/usr/share/koha/opac/htdocs/robots.txt" containing
this:
Sitemap: sitemapindex.xml
User-agent: *
Disallow: /cgi-bin/
2. I generated a Koha sitemap using the seemingly undocumented Perl
script "sitemap.pl" (according to
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11190) which
created the file "/usr/share/koha/opac/htdocs/sitemapindex.xml" and the
file "/usr/share/koha/opac/htdocs/sitemap0001.xml" containing the URLs.
3. Even after a complete reboot of the host the "opac-search.pl"
processes were still there, appearing immediately after the reboot!
4. I went to Google Webmaster Tools where I downloaded the HTML
confirmation file "googleb56bd3db2af352b1.html" and placed it in
"/usr/share/koha/opac/htdocs" as well. I also followed the steps given
on the Wemaster Tools page, i. e. I called the URL and I confirmed the
download.
5. Even after a complete reboot of the host the "opac-search.pl"
processes were still there, appearing immediately after the reboot!
6. I then installed the Uncomplicated Firewall / UFW where I applied the
following rules and enabled it:
# ufw status
Status: active
To Action From
-- ------ ----
22/tcp ALLOW Anywhere
80/tcp ALLOW Anywhere
8080/tcp ALLOW Anywhere
Anywhere DENY 66.249.64.32
But however this is possible, still Googlebot is crawling and eating
CPU! This can be seen in the log file "plack.log" where hundreds and
thousands of lines like the following can be seen:
66.249.64.32 - - [03/May/2017:15:48:28 +0200] "GET
/opac/opac-authoritiesdetail.pl?authid=12872 HTTP/1.1" 200 17703 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
And I also found another bot:
62.138.14.218 - - [03/May/2017:15:48:29 +0200] "GET
/opac/opac-search.pl?q=se,phr:%22Zeitreise%22 HTTP/1.1" 200 54672 "-"
"Linguee Bot (http://www.linguee.com/bot; b...@linguee.com)"
Now what I don't understand is how Googlebot (66.249.64.32) can access
the webserver even if it is blocked by UFW?!
9. Already quite desperate I finally executed the following line to drop
all packets from 66.249.64.32.
# iptables -I INPUT -s 66.249.64.32 -j DROP
# iptables -I INPUT -s 62.138.14.218 -j DROP
And yes - this actually stopped these harassing bots.
But of course, next was this:
66.249.64.35 - - [03/May/2017:15:59:21 +0200] "GET
/opac/opac-authoritiesdetail.pl?authid=16429 HTTP/1.1" 200 17661 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
I also dropped this IP address and now - finally! - the OPAC search for
the normal user works as fast as expected.
In fact I can't believe I should be the only one experiencing this
behavior (especially since the stuff about creating "sitemap.pl" is
quite hidden and however undocumented in the Koha manual).
The other thing is people usually say it's a good thing to be indexed by
Google. Today however, I won't agree. Maybe tomorrow, I will then try to
delete the rule which drops the Google packets and I really hope Google
will then do what it is told to do in "robots.txt", using the Koha sitemap.
So all this just for the record - maybe it will help someone in the future.
Best wishes: Michael
--
Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis
Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz
T 0041 (0)61 261 55 61 · E m...@adminkuhn.ch · W www.adminkuhn.ch
_______________________________________________
Koha mailing list http://koha-community.org
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha
