Re: [sqlite] Spiders vandalizing the wiki
On 25 Nov 2003, at 12:48, D. Richard Hipp wrote: In the past couple of days, I've been having problems with spiders vandalizing the Wiki at http://www.sqlite.org/cvstrac/wiki. The damage (so far) has been relatively minor and easy to fix. But I've been monitoring these spiders for a while and notice that they are becoming increasingly aggressive. If you have any suggestions on what to do about them, I'd like to hear from you. My suggestion is to use a trap. A robots.txt guarded area that users won't click on but occurs early at the top of the main index page. If any IP visits that page install a firewall or httpd.conf block of some description. Beyond that, ensuring that to change a page requires a secure hash auth system (i.e. can't submit earlier than 5 seconds after downloading the edit page, and can't submit later than N minutes) works very well. Matt. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [sqlite] Spiders vandalizing the wiki
Good suggestions, IMO, Peter. I normally really hate this, but you could try to mangle the email adresses they look for with some JavaScript gimmicks using document.write. OTOH this sucks big time, because it will make the site harder to use for those who surf with JavaScript disabled or without JavaScript support at all. I'd probably go this way, at least for a temporary solution. -- Gerhard - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [sqlite] Spiders vandalizing the wiki
D. Richard Hipp wrote: If you have any suggestions on what to do about them, I'd like to hear from you. Block the entire IP range, for say 2 weeks at a time. I'm guessing that these spiders are coming from spammers looking to harvest email addresses. Last nights attack came from 61.51.123.205. No reverse DNS is available. Can anybody provide any insight into where the attacks are coming from? ~> whois 61.51.123.205 ... inetnum: 61.48.0.0 - 61.51.255.255 ... role: CNCGroup Hostmaster e-mail: [EMAIL PROTECTED] address: No.156,Fu-Xing-Men-Nei Street, address: Beijing,100031,P.R.China nic-hdl: CH455-AP phone:+86-10-68019956 fax-no: +86-10-68019958 country: CN admin-c: CH444-AP tech-c: CH444-AP changed: [EMAIL PROTECTED] 20031016 mnt-by: MAINT-CNCGROUP source: APNIC http://openrbl.org/#61.51.123.205 and http://groups.google.co.uk/groups?q=abuse+Beijing+province will tell you that they are spammer friendly, so however loud you shout at their abuse address, you are unlikely to get a satisfactory answer, although I'd be interested (but highly sceptical) to see if you get any answer from them. But if the attackers start doing a better job of disguising their spiders, my detection might fail and this defense would become ineffective. They will, as they can usually masquerade as any user agent, ignore robots.txt, and if they really want to, will use proxies to disguise their source. In the end it will just become a game of whack a mole, till they get fed up and leech on someone else's site. Regards P. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [sqlite] Spiders vandalizing the wiki
Google won't submit forms. Robots can't read. Require a challenge before allowing submissions- whether it be an email-based challenge, or an image that contains distorted text. I don't know if there are any blind users of SQLite but they would probably prefer the former. On Tue, 2003-11-25 at 07:48, D. Richard Hipp wrote: > In the past couple of days, I've been having problems with > spiders vandalizing the Wiki at http://www.sqlite.org/cvstrac/wiki. > The damage (so far) has been relatively minor and easy to fix. > But I've been monitoring these spiders for a while and notice > that they are becoming increasingly aggressive. > > If you have any suggestions on what to do about them, I'd > like to hear from you. > > I'm guessing that these spiders are coming from spammers looking > to harvest email addresses. Last nights attack came from > 61.51.123.205. No reverse DNS is available. Can anybody > provide any insight into where the attacks are coming from? > > I've added some spider detection logic to cvstrac which > shuts down large portions of the website to spiders. I > used to have problems with spiders pulling out every historical > version of every source code file every night. I've solved > that now. But I've been leaving the wiki open to spiders > because I *want* Google spiders to visit the wiki. > > My next defense will be to try to make the wiki read-only > to spiders. That will prevent the vandalism I'm been seeing > for the past couple of days, assuming my spider detection > continues to work. But if the attackers start doing a better > job of disguising their spiders, my detection might fail > and this defense would become ineffective. > > Any other ideas? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [sqlite] Spiders vandalizing the wiki
D. Richard Hipp wrote: In the past couple of days, I've been having problems with spiders [...] You could use a robots.txt to guard against those spiders that behave well. If the misbehaving spiders use a certain distinguishable User-Agent header, you could block that. -- Gerhard - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
