Hi Christoper, Henry, all: The main problem is, imho:
1. the basic attitude of Semantic Web research that the works done in the past or in other communities were irrelevant historical relicts (databases, middleware, EDI) and that the old fellows were simply too stupid to understand the "power of semantics that will make machines understand our data with ease", just by adding a bit of OWL 2 DL axioms, properly dereferencing data entity URIs according to their nice data publishing guidelines that turn toy examples into a magic art; 2. this implanted into the heads of eager young people who excelled in the "AI for freshmen", "complexity theory", and "theorem proving" exams and who now apply the gained self-confidence from a small subset of life to a broader range of fields, and 3. allocating a lot of money (EU funding) and an abundance of IT resources (university servers, bandwidth,...) to those folks. This mindset is the petri dish for stupid crawlers as described. Unfortunately, authentication techniques won't help protecting typical site-owners from the dangerous creatures written by Semantic Web researchers gathering data for the evaluation of their ISWC 2011 submission, because the site-owners at www.godaddy.com know nothing about WebID at this point ;-) Martin PS: I will not release the IP ranges from which the trouble originated, but rest assured, there were top research institutions among them. On Jun 21, 2011, at 10:48 AM, Christopher Gutteridge wrote: > Would some kind of caching crawler mitigate this issue? Have someone write a > well behaved crawler which allowed you to download a recent .ttl.tgz of > various sites. Of course, that assumes the student is able to find such a > cache. > > Asking people nicely will only work in a very small community. > > Henry Story wrote: >> A solution to stupid crawlers would be to put the linked data behind https >> endpoints, and use WebID >> for authentication. You could still allow everyone access, but at least you >> would force the crawler to identify >> himself, and use these WebIDs to learn who was making the crawler. This >> could then be used as a piece of the evaluation of the quality of a semantic >> web stack. >> >> Henry >> >> 10 minute intro to WebID >> http://bblfish.net/blog/2011/05/25/ >> (in browsers, but the browser is not really necessary) >> >> On 21 Jun 2011, at 09:49, Martin Hepp wrote: >> >> >> >>> Hi all: >>> >>> For the third time in a few weeks, we had massive complaints from >>> site-owners that Semantic Web crawlers from Universities visited their >>> sites in a way close to a denial-of-service attack, i.e., crawling data >>> with maximum bandwidth in a parallelized approach. >>> >>> It's clear that a single, stupidly written crawler script, run from a >>> powerful University network, can quickly create terrible traffic load. >>> >>> Many of the scripts we saw >>> >>> - ignored robots.txt, >>> - ignored clear crawling speed limitations in robots.txt, >>> - did not identify themselves properly in the HTTP request header or lacked >>> contact information therein, >>> - used no mechanisms at all for limiting the default crawling speed and >>> re-crawling delays. >>> >>> This irresponsible behavior can be the final reason for site-owners to say >>> farewell to academic/W3C-sponsored semantic technology. >>> >>> So please, please - advise all of your colleagues and students to NOT write >>> simple crawler scripts for the billion triples challenge or whatsoever >>> without familiarizing themselves with the state of the art in "friendly >>> crawling". >>> >>> Best wishes >>> >>> Martin Hepp >>> >>> >>> >> >> Social Web Architect >> >> http://bblfish.net/ >> >> >> >> >> > > -- > Christopher Gutteridge -- > http://id.ecs.soton.ac.uk/person/1248 > > > You should read the ECS Web Team blog: > http://blogs.ecs.soton.ac.uk/webteam/
