28.03.2019, 23:58, "Alexey Proskuryakov" <a...@webkit.org>:
> Hello,
> The robots.txt file that we have on bugs.webkit.org currently allows search 
> engines access to individual bug pages, but not to any bug lists. As a 
> result, search engines and the Internet Archive only index bugs that were 
> filed before robots.txt changes a few years ago, and bugs that are directly 
> linked from webpages elsewhere. These bugs are where most spam content 
> naturally ends up on.
> This is quite wrong, as indexing just a subset of bugs is not beneficial to 
> anyone other than spammers. So we can go in either direction:
> 1. Allow indexers to enumerate bugs, thus indexing all of them.
> Seems reasonable that people should be able to find bugs using search engines.

Yes, and it may give better result even than searching bugzilla directly

>On the other hand, we'll need to do something to ensure that indexers don't 
>destroy Bugzilla performance,

This can be solved by caching

>and of course spammers will love having more flexibility.

rel="nofollow" on all links in comments should be enough to make spamming 

> 2. Block indexing completely.
> Seems like no one was bothered by lack of indexing on new bugs so far.

That's survival bias - if nobody can find relevant bugs, nobody will ever 

> Thoughts?
> For reference, here is the current robots.txt content:
> $ curl https://bugs.webkit.org/robots.txt
> User-agent: *
> Allow: /index.cgi
> Allow: /show_bug.cgi
> Disallow: /
> Crawl-delay: 20
> - Alexey
> - Alexey
> _______________________________________________
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev


webkit-dev mailing list

Reply via email to