On 09/12/2011, at 2:59 AM, Lluís Batlle i Rossell wrote: > On Thu, Dec 08, 2011 at 05:49:16PM +0100, Lluís Batlle i Rossell wrote: >> On Thu, Dec 08, 2011 at 05:40:08PM +0100, Lluís Batlle i Rossell wrote: >>> How can it be that it finds the links? That URL is valid since one hour or >>> two >>> ago; before, it never existed. >> >> 1st point I found: timeline.rss >> 2nd point: the (apparently) linkless checkin pages contain three links >> always: >> [show unified diffs | show side-by-side diffs | patch] >> >> Let's see how far the bot goes, still. > > Not further, by now. But I want to tell that I once had a repository with > links > opened for everyone, months ago. I disabled the links, but googlebot > remembered > all URLs, so it was still getting all annotates and so. > > I then disabled all access unless logged in, and although months passed, > googlebot still remembers all URLs. So, it goes at every old url, gets a > redirection to the login page (with a different g= parameter in a GET > parameter > for each form), and googlebot thinks that's the final page. > > So, for months, I'm having all that traffic of googlebot asking those > thousands > of URLs and for each, the redirect with different 'g=' parameters. And fossil > giving the login page for all. > > How can I get rid of all that traffic, other than changing the whole url of > the > project? > > (oth, I'd like timeline.rss and the checkin info pages not to have links, or > somehow that they could be disabled). > > And sorry for the multiple messages to the mailing list, instead of > summarizing > all into one.
Why not just add a robots.txt? -- µWeb: Embedded Web Framework - http://uweb.workware.net.au/ WorkWare Systems Pty Ltd W: www.workware.net.au P: +61 434 921 300 E: [email protected] F: +61 7 3391 6002 _______________________________________________ fossil-users mailing list [email protected] http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
