Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-11 Thread Greg Stark
On Wed, Jul 10, 2013 at 9:36 AM, Magnus Hagander mag...@hagander.net wrote: We already run this, that's what we did to make it survive at all. The problem is there are so many thousands of different URLs you can get to on that site, and google indexes them all by default. There's also

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-11 Thread Andres Freund
On 2013-07-11 14:43:21 +0100, Greg Stark wrote: On Wed, Jul 10, 2013 at 9:36 AM, Magnus Hagander mag...@hagander.net wrote: We already run this, that's what we did to make it survive at all. The problem is there are so many thousands of different URLs you can get to on that site, and google

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-11 Thread Magnus Hagander
On Thu, Jul 11, 2013 at 3:43 PM, Greg Stark st...@mit.edu wrote: On Wed, Jul 10, 2013 at 9:36 AM, Magnus Hagander mag...@hagander.net wrote: We already run this, that's what we did to make it survive at all. The problem is there are so many thousands of different URLs you can get to on that

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-10 Thread Craig Ringer
On 07/09/2013 11:30 PM, Andres Freund wrote: On 2013-07-09 16:24:42 +0100, Greg Stark wrote: I note that git.postgresql.org's robot.txt refuses permission to crawl the git repository: http://git.postgresql.org/robots.txt User-agent: * Disallow: / I'm curious what motivates this. It's

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-10 Thread Dave Page
On Wed, Jul 10, 2013 at 9:25 AM, Craig Ringer cr...@2ndquadrant.com wrote: On 07/09/2013 11:30 PM, Andres Freund wrote: On 2013-07-09 16:24:42 +0100, Greg Stark wrote: I note that git.postgresql.org's robot.txt refuses permission to crawl the git repository:

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-10 Thread Magnus Hagander
On Wed, Jul 10, 2013 at 10:25 AM, Craig Ringer cr...@2ndquadrant.com wrote: On 07/09/2013 11:30 PM, Andres Freund wrote: On 2013-07-09 16:24:42 +0100, Greg Stark wrote: I note that git.postgresql.org's robot.txt refuses permission to crawl the git repository:

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-09 Thread Andres Freund
On 2013-07-09 16:24:42 +0100, Greg Stark wrote: I note that git.postgresql.org's robot.txt refuses permission to crawl the git repository: http://git.postgresql.org/robots.txt User-agent: * Disallow: / I'm curious what motivates this. It's certainly useful to be able to search for

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-09 Thread Andrew Dunstan
On 07/09/2013 11:24 AM, Greg Stark wrote: I note that git.postgresql.org's robot.txt refuses permission to crawl the git repository: http://git.postgresql.org/robots.txt User-agent: * Disallow: / I'm curious what motivates this. It's certainly useful to be able to search for commits. I

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-09 Thread Dimitri Fontaine
Andres Freund and...@2ndquadrant.com writes: Gitweb is horribly slow. I don't think anybody with a bigger git repo using gitweb can afford to let all the crawlers go through it. What's blocking alternatives to be considered? I already did mention cgit, which has the advantage to clearly show

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-09 Thread Magnus Hagander
On Tue, Jul 9, 2013 at 5:30 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-07-09 16:24:42 +0100, Greg Stark wrote: I note that git.postgresql.org's robot.txt refuses permission to crawl the git repository: http://git.postgresql.org/robots.txt User-agent: * Disallow: / I'm

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-09 Thread Magnus Hagander
On Tue, Jul 9, 2013 at 5:56 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Andres Freund and...@2ndquadrant.com writes: Gitweb is horribly slow. I don't think anybody with a bigger git repo using gitweb can afford to let all the crawlers go through it. What's blocking alternatives to be

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-09 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Tue, Jul 9, 2013 at 5:56 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: What's blocking alternatives to be considered? I already did mention cgit, which has the advantage to clearly show the latest patch on all the active branches in its

Re: [HACKERS] robots.txt on git.postgresql.org

2013-07-09 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: Oh, and we need stable wheezy packages for them, or we'll be paying even more in maintenance. AFAICT, there aren't any for cgit, but maybe I'm searching for the wrong thing.. Seems to be a loser on that front too. -- Dimitri Fontaine