Re: [Wikimedia-l] Internet archive and automatic retroactive robots.txt (was Re: Internet archive and strategy survey (was Re: 24 TB for User:Dispenser on Tool Labs please))

2014-07-07 Thread David Gerard
On 7 July 2014 21:08, Brad Jorsch (Anomie) wrote: > On Mon, Jul 7, 2014 at 1:49 PM, Federico Leva (Nemo) > wrote: >> Brad Jorsch (Anomie), 07/07/2014 17:37: >> > And the robots.txt for the new >> > version of the site denies everything, likely because the new owners >> don't >> > want the redire

Re: [Wikimedia-l] Internet archive and automatic retroactive robots.txt (was Re: Internet archive and strategy survey (was Re: 24 TB for User:Dispenser on Tool Labs please))

2014-07-07 Thread Brad Jorsch (Anomie)
On Mon, Jul 7, 2014 at 1:49 PM, Federico Leva (Nemo) wrote: > Brad Jorsch (Anomie), 07/07/2014 17:37: > > And the robots.txt for the new > > version of the site denies everything, likely because the new owners > don't > > want the redirects or other old content showing up in Google searches. > Bu

Re: [Wikimedia-l] Internet archive and automatic retroactive robots.txt (was Re: Internet archive and strategy survey (was Re: 24 TB for User:Dispenser on Tool Labs please))

2014-07-07 Thread Federico Leva (Nemo)
Brad Jorsch (Anomie), 07/07/2014 17:37: > And the robots.txt for the new > version of the site denies everything, likely because the new owners don't > want the redirects or other old content showing up in Google searches. But > this has the unfortunate side effect that IA removes all the old conte

[Wikimedia-l] Internet archive and automatic retroactive robots.txt (was Re: Internet archive and strategy survey (was Re: 24 TB for User:Dispenser on Tool Labs please))

2014-07-07 Thread Brad Jorsch (Anomie)
On Mon, Jul 7, 2014 at 5:21 AM, James Salsman wrote: > Kevin Gorman wrote: > > > Regarding the IA: they have a significant interest in working with the > > Wikimedia projects, a lot more experience than the Wikimedia projects > have > > caching absolutely tremendous quantities of data, a willinne