On Thu, May 7, 2009 at 11:10 AM, Brian Eaton <bea...@google.com> wrote:
> On Thu, May 7, 2009 at 10:57 AM, John Hjelmstad <fa...@google.com> wrote: > > None of which I'm aware. The RMR relay is (receiver's) > > protocol://host:port/robots.txt, which doesn't even need to exist or be > > served "correctly" (404/500 OK). It would be nice for this file to be > > heavily cached but at least it's a one-time load. > > There are a couple of potential gotchas you might want to investigate. > > - no-store cache control headers Don't appear to affect this. > > - 302s from https to http > - 302s in general 302s to an alternate protocol/domain/port would likely be a problem. > > - 401s with WWW-Authenticate ...as would 401s on account of auth popups. This is one reason for choosing robots.txt though -- for what purpose is robots.txt but for anyone to access it? Arguably a robots.txt hidden behind 401 is a misconfiguration, no? (optimistically) > > - 403s Doesn't seem to affect things. Thanks, John
