I'd be very surprised if Google _automatically_ took any notice of anything in an HTTP header to relax protection against what they consider harvesting of data because all HTTP headers can be set to anything: that is, if I wanted to suck Google dry of bib data, I could simply pretend to be forwarding requests for "real" clients behind a NAT barrier.
But they may well investigate such cases and configure their traffic monitoring software for known legitimate proxies. Kent Fitch On Wed, Mar 19, 2008 at 3:29 AM, Joe Hourcle <[EMAIL PROTECTED]> wrote: > On Tue, 18 Mar 2008, Jonathan Rochkind wrote: > > > Wait, now ALL of your clients calls are coming from one single IP? > > Surely that will trigger Googles detectors, if the NAT did. Keep us > > updated though. > > I don't know what Peter's exact implementation is, but they might relax > the limits when they see an 'X-Forwarded-For' header, or something else to > suggest it's coming through a proxy. It used to be pretty common when > writing rate limiting code to use X-Forwarded-For in place of HTTP_ADDR so > you didn't accidentally ban groups behind proxies. (of course, I don't > know if the X-Forwarded-For value is something that's not routable (in > 10/8), or the NAT IP, so it might still look like 1 IP address behind a > proxy) > > Also, by using a caching proxy (if the responses are cachable), the total > number of requests going to Google might be reduced. > > I would assume they'd need to have some consideration for proxies, as I > remember the days when AOL's proxy servers channeled all requests through > less than a dozen unique IP addresses. (or at least, those were the only > ones hitting my servers) > > -Joe >