Thanks for your replies. In my panic I seemed to miss some obvious evidence, and only had to look at the user agent to see that the requests are coming from Google's own crawlers. Adding the url to robots.txt seems to put an end the local requests. I'm still getting some from "Feedfetcher-Google", apparently an RSS reader of Google's that ignores robots.txt as it's used by humans. I've started responding to these requests with a 410 as suggested in Google's docs for Feedfetcher. Here's the log for one of them:
66.249.92.99 - - [03/Feb/2015:12:39:42 -0800] "GET /rss?page=http%3A%2F%2Fwww.torrentman.net/commnet.html&opts=8498&num=20&mod=%241%24wq1rdBcg%24Z17SROlxqlo6.pL8QVJi0.&v=1421906055999 HTTP/1.1" 200 344 - "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=15992684286155802798)" "www.htmlcommentbox.com" ms=149 cpu_ms=21 cpm_usd=0.000038 app_engine_release=1.9.17 instance= 00c61b117c6945bbc0f8c36160688fcec149d2 <https://appengine.google.com/instances?app_id=s~easycomment-hrd&version_id=1.381982625227369663&key=00c61b117c6945bbc0f8c36160688fcec149d2#00c61b117c6945bbc0f8c36160688fcec149d2> On Monday, 2 February 2015 12:05:04 UTC-8, Clark Van Oyen wrote: > > > Hi everyone. I'm stumped by something I'm seeing happen to my app on > Google App Engine - tried asking on Stack Overflow with the App Engine tag > but my question wasn't received well because it's not a programming > question. I'd be very happy if anyone can shed some light on this though :) > > > <https://lh6.googleusercontent.com/-yxqcg9RRk74/VM_YYN0eeoI/AAAAAAAAAb8/rXQQAZ572hM/s1600/Capture.PNG> > > I'm receiving a huge number of requests from IP addresses starting with > 10. Afaik, this is a local IP. So, my question is why would Google's > network be swarming my app with requests? It's costing me quite a few bucks > in extra quota, so is cause for concern. I've attached the log I'm looking > at. Thanks in advance if anyone has any idea what's going on! > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/52a4794e-1016-46c7-8588-b5a2148ddf86%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
