Re: [google-appengine] unusual traffic from your computer network
This just impacted our live site as well. Users going to domain.com.au are blocked, users going to www.domain.com.au are not. We're using cloudflare, everything seems ok there. We do have monitoring (Pingdom) which hits the naked domain, not sure if this is what triggered this. Either way, it appears to block out all users. On Thursday, April 4, 2013 3:05:43 AM UTC+11, Jeff Schnitzer wrote: If you've been reading about my troubles with this issue in the past, you're going to laugh at my suggestion: Use CloudFlare. CF's IP blocks are apparently whitelisted by Google now and won't trip Google's alarms. You can disable CF's threat monitoring and response system - and even better, you get metrics so that you have some idea when/why it's being tripped when it is enabled. This seems like a silly way of routing around Google's undocumented and unwanted service, but it should get the job done. Client - Client's Proxy - CF - GAE Jeff On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren pe...@treehouselogic.comjavascript: wrote: I see posts about this issue going back years, so sorry if I'm kicking a dead horse, but I haven't been able to find any resolution. (I’ve posted this message twice on a new account, once 5 days ago and once 3 days ago, and neither message has actually made it into the forum. So I’m trying my old account. Sorry if this post ends up getting duplicated.) We have a paid app on app engine we've been using to serve a commercial web app for 3 years. The app is mapped to a custom domain via Google Apps. I think that’s the crux here. We have one application that serves different content for different clients. Each of our clients has reverse proxy set up on their web server to fetch the content from our custom domain on app engine. We use reverse proxy simply to mask our domain to the clients' domains. There is no caching, and the reverse proxy is Apache2 with out of the box configuration. On March 26, after 2 years of happily serving content to a particular client's server, Google for some reason decided that this server was violating its Terms of Service and started denying content to that client's reverse proxy, redirecting users to the www.google.com/sorry/misc page with the message that: Our systems have detected unusual traffic from your computer network. This of course caused our application to be totally unusable. We sent requests to Google for more information and heard nothing. The next day App Engine decided that particular server was ok again and resumed serving our content to the problem server. Then again on March 30 Google decided to ban this particular server. Our app is very low volume, averaging about .05 requests/second. There were no traffic spikes that day. There were no configuration changes to the reverse proxy or any of our infrastructure. The only information I can find on the issue is here: http://support.google.com/websearch/bin/answer.py?hl=enanswer=86640rd=1. That page suggests that the client's server was doing one of these things: •Sending automated queries •Using software that sends queries to Google to determine how a website or webpage ranks on Google for various queries •'Meta searching' Google •Performing 'offline' searches on Google I could find no evidence of any requests being sent to Google search. There were open requests to one of Google's nameservers, presumably to look up our app's ip from its Google Apps custom domain. Surely that isn't a violation of Terms of Service. We found no malware on the machine. So at this point we have no idea why Google stopped serving the content to that particular server, or why it resumed service. Additionally all our other clients' reverse proxies continued to work fine. There was even another reverse proxy successfully fetching the same content that Google was denying to the other proxy. Switching to the yyy.appspot.com domain from our custom domain seems to fix the problem, so I really suspect the problem is with the domain mapping. I sent a support request to Google Apps, and of course they said they couldn’t look into it, stating: “You are correct that the custom domain mapping is created in the Google Apps Control Panel and is handled there however any issues with the mapping of Google App Engine apps needs to be investigated and supported by the App Engine team.” So I’m left wondering why Google has denied requests from this particular server after 2 years when nothing has changed. And yet Google continues to happily serve our other clients who are using the exact same proxy settings on other machines. Searching through previous posts, the best information I can gather is that maybe our proxies headers are
Re: [google-appengine] unusual traffic from your computer network
Funny, because in the past CloudFlare was getting banned quickly for unusually high traffic. I suppose they have been whitelisted since. On Wednesday, April 3, 2013 6:05:43 PM UTC+2, Jeff Schnitzer wrote: If you've been reading about my troubles with this issue in the past, you're going to laugh at my suggestion: Use CloudFlare. CF's IP blocks are apparently whitelisted by Google now and won't trip Google's alarms. You can disable CF's threat monitoring and response system - and even better, you get metrics so that you have some idea when/why it's being tripped when it is enabled. This seems like a silly way of routing around Google's undocumented and unwanted service, but it should get the job done. Client - Client's Proxy - CF - GAE Jeff On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren pe...@treehouselogic.comjavascript: wrote: I see posts about this issue going back years, so sorry if I'm kicking a dead horse, but I haven't been able to find any resolution. (I’ve posted this message twice on a new account, once 5 days ago and once 3 days ago, and neither message has actually made it into the forum. So I’m trying my old account. Sorry if this post ends up getting duplicated.) We have a paid app on app engine we've been using to serve a commercial web app for 3 years. The app is mapped to a custom domain via Google Apps. I think that’s the crux here. We have one application that serves different content for different clients. Each of our clients has reverse proxy set up on their web server to fetch the content from our custom domain on app engine. We use reverse proxy simply to mask our domain to the clients' domains. There is no caching, and the reverse proxy is Apache2 with out of the box configuration. On March 26, after 2 years of happily serving content to a particular client's server, Google for some reason decided that this server was violating its Terms of Service and started denying content to that client's reverse proxy, redirecting users to the www.google.com/sorry/misc page with the message that: Our systems have detected unusual traffic from your computer network. This of course caused our application to be totally unusable. We sent requests to Google for more information and heard nothing. The next day App Engine decided that particular server was ok again and resumed serving our content to the problem server. Then again on March 30 Google decided to ban this particular server. Our app is very low volume, averaging about .05 requests/second. There were no traffic spikes that day. There were no configuration changes to the reverse proxy or any of our infrastructure. The only information I can find on the issue is here: http://support.google.com/websearch/bin/answer.py?hl=enanswer=86640rd=1. That page suggests that the client's server was doing one of these things: •Sending automated queries •Using software that sends queries to Google to determine how a website or webpage ranks on Google for various queries •'Meta searching' Google •Performing 'offline' searches on Google I could find no evidence of any requests being sent to Google search. There were open requests to one of Google's nameservers, presumably to look up our app's ip from its Google Apps custom domain. Surely that isn't a violation of Terms of Service. We found no malware on the machine. So at this point we have no idea why Google stopped serving the content to that particular server, or why it resumed service. Additionally all our other clients' reverse proxies continued to work fine. There was even another reverse proxy successfully fetching the same content that Google was denying to the other proxy. Switching to the yyy.appspot.com domain from our custom domain seems to fix the problem, so I really suspect the problem is with the domain mapping. I sent a support request to Google Apps, and of course they said they couldn’t look into it, stating: “You are correct that the custom domain mapping is created in the Google Apps Control Panel and is handled there however any issues with the mapping of Google App Engine apps needs to be investigated and supported by the App Engine team.” So I’m left wondering why Google has denied requests from this particular server after 2 years when nothing has changed. And yet Google continues to happily serve our other clients who are using the exact same proxy settings on other machines. Searching through previous posts, the best information I can gather is that maybe our proxies headers are malformed and Google doesn't like them. Why would Google randomly complain after 2 years of happily serving content to this same proxy with the same headers? Previous posts described
Re: [google-appengine] unusual traffic from your computer network
If you've been reading about my troubles with this issue in the past, you're going to laugh at my suggestion: Use CloudFlare. CF's IP blocks are apparently whitelisted by Google now and won't trip Google's alarms. You can disable CF's threat monitoring and response system - and even better, you get metrics so that you have some idea when/why it's being tripped when it is enabled. This seems like a silly way of routing around Google's undocumented and unwanted service, but it should get the job done. Client - Client's Proxy - CF - GAE Jeff On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren pe...@treehouselogic.com wrote: I see posts about this issue going back years, so sorry if I'm kicking a dead horse, but I haven't been able to find any resolution. (I’ve posted this message twice on a new account, once 5 days ago and once 3 days ago, and neither message has actually made it into the forum. So I’m trying my old account. Sorry if this post ends up getting duplicated.) We have a paid app on app engine we've been using to serve a commercial web app for 3 years. The app is mapped to a custom domain via Google Apps. I think that’s the crux here. We have one application that serves different content for different clients. Each of our clients has reverse proxy set up on their web server to fetch the content from our custom domain on app engine. We use reverse proxy simply to mask our domain to the clients' domains. There is no caching, and the reverse proxy is Apache2 with out of the box configuration. On March 26, after 2 years of happily serving content to a particular client's server, Google for some reason decided that this server was violating its Terms of Service and started denying content to that client's reverse proxy, redirecting users to the www.google.com/sorry/misc page with the message that: Our systems have detected unusual traffic from your computer network. This of course caused our application to be totally unusable. We sent requests to Google for more information and heard nothing. The next day App Engine decided that particular server was ok again and resumed serving our content to the problem server. Then again on March 30 Google decided to ban this particular server. Our app is very low volume, averaging about .05 requests/second. There were no traffic spikes that day. There were no configuration changes to the reverse proxy or any of our infrastructure. The only information I can find on the issue is here: http://support.google.com/websearch/bin/answer.py?hl=enanswer=86640rd=1. That page suggests that the client's server was doing one of these things: •Sending automated queries •Using software that sends queries to Google to determine how a website or webpage ranks on Google for various queries •'Meta searching' Google •Performing 'offline' searches on Google I could find no evidence of any requests being sent to Google search. There were open requests to one of Google's nameservers, presumably to look up our app's ip from its Google Apps custom domain. Surely that isn't a violation of Terms of Service. We found no malware on the machine. So at this point we have no idea why Google stopped serving the content to that particular server, or why it resumed service. Additionally all our other clients' reverse proxies continued to work fine. There was even another reverse proxy successfully fetching the same content that Google was denying to the other proxy. Switching to the yyy.appspot.com domain from our custom domain seems to fix the problem, so I really suspect the problem is with the domain mapping. I sent a support request to Google Apps, and of course they said they couldn’t look into it, stating: “You are correct that the custom domain mapping is created in the Google Apps Control Panel and is handled there however any issues with the mapping of Google App Engine apps needs to be investigated and supported by the App Engine team.” So I’m left wondering why Google has denied requests from this particular server after 2 years when nothing has changed. And yet Google continues to happily serve our other clients who are using the exact same proxy settings on other machines. Searching through previous posts, the best information I can gather is that maybe our proxies headers are malformed and Google doesn't like them. Why would Google randomly complain after 2 years of happily serving content to this same proxy with the same headers? Previous posts described this problem as a landmine, where stepping in the wrong place can trigger it. Seems more like a surprise missile attack to me because we were simply walking the same path we'd walked every day for 2 years when everything blew up. Obviously this is totally unacceptable. We can't very well offer a commercial service to clients with the caveat that it might blow up at any time, and we have no idea when or why. I also don't understand the
