Hopefully they will stop once they realizes they're being blocked - if it's a
legitimate service. If they are not, and they are reading this thread, they
will probably just change their user agent, though. The only way to stop them
from reaching your app entirely would be to use a service like CloudFlare. I
have no personal experience with CloudFlare, however, so I can not state
definitively that it will solve your problem. And adding an additional layer
can result in it's own issues, as demonstrated by the August 1st CloudFlare
block.
- Kyle
On Aug 6, 2012, at 1:58 PM, Kate wrote:
> But I still don't like them hitting my site every 500 ms! e.g.
> 2012-08-06 13:56:02.725 / 401 32ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:56:02.279 / 401 70ms 0kb curl/7.19.7 (i386-redhat-linux-gnu)
> libcurl/7.19.7 NSS/3.12.6.2 zlib/1.2.3 libidn/1.9 libssh2/1.2.4
>
> 2012-08-06 13:55:57.921 / 401 29ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:55.403 / 401 9ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:54.323 / 401 52ms 0kb curl/7.19.7 (i386-redhat-linux-gnu)
> libcurl/7.19.7 NSS/3.12.6.2 zlib/1.2.3 libidn/1.9 libssh2/1.2.4
>
> 2012-08-06 13:55:54.283 / 401 33ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:52.814 / 401 82ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:52.437 / 401 50ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:49.063 / 401 33ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:45.986 / 401 10ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:43.610 / 401 18ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:42.189 / 401 29ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:42.114 / 401 76ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:39.592 / 401 51ms 0kb curl/7.21.0 (x86_64-redhat-linux-gnu)
> libcurl/7.21.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4
>
> 2012-08-06 13:55:38.948 / 401 85ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
> 2012-08-06 13:55:38.945 / 401 130ms 0kb curl/7.19.7 (i386-redhat-linux-gnu)
> libcurl/7.19.7 NSS/3.12.6.2 zlib/1.2.3 libidn/1.9 libssh2/1.2.4
>
> 2012-08-06 13:55:38.944 / 401 123ms 0kb curl/7.18.2 (i386-redhat-linux-gnu)
> libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
>
>
> On Monday, August 6, 2012 2:50:55 PM UTC-4, Kate wrote:
> Just read this. Thanks.
>
> Looks like it works now.
>
> Thanks to all!!!!!
>
> Kate
>
>
>
> On Friday, August 3, 2012 10:51:23 AM UTC-4, Joshua Smith wrote:
> There are couple problems with your snippet.
>
> First, she's getting HEAD not GET requests, so you need to use different
> handler.
>
> Also, you aren't returning, so if you were in a GET request, it would proceed
> to handle the request regardless.
>
> Something more like this (untested):
>
> class MainHandler(webapp.RequestHandler):
> def head(self):
> self.error(401)
>
> def get(self):
> if (self.request.headers['User-Agent'].startswith('curl'))
> self.error(401)
> return
> # rest of the get handler
>
> On Aug 3, 2012, at 10:34 AM, Kyle Finley <[email protected]> wrote:
>
>> Hi Joshua,
>>
>> Thank you, that's a good thought.
>>
>> Kate sent me some files offline, and I believe we've figured out the
>> problem. For the middleware to work you must be using WSGI not CGI. Someone
>> please correct me if I'm wrong, but I believe she would have to upgrade here
>> App to python27 to use it. The alternative is to do the check in the webapp
>> request handler:
>>
>> def check_for_curl(self):
>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
>> return self.error(401)
>>
>> class MainHandler(webapp.RequestHandler):
>> def get(self):
>> check_for_curl(self)
>> # handle request
>>
>> The problem is that webapp doesn't recognize error code 429 so we have to
>> use something else. Unless there's a simple way to make it write 429?
>>
>> - Kyle
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/ClwBVQQTxesJ.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
