[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
I figured out this problem; it was not related after all. I needed to set the user-agent when using curl, with the curl_setopt command (in PHP). Once I did that I did not have problems using the Search API. On Aug 29, 7:03 pm, Dan danec...@gmail.com wrote: I'm not sure if this is related. I've been using Services_Twitter to use theSearchAPI and I keep getting the error message Unsupportedendpointsearch. I'm searching a simple 7-letter word. Anyone have any idea what that message means? Maybe this is related to something going on with Twitter'sSearchAPI?
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
I'm not sure if this is related. I've been using Services_Twitter to use the Search API and I keep getting the error message Unsupported endpoint search. I'm searching a simple 7-letter word. Anyone have any idea what that message means? Maybe this is related to something going on with Twitter's Search API?
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
I agree I have been using the Twitter search APIs for more than a year on 3 App Engine apps, this is seriously handicapping my usage of the API. -Ben Hedrington On Aug 27, 1:42 am, Paul Kinlan paul.kin...@gmail.com wrote: Hi, I still think something is going on (or at least different) - I have never seen this level of throttling on the Google App Engine. I am doing far less than 1 request a second and it is getting massively rate limited. In the past I have performed searches far more frequently. Paul 2009/8/26 Paul Kinlan paul.kin...@gmail.com Hi Chad, Has this limit changed recently? I used to query it far more frequently from the app engine. Obviously, Google use a lot of different IP addresses so I presuming it can fluctuate. But over the last couple of days I have noticed far more that I used to get. If it is by IP first what is the point of using the User-Agent (it was stated a little while back that we must include it now for rate limiting) - is it just for tracking of an application? Paul 2009/8/26 Chad Etzel c...@twitter.com Hi Paul, If you are sharing your IP with any other GAE twitter apps that are also doing search, then you are sharing the resource at that point. The limiting is by IP first, then user-agent. Also, 1 search per second is on the borderline of the normal rate-limit anyway, so I would try calling less frequently if possible. -Chad On Wed, Aug 26, 2009 at 12:29 PM, Paul Kinlanpaul.kin...@gmail.com wrote: Hi, Just a question, I am starting to see very heavy throttling to the Twitter Search API from the Google App engine. I am receiving 503's enhance your calm very frequently. I have a custom set User-Agent string and I am probably doing less than 1 search per second. It has been happening for a couple of days now. Has there been a recent change to cause this behaviour. Paul.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Hi, Just a question, I am starting to see very heavy throttling to the Twitter Search API from the Google App engine. I am receiving 503's enhance your calm very frequently. I have a custom set User-Agent string and I am probably doing less than 1 search per second. It has been happening for a couple of days now. Has there been a recent change to cause this behaviour. Paul.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Hi Paul, If you are sharing your IP with any other GAE twitter apps that are also doing search, then you are sharing the resource at that point. The limiting is by IP first, then user-agent. Also, 1 search per second is on the borderline of the normal rate-limit anyway, so I would try calling less frequently if possible. -Chad On Wed, Aug 26, 2009 at 12:29 PM, Paul Kinlanpaul.kin...@gmail.com wrote: Hi, Just a question, I am starting to see very heavy throttling to the Twitter Search API from the Google App engine. I am receiving 503's enhance your calm very frequently. I have a custom set User-Agent string and I am probably doing less than 1 search per second. It has been happening for a couple of days now. Has there been a recent change to cause this behaviour. Paul.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Hi Chad, Has this limit changed recently? I used to query it far more frequently from the app engine. Obviously, Google use a lot of different IP addresses so I presuming it can fluctuate. But over the last couple of days I have noticed far more that I used to get. If it is by IP first what is the point of using the User-Agent (it was stated a little while back that we must include it now for rate limiting) - is it just for tracking of an application? Paul 2009/8/26 Chad Etzel c...@twitter.com Hi Paul, If you are sharing your IP with any other GAE twitter apps that are also doing search, then you are sharing the resource at that point. The limiting is by IP first, then user-agent. Also, 1 search per second is on the borderline of the normal rate-limit anyway, so I would try calling less frequently if possible. -Chad On Wed, Aug 26, 2009 at 12:29 PM, Paul Kinlanpaul.kin...@gmail.com wrote: Hi, Just a question, I am starting to see very heavy throttling to the Twitter Search API from the Google App engine. I am receiving 503's enhance your calm very frequently. I have a custom set User-Agent string and I am probably doing less than 1 search per second. It has been happening for a couple of days now. Has there been a recent change to cause this behaviour. Paul.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
150 is a per-user rate, not a per-IP rate, to begin with, isn't it? The issue here is whitelisted IPs sharing 20k total, right? ∞ Andy Badera ∞ This email is: [ ] bloggable [x] ask first [ ] private ∞ Google me: http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera) On Sun, Aug 23, 2009 at 10:17 AM, boazsapirb...@gmail.com wrote: Hello everyone, I checked the behavior on an AWS instance _without_ static IP (which is called by Amazon elastic IP) and I do not see any problem with the limits. The limit status shows that I have exactly 150 calls left minus the ones I have explicitly used. I do not obeserve any behvior where my limits are affected by other users with which I share the resource. Am I missing something? Could it be just a matter of luck/random behavior? Thank you, Boaz On Aug 22, 12:03 am, Darren Bounds (Cliqset) dbou...@gmail.com wrote: Hello Chad, Can you confirm that this is not the case for AWS elastic IPs which had been previously whitelisted by Twitter? Thanks, Darren On Aug 21, 4:35 pm, Chad Etzel jazzyc...@gmail.com wrote: Hello, I have replied to Jud off-list, but for everyone's benefit we'd like to reiterate that AWS and GAE are shared resources and therefore share the rate limit across applications. A dedicated IP and unique UA will guarantee the maximum API limits. There are several cheap and reliable VPS hosting services available which can provide a dedicated IP address and full control over the server. Thanks, -Chad On Fri, Aug 21, 2009 at 12:48 PM, Judjvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that querieshttp://search.twitter.com/search.atom?q=forsimple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Another data point: I also run a Python app on Google App Engine. It hits the Twitter search API 3 times per minute, with different search parameters. About 20% of my app's search requests get a 503L error code, and the other 80% return search results as expected. There is no clear pattern or grouping of these errors. Other calls to the Twitter API that are not search-related, like users/show, fail only about 0.1% of the time. Anecdotal evidence: I seem to remember the success rate of my app's search requests was higher before the DDOS attacks two weeks ago. As I haven't saved the logs, I can't be sure. It would be better if the failure rate for searches were lower. But 20% failure is not a big deal for me as the searches run as cron jobs, which can always retry later. /Martin On Aug 23, 7:51 am, Andrew Badera and...@badera.us wrote: 150 is a per-user rate, not a per-IP rate, to begin with, isn't it? The issue here is whitelisted IPs sharing 20k total, right? ∞ Andy Badera ∞ This email is: [ ] bloggable [x] ask first [ ] private ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera) On Sun, Aug 23, 2009 at 10:17 AM, boazsapirb...@gmail.com wrote: Hello everyone, I checked the behavior on an AWS instance _without_ static IP (which is called by Amazon elastic IP) and I do not see any problem with the limits. The limit status shows that I have exactly 150 calls left minus the ones I have explicitly used. I do not obeserve any behvior where my limits are affected by other users with which I share the resource. Am I missing something? Could it be just a matter of luck/random behavior? Thank you, Boaz On Aug 22, 12:03 am, Darren Bounds (Cliqset) dbou...@gmail.com wrote: Hello Chad, Can you confirm that this is not the case for AWS elastic IPs which had been previously whitelisted by Twitter? Thanks, Darren On Aug 21, 4:35 pm, Chad Etzel jazzyc...@gmail.com wrote: Hello, I have replied to Jud off-list, but for everyone's benefit we'd like to reiterate that AWS and GAE are shared resources and therefore share the rate limit across applications. A dedicated IP and unique UA will guarantee the maximum API limits. There are several cheap and reliable VPS hosting services available which can provide a dedicated IP address and full control over the server. Thanks, -Chad On Fri, Aug 21, 2009 at 12:48 PM, Judjvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that querieshttp://search.twitter.com/search.atom?q=forsimple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Just to clarify: I am testing with unauthenticated calls On Aug 23, 5:17 pm, boaz sapirb...@gmail.com wrote: Hello everyone, I checked the behavior on an AWS instance _without_ static IP (which is called by Amazon elastic IP) and I do not see any problem with the limits. The limit status shows that I have exactly 150 calls left minus the ones I have explicitly used. I do not obeserve any behvior where my limits are affected by other users with which I share the resource. Am I missing something? Could it be just a matter of luck/random behavior? Thank you, Boaz On Aug 22, 12:03 am, Darren Bounds (Cliqset) dbou...@gmail.com wrote: Hello Chad, Can you confirm that this is not the case for AWS elastic IPs which had been previously whitelisted by Twitter? Thanks, Darren On Aug 21, 4:35 pm, Chad Etzel jazzyc...@gmail.com wrote: Hello, I have replied to Jud off-list, but for everyone's benefit we'd like to reiterate that AWS and GAE are shared resources and therefore share the rate limit across applications. A dedicated IP and unique UA will guarantee the maximum API limits. There are several cheap and reliable VPS hosting services available which can provide a dedicated IP address and full control over the server. Thanks, -Chad On Fri, Aug 21, 2009 at 12:48 PM, Judjvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that querieshttp://search.twitter.com/search.atom?q=forsimple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Hopefully to add some clarity: I have not used AWS or EC2, but as I understand it, Elastic IP's are IP's that you 'own' under your account that you can assign at will to your instances. In this way, it acts as a Static IP since you use it exclusively. -Chad On Sun, Aug 23, 2009 at 11:06 AM, boazsapirb...@gmail.com wrote: Just to clarify: I am testing with unauthenticated calls On Aug 23, 5:17 pm, boaz sapirb...@gmail.com wrote: Hello everyone, I checked the behavior on an AWS instance _without_ static IP (which is called by Amazon elastic IP) and I do not see any problem with the limits. The limit status shows that I have exactly 150 calls left minus the ones I have explicitly used. I do not obeserve any behvior where my limits are affected by other users with which I share the resource. Am I missing something? Could it be just a matter of luck/random behavior? Thank you, Boaz On Aug 22, 12:03 am, Darren Bounds (Cliqset) dbou...@gmail.com wrote: Hello Chad, Can you confirm that this is not the case for AWS elastic IPs which had been previously whitelisted by Twitter? Thanks, Darren On Aug 21, 4:35 pm, Chad Etzel jazzyc...@gmail.com wrote: Hello, I have replied to Jud off-list, but for everyone's benefit we'd like to reiterate that AWS and GAE are shared resources and therefore share the rate limit across applications. A dedicated IP and unique UA will guarantee the maximum API limits. There are several cheap and reliable VPS hosting services available which can provide a dedicated IP address and full control over the server. Thanks, -Chad On Fri, Aug 21, 2009 at 12:48 PM, Judjvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that querieshttp://search.twitter.com/search.atom?q=forsimple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
not sure about GAE, but for AWS, you can request for a static IP, it costs some $, but it's the only way to work with Twitter API if rate limit is an issue for you On Aug 21, 1:29 pm, BenHedrington b...@hedrington.com wrote: I agree GAE throttle on the Search API is not behaving as it has in the past, Can someone please look into this? -Ben Hedrington On Aug 21, 11:48 am, Jud jvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that querieshttp://search.twitter.com/search.atom?q=forsimple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
On Sat, Aug 22, 2009 at 6:26 AM, Beierbeier...@gmail.com wrote: not sure about GAE, but for AWS, you can request for a static IP, it costs some $, but it's the only way to work with Twitter API if rate limit is an issue for you GAE has no such IP offering yet. Also, by its very nature, all activity on GAE is global -- which is why it's taking them a while to deliver SSL services as well. ∞ Andy Badera ∞ This email is: [ ] bloggable [x] ask first [ ] private ∞ Google me: http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Hello Chad, Can you confirm that this is not the case for AWS elastic IPs which had been previously whitelisted by Twitter? Thanks, Darren On Aug 21, 4:35 pm, Chad Etzel jazzyc...@gmail.com wrote: Hello, I have replied to Jud off-list, but for everyone's benefit we'd like to reiterate that AWS and GAE are shared resources and therefore share the rate limit across applications. A dedicated IP and unique UA will guarantee the maximum API limits. There are several cheap and reliable VPS hosting services available which can provide a dedicated IP address and full control over the server. Thanks, -Chad On Fri, Aug 21, 2009 at 12:48 PM, Judjvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that querieshttp://search.twitter.com/search.atom?q=for simple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
I agree GAE throttle on the Search API is not behaving as it has in the past, Can someone please look into this? -Ben Hedrington On Aug 21, 11:48 am, Jud jvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that querieshttp://search.twitter.com/search.atom?q=for simple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
[twitter-dev] Re: heavy throttling by search.twitter.com API from GAE application
Hello, I have replied to Jud off-list, but for everyone's benefit we'd like to reiterate that AWS and GAE are shared resources and therefore share the rate limit across applications. A dedicated IP and unique UA will guarantee the maximum API limits. There are several cheap and reliable VPS hosting services available which can provide a dedicated IP address and full control over the server. Thanks, -Chad On Fri, Aug 21, 2009 at 12:48 PM, Judjvale...@gmail.com wrote: I've got a python app running on Google App Engine (appspot hosted) that queries http://search.twitter.com/search.atom?q= for simple queries (e.g. foo OR bar), and it's being severely throttled (e.g. can't get a successful request through (response 200 w/ data) more than a couple of times per _hour_). - I'm setting the UA string to something unique/identifiable (e.g. my company name) - I'm respecting the retry-after header coming back when I see a 503 (average retry-after duration is ~750) - GAE turns the IP address behind the app over ~ every 6 hours - app hits tries to hit search.twitter.com every 5 minutes. I've successfully polled the endpoint at much higher rates (in completely different IP address ranges) in the past, without issue. Unclear what's going on. Any suggestions would be greatly appreciated.
