[twitter-dev] Re: Search API
The best way to do this is to use the streaming API and catch all the tweets containing stanley cup when they happen. The search API is very limited and you will never get more then a couple of thousand results in the past. Oftentimes much less. Best regards, Stefan -- Twitter developer documentation and resources: https://dev.twitter.com/doc API updates via Twitter: https://twitter.com/twitterapi Issues/Enhancements Tracker: https://code.google.com/p/twitter-api/issues/list Change your membership to this group: https://groups.google.com/forum/#!forum/twitter-development-talk
[twitter-dev] Re: Search API return 402 You have been rate limited. Enhance your calm each time
1 - Hosted on GAE is probably your problem you are sharing a limited pool of IP adresses shared by many other GAE based appls using Twitter API. see here : http://groups.google.com/group/google-appengine/browse_thread/thread/20931a508f4dd6e9 happy coding:-) Nick http://gaengine.blogspot.com/ On Apr 18, 11:49 pm, kghate kgh...@gmail.com wrote: I am writing a new application and all was going smoothly until I deployed the application and am getting a 402 on all requests! The application searches based on both geo-location and query terms. Am literally making only test api calls from the application (less than 10 every hour) and each one of it returns a 402. What could be happening? Here are some details 1. Test Application hosted on the Google App Engine 2. Using JTwitter 3. Using OAuth The first time, I thought Twitter might be having issues; but it cant be true all the time. Please help! -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API: retrieve the number of times a particular URL was tweeted?
Got it. Thanks, Taylor. On Mar 22, 11:16 am, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Doug, We don't have a search result counting API available at this time. One approach would be to prepare ahead of time and use the Streaming API's track filter on the URL you're interested in, keeping the stream open and counting tweets featuring your URL as it spreads on Twitter.http://dev.twitter.com/pages/streaming_api_methods#track Taylor @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Tue, Mar 22, 2011 at 7:13 AM, doug douglas.r...@gmail.com wrote: The Facebook API has a FQL query that allows one to retrieve the number of Likes or Shares a particular URL got. I can certainly find a way to use the Twitter Search API to retrieve the raw statuses that mention a particular URL... but it seems like overkill when all I would like is the count... the number of statuses that mention that URL. Is there a way to simply retrieve _just_ the count of URL mentions? Thanks, Doug -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API rate limit change?
I just wanted to add to this. The 420s have let up for the most part and I'm no longer seeing rate limiting behavior significantly different from the norm. I've noticed that many result pages are coming back with empty results but if I re-request the same page (after a couple second delay), I can often get results for that page. These are for queries with very low tweet velocity, so it's not like these are new results coming in. Is this related to http://status.twitter.com/post/3785043723/slow-searches ? Thanks. Hayes On Mon, Mar 21, 2011 at 6:13 PM, Matt Harris thematthar...@twitter.comwrote: In many cases we are forced to change the rate limits in response to a significant increase in requests, which means it isn't always possible to give advanced notice of rate limit changes. For some of you it sounds like your code that handles rate limiting didn't react appropriately. When receiving a 420 response we recommend you stop making requests and then after the retry-after, slowly build up the number of requests you make. Put another way it isn't a good idea to make requests to the API at the velocity that caused the 420 response before. As always, the rate limits are there to ensure the system is responsive and available to as many users as possible. This means it is necessary to reduce the number of queries you can make without notice. The best place to stay informed about issues like this are posted through @twitterapi and published on the Twitter status blog: http://status.twitter.com/post/3785043723/slow-searches Best, @themattharris Developer Advocate, Twitter http://twitter.com/themattharris On Mon, Mar 21, 2011 at 7:41 AM, Colin Surprenant colin.surpren...@gmail.com wrote: By adjusting the rate limits to reduce the stress on your search api without notice you have significantly increased the stress level on our end :P Seriously, advanced notice of the situation would have been welcome. In particular what created lots of confusion on our end is that even after pausing for the specified retry_after delay we would immediately get repeated 420s at which point we started to assume our IPs were banned (which also contributed to increase the stress level). Colin On Mar 21, 9:12 am, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: Taylor, Yeah this was definitely NOT good.In the past, when there is a service disruption, your api group would post something on your status page and tweet about it... Instead, I'm finding out about this from my customers... Did y'all tweet about this or present this somewhere where I could find it? Jeffrey Tweettronics.com On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner waldronfaulk...@gmail.com wrote: Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about
Re: [twitter-dev] Re: Search API rate limit change?
Taylor, Yeah this was definitely NOT good.In the past, when there is a service disruption, your api group would post something on your status page and tweet about it... Instead, I'm finding out about this from my customers... Did y'all tweet about this or present this somewhere where I could find it? Jeffrey Tweettronics.com On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner waldronfaulk...@gmail.com wrote: Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API rate limit change?
By adjusting the rate limits to reduce the stress on your search api without notice you have significantly increased the stress level on our end :P Seriously, advanced notice of the situation would have been welcome. In particular what created lots of confusion on our end is that even after pausing for the specified retry_after delay we would immediately get repeated 420s at which point we started to assume our IPs were banned (which also contributed to increase the stress level). Colin On Mar 21, 9:12 am, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: Taylor, Yeah this was definitely NOT good. In the past, when there is a service disruption, your api group would post something on your status page and tweet about it... Instead, I'm finding out about this from my customers... Did y'all tweet about this or present this somewhere where I could find it? Jeffrey Tweettronics.com On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner waldronfaulk...@gmail.com wrote: Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker:http://code.google.com/p/twitter-api/issues/list Change your membership to this group:http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API: Can the geocode param only give me lat/long results?
Hi Augusto, Thanks for your reply. The problem with the Streaming API is that I'd have to have set some database app listening to the stream for the past few years to be able to get all the data (especially for remote locations). I also don't know where my users are going to be, so I don't have the ability to set any bounding boxes... I need something that I can search back in time, rather than set listening.. without knowing in advance where the users will be. Hence the need for the search API... S. On Mar 21, 4:05 pm, Augusto Santos augu...@gemeos.org wrote: Streaming API will give what you need through locations method.http://dev.twitter.com/pages/streaming_api_methods#locations On Mon, Mar 21, 2011 at 12:22 PM, Stu stuart.batter...@gmail.com wrote: Hi, I have a pre-question before my question. With the search API's geocode based search, if it falls back on the user's profile information does it use GPS positions in their profile or some location such as 'London'. The problem is that I need much greater precision than that. Thus, if I perform this search: http://search.twitter.com/search.json?geocode=51.53,-0.14,1mi Am I able to get results back that only contain lat/long values of the tweet? The json returned here has basically no values for 'geo'. Thanks S. -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- 氣 -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API: Can the geocode param only give me lat/long results?
Hi Stu, If you need to use the search API for this, you'll need to tolerate the greedy-matching on the profile location field, by discarding the results uninteresting for your purposes (those tweets with no explicit geotagging). Taylor On Mon, Mar 21, 2011 at 9:15 AM, Stu stuart.batter...@gmail.com wrote: Hi Augusto, Thanks for your reply. The problem with the Streaming API is that I'd have to have set some database app listening to the stream for the past few years to be able to get all the data (especially for remote locations). I also don't know where my users are going to be, so I don't have the ability to set any bounding boxes... I need something that I can search back in time, rather than set listening.. without knowing in advance where the users will be. Hence the need for the search API... S. On Mar 21, 4:05 pm, Augusto Santos augu...@gemeos.org wrote: Streaming API will give what you need through locations method. http://dev.twitter.com/pages/streaming_api_methods#locations On Mon, Mar 21, 2011 at 12:22 PM, Stu stuart.batter...@gmail.com wrote: Hi, I have a pre-question before my question. With the search API's geocode based search, if it falls back on the user's profile information does it use GPS positions in their profile or some location such as 'London'. The problem is that I need much greater precision than that. Thus, if I perform this search: http://search.twitter.com/search.json?geocode=51.53,-0.14,1mi Am I able to get results back that only contain lat/long values of the tweet? The json returned here has basically no values for 'geo'. Thanks S. -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- 氣 -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API rate limit change?
In many cases we are forced to change the rate limits in response to a significant increase in requests, which means it isn't always possible to give advanced notice of rate limit changes. For some of you it sounds like your code that handles rate limiting didn't react appropriately. When receiving a 420 response we recommend you stop making requests and then after the retry-after, slowly build up the number of requests you make. Put another way it isn't a good idea to make requests to the API at the velocity that caused the 420 response before. As always, the rate limits are there to ensure the system is responsive and available to as many users as possible. This means it is necessary to reduce the number of queries you can make without notice. The best place to stay informed about issues like this are posted through @twitterapi and published on the Twitter status blog: http://status.twitter.com/post/3785043723/slow-searches Best, @themattharris Developer Advocate, Twitter http://twitter.com/themattharris On Mon, Mar 21, 2011 at 7:41 AM, Colin Surprenant colin.surpren...@gmail.com wrote: By adjusting the rate limits to reduce the stress on your search api without notice you have significantly increased the stress level on our end :P Seriously, advanced notice of the situation would have been welcome. In particular what created lots of confusion on our end is that even after pausing for the specified retry_after delay we would immediately get repeated 420s at which point we started to assume our IPs were banned (which also contributed to increase the stress level). Colin On Mar 21, 9:12 am, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: Taylor, Yeah this was definitely NOT good.In the past, when there is a service disruption, your api group would post something on your status page and tweet about it... Instead, I'm finding out about this from my customers... Did y'all tweet about this or present this somewhere where I could find it? Jeffrey Tweettronics.com On Sun, Mar 20, 2011 at 3:14 PM, Waldron Faulkner waldronfaulk...@gmail.com wrote: Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via
[twitter-dev] Re: Search API rate limit change?
Without prior notice, I can understand (circumstances), but without any kind of subsequent announcement?? Means we have to discover issues ourselves, verify that they're Twitter related (and not internal), then search around for existing discussion on the topic. Saves us a lot of time and headaches if Twitter would just announce stuff like this. On Mar 18, 2:51 pm, Taylor Singletary taylorsinglet...@twitter.com wrote: We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API rate limit change?
We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API rate limit change?
We're working to reinstate the usual limits on the Search API; due to the impact of the Japanese earthquake and resultant query increase against the Search API, some rates were adjusted to cope better serve queries. Will give everyone an update with the various limits are adjusted. @episod http://twitter.com/episod - Taylor Singletary - Twitter Developer Advocate On Fri, Mar 18, 2011 at 11:39 AM, Hayes Davis ha...@appozite.com wrote: Hi, We're seeing this as well starting at approximately the same time as described. We've backed off on searching but are seeing no reduction in the sporadic limiting. It also appears that the amount of results returned on successful queries is severely limited. Some queries that often have 1500 tweets from the last 5 days are returning far fewer results from only the last day. Could we get an update on this? Hayes On Fri, Mar 18, 2011 at 10:13 AM, Eric e...@telvetto.com wrote: We're also seeing 400s on different boxes across different IP addresses with different queries (so it does not appear to be server or query specific). These began on all boxes at 2 a.m. UTC. We've backed off on both number and rate of queries with no effect. We've also noticed an increase in sporadic fail whales via browser based search (atom and html) from personal accounts, although we haven't attempted to quantify it. On Mar 18, 7:40 am, zaver zave...@hotmail.com wrote: Hello, After the latest performance issues with the search api i have been seeing a lot of 420 response codes.From yesterday until now i only get 420 responses on the every search i make. In particular, i search for about 100 keywords simultaneously every 6 mins. Why is this happening? Was there any change on the Search API limit? Any help is greatly appreciated. Thanks, Zaver -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API: Searching world twits (location enabled)
You could use the streaming api and throw away tweets that have no location data/show them as having a default location. Whether or not this is a viable option for you depends on how often the keyword is tweeted and whether you need to index absolutely all tweets for the keyword... On Mar 4, 11:02 am, mahorad maho...@gmail.com wrote: Does any body know how is it possible to: search a keyword among all twits in the whole world, while each returned twit contains the location it was generated? I know that twits can be searched within an area passing lat,lon and radius but I want to search the keyword within the twits in the whole world. Any help will be truly appreciated. -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API the public alternative to URL count API?
Hi Carlos, I am not sure I understand the relevance of your question. I am planning on using OAuth for authentication. The URL count API (url.api.twit..) does not need authentication. Assuming that with callback you refer to the Javascript callbacks; The processing in question will happen in a backend component and thus wont be able to receive the callbacks? Martin On Feb 14, 7:14 pm, Carlos Hugo Gonzalez Castell carlos.hugo.gonzalez.cast...@gmail.com wrote: are usign oauth api? in this api your manage the callbacks twitter On 13 feb, 07:51, Martin Cronjé martincronj...@gmail.com wrote: Hi there, I am busy writing an aggregator and I am looking at using the Twitter API to get URL counts. I seems that public developers are not allowed to use the URL counting API based on the Tweet Button FAQ. Which leaves me with not other option but to use the search API for URL counting. Using the search API makes not sense if there a Count API. This leaves me with the following questions 1. Will my application / I.P. get banned if I use the Count API? 2. Is there a way to request multiple URLs at once to limit round- trips? 3. The URL count API returns not threshold information. So if I am allowed to use it, should I manage the thresholds myself FAQ -http://dev.twitter.com/pages/tweet_button_faq#count-api URL Count API -http://urls.api.twitter.com/1/urls/count.json?url=URL Search API -http://search.twitter.com/search.format My application aggregates URLs on a central server using a shared account so the request numbers may be quite high Martin- Hide quoted text - - Show quoted text - -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API the public alternative to URL count API?
are usign oauth api? in this api your manage the callbacks twitter On 13 feb, 07:51, Martin Cronjé martincronj...@gmail.com wrote: Hi there, I am busy writing an aggregator and I am looking at using the Twitter API to get URL counts. I seems that public developers are not allowed to use the URL counting API based on the Tweet Button FAQ. Which leaves me with not other option but to use the search API for URL counting. Using the search API makes not sense if there a Count API. This leaves me with the following questions 1. Will my application / I.P. get banned if I use the Count API? 2. Is there a way to request multiple URLs at once to limit round- trips? 3. The URL count API returns not threshold information. So if I am allowed to use it, should I manage the thresholds myself FAQ -http://dev.twitter.com/pages/tweet_button_faq#count-api URL Count API -http://urls.api.twitter.com/1/urls/count.json?url=URL Search API -http://search.twitter.com/search.format My application aggregates URLs on a central server using a shared account so the request numbers may be quite high Martin -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API intermittently returning invalid results
We're seeing the exact same problem in our application. We happen to be using the Twitter ruby gem, but we are experiencing the same behavior. -Ryan On Feb 9, 3:22 pm, chouck cho...@gnipcentral.com wrote: I've been using curl to access search.twitter.com and recently I've noticed that occasionally it is returning invalid tweets. I'm searching for a query-term that shows up very infrequently in the tweet stream, and am using the curl command: curl http://search.twitter.com/search.atom?q=query- termrpp=99result_type=recentsince_id=35409539146719234 What I've found is that about 99% of the time I get back an empty response, just the feed tag wrapped around the meta data for the query but no results. But if I just run the exact same curl command over and over again, occasionally I'll get a response with a full payload of 99 tweets, none of which have anything to do with my search terms. Subsequent executions of the same curl command return the response w/o any tweets in it. Its a little hard to explain, but I have a log file that shows: curl - no tweets curl - no tweets curl - no tweets curl - 99 unrelated tweets curl - no tweets All running the same command via cut-and-paste and all within the space of a few seconds. More often than not, it seems like the invalid tweets are all somehow related, as if I had gotten the response for some other active query. Anyone else seeing anything this problem? The log file is 16K compressed, let me know if you'd like me to send it in. Thanks, -Chris -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API rate limit on some keywords
have observed that sometimes some of the keywords get a 420 code? Any ideas why is this happening? You get a 420 NOT USED when a search term hasn't been used recently where the recently is whatever small timeframe (sometimes 7 days, often less) is currently available in the search index. I get it all the time for things like #stlcards :) -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API rate limit on some keywords
Thanks for the reply. So as i understand it i am not being rate limited, it just doesn't find any results to return for the specific keywords. In that case should i wait for the amount of time specified on the Retry-after field to make a new search? If i don't wait will that lead to my ip getting blacklisted or it will just not return accurate search results? Thanks in advance. On Feb 2, 3:04 am, @IDisposable idisposa...@gmail.com wrote: have observed that sometimes some of the keywords get a 420 code? Any ideas why is this happening? You get a 420 NOT USED when a search term hasn't been used recently where the recently is whatever small timeframe (sometimes 7 days, often less) is currently available in the search index. I get it all the time for things like #stlcards :) -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API and location
Hi Tom, DataSift is still in closed Alpha but we have enabled a large number of users within the Alpha to date and will add additional users for the Beta so if you'd like early access it's well worth signing up on http://datasift.net and you may find you get in before we launch the consumer release version. Many thanks Sarah Community Manager DataSift.net On Dec 29 2010, 11:50 am, Tom van der Woerdt i...@tvdw.eu wrote: On 12/29/10 10:40 AM, L. Mohan Arun wrote: I just tried to construct a query that searches for users by location, as it is registered in the location field of their profiles. I had no luck and it seems this is not possible. You can also do this using Datasift's FSDL. ✿✿✿ Mohan ✿✿✿ DataSift is still in closed alpha testing phase... Tom -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API and location
Hi Sarah, I'm already a member. Thanks for the offer though :-) Tom On 1/4/11 3:45 PM, Sarah - DataSift wrote: Hi Tom, DataSift is still in closed Alpha but we have enabled a large number of users within the Alpha to date and will add additional users for the Beta so if you'd like early access it's well worth signing up on http://datasift.net and you may find you get in before we launch the consumer release version. Many thanks Sarah Community Manager DataSift.net On Dec 29 2010, 11:50 am, Tom van der Woerdti...@tvdw.eu wrote: On 12/29/10 10:40 AM, L. Mohan Arun wrote: I just tried to construct a query that searches for users by location, as it is registered in the location field of their profiles. I had no luck and it seems this is not possible. You can also do this using Datasift's FSDL. ✿✿✿ Mohan ✿✿✿ DataSift is still in closed alpha testing phase... Tom -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API and location
I just tried to construct a query that searches for users by location, as it is registered in the location field of their profiles. I had no luck and it seems this is not possible. You can also do this using Datasift's FSDL. ✿✿✿ Mohan ✿✿✿ -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API and location
On 12/29/10 10:40 AM, L. Mohan Arun wrote: I just tried to construct a query that searches for users by location, as it is registered in the location field of their profiles. I had no luck and it seems this is not possible. You can also do this using Datasift's FSDL. ✿✿✿ Mohan ✿✿✿ DataSift is still in closed alpha testing phase... Tom -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API and location
I just tried to construct a query that searches for users by location, as it is registered in the location field of their profiles. I had no luck and it seems this is not possible. Google find twitter users by location See localtweeps.com ### Mohan ### -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API from_user_id doesn't match up with the proper Twitter user_id
Also, while it would be possible to use screen names for relations (i.e. from_user), this would have a very negative side effect. Mainly, if a user were to change their Twitter account name, previous relations would be lost. On Dec 22, 9:44 am, Corey Ballou ball...@gmail.com wrote: For clarification, I had intended to say from_user_id, as the username is returned properly. On Dec 22, 9:42 am, Corey Ballou ball...@gmail.com wrote: I just wanted to bring group-wide awareness to the fact that search results from Twitter do not return an actual user_id. This has been a known defect (and yes, I do believe it's a *very large* defect) going on over 2 years now. This is a call to arms to get this shit fixed. I can't believe it's marked as an enhancement. There's nobody else to blame for providing a return param of from_user that doesn't actually map to an actual user. For those of us storing relational data, you're costing precious API calls for those users who are still utilizing the search API. The streaming API is not sufficient for all use cases, so that's not a valid answer. Below is the direct link to the issue tracker. https://code.google.com/p/twitter-api/issues/detail?id=214 -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API from_user_id doesn't match up with the proper Twitter user_id
I'm sure I came off a little strong in the initial post; unfortunately for me google groups doesn't supply an edit button. I think there is still a grain of merit to the request to fix the issue, regardless of the API being free. I'm interest in knowing the trade-offs of Twitter essentially requiring third party apps to make subsequent calls to users/lookup for each unique user in a batch of results. The current problem I see, from Twitter's end, is that the subsequent call returns far more data than necessary. It's doubling the RTTs on both ends and creating an excessive amount of bandwidth for a trivial amount of data. I've got to imagine there's a number of cache misses going on due to the frequency of user updates and pulling the latest tweet, so it would seem rather costly. On Dec 22, 4:33 pm, Robbie Coleman rob...@robnrob.com wrote: I think twitter's response to this call to arms should be the HTTP Status Code: 420 - Chill ;-} -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API from_user_id doesn't match up with the proper Twitter user_id
Yeah, well the call to arms may have been over the top. :) I agree that Twitter should fix the search API. Every time I ask, the answer is that it will be done eventually, and that it will have entities and everything else the streaming API has. I think this means that it will be the streaming API code with the ability to look backwards added. Think about it, isn't that what any architect would do? You combine your code bases. The real problem with search is its inability to go back beyond 5-7 days. Since Twitter plans to make its money from search ads and compete for Google ad revenue, more search results means more searching and more ad revenue. I bet they plan on an IPO within a year, and the story that Twitter search is just a tiny fraction of Google search but growing like crazy is exactly the type of promise that makes investors crazy for an IPO. It is also pretty sad that Google just added the ability to search millions of books going back 500 years, and Twitter only goes back 5 days! So search is clearly very important. I just don't think they want to fix this code. It is Summize code, and they show no interest in diving into it. Until they rewrite it, we have to wait. On Wed, Dec 22, 2010 at 8:55 PM, Corey Ballou ball...@gmail.com wrote: I'm sure I came off a little strong in the initial post; unfortunately for me google groups doesn't supply an edit button. I think there is still a grain of merit to the request to fix the issue, regardless of the API being free. I'm interest in knowing the trade-offs of Twitter essentially requiring third party apps to make subsequent calls to users/lookup for each unique user in a batch of results. The current problem I see, from Twitter's end, is that the subsequent call returns far more data than necessary. It's doubling the RTTs on both ends and creating an excessive amount of bandwidth for a trivial amount of data. I've got to imagine there's a number of cache misses going on due to the frequency of user updates and pulling the latest tweet, so it would seem rather costly. On Dec 22, 4:33 pm, Robbie Coleman rob...@robnrob.com wrote: I think twitter's response to this call to arms should be the HTTP Status Code: 420 - Chill ;-} -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Adam Green Twitter API Consultant and Trainer http://140dev.com @140dev -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API cURL strangeness
So it must be based on the IP Address and the UserAgent...
I have changed the UserAgent, so it works now, but I don't
particularly like this solution. It would be nice to know what
happened, and what caused it, so I can try to prevent it from
happening in the future.
On Dec 14, 11:24 am, Tom van der Woerdt i...@tvdw.eu wrote:
Tested it myself with :
tom-mbp:~ tom$ curl --user-agent PivotalVeracity/0.4
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...;
Result :
{results:[],max_id:14696108638863361,since_id:9431322892177408,refresh_url:?since_id=14696108638863361q=batteryoperatedcandles.net,results_per_page:100,page:1,completed_in:0.006856,since_id_str:9431322892177408,max_id_str:14696108638863361,query:batteryoperatedcandles.net}
Seems to work fine... Getting exactly the same results when using the
default User Agent.
Tom
On 12/14/10 5:15 PM, Brian Medendorp wrote:
UserAgent is 'PivotalVeracity/0.4'
Here's the test script that helped me track down the problem:
[code]
?php
$timeout = 30;
$useragent = 'PivotalVeracity/0.4';
#$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:
1.9.2.13) Gecko/20101203 Firefox/3.6.13';
$url = 'http://search.twitter.com/search.json?
q=batteryoperatedcandles.netrpp=100since_id=9431322892177408since=until=';
#$url = 'http://search.twitter.com/search.json?
q=carnationbreakfastessentials.comrpp=100since_id=since=2010-12-14until=';
#$url = 'http://search.twitter.com/search.json?
q=apple.comrpp=100since_id=since=2010-12-14until=';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
$content = curl_exec($ch);
if(curl_errno($ch))
{
print curl error: .curl_error($ch).\n;
print_r(curl_getinfo($ch));
}
print_r($content);
[/code]
On Dec 14, 11:03 am, Tom van der Woerdti...@tvdw.eu wrote:
And your UserAgent is?
Tom
On 12/14/10 5:02 PM, Brian Medendorp wrote:
I'm building an application that uses the search API to check for data
related to particular domains, and suddenly (within the last week or
so), I have started to experience a strange problem. Some of my
requests are coming back with a cURL error Empty reply from server,
but only when I am searching for a specific set of domains (all of the
other domains work fine).
I wrote a small test script to try and track down the problem, and it
seems that the UserAgent I am setting with cURL seems to be causing
the problem (or part of the problem). If I change the UserAgent to
anything else, I get a normal response.
I remember reading in the documentation that Twitter expects a unique
UserAgent for the application, so that's what I did, but that seems to
be causing problems. This seems like it's likely some sort of
blacklist problem, but I can't figure out why it would work in this
manner (only blocking a small subset of my queries, and not IP-based).
Here are some sample queries I am trying to cURL:
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...
http://search.twitter.com/search.json?q=carnationbreakfastessentials
http://search.twitter.com/search.json?q=apple.comrpp=100since_id=s...
The first two don't work unless I change my UserAgent to something
else, but the last one works no matter what.
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API cURL strangeness
UserAgent is 'PivotalVeracity/0.4'
Here's the test script that helped me track down the problem:
[code]
?php
$timeout = 30;
$useragent = 'PivotalVeracity/0.4';
#$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:
1.9.2.13) Gecko/20101203 Firefox/3.6.13';
$url = 'http://search.twitter.com/search.json?
q=batteryoperatedcandles.netrpp=100since_id=9431322892177408since=until=';
#$url = 'http://search.twitter.com/search.json?
q=carnationbreakfastessentials.comrpp=100since_id=since=2010-12-14until=';
#$url = 'http://search.twitter.com/search.json?
q=apple.comrpp=100since_id=since=2010-12-14until=';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
$content = curl_exec($ch);
if(curl_errno($ch))
{
print curl error: .curl_error($ch).\n;
print_r(curl_getinfo($ch));
}
print_r($content);
[/code]
On Dec 14, 11:03 am, Tom van der Woerdt i...@tvdw.eu wrote:
And your UserAgent is?
Tom
On 12/14/10 5:02 PM, Brian Medendorp wrote:
I'm building an application that uses the search API to check for data
related to particular domains, and suddenly (within the last week or
so), I have started to experience a strange problem. Some of my
requests are coming back with a cURL error Empty reply from server,
but only when I am searching for a specific set of domains (all of the
other domains work fine).
I wrote a small test script to try and track down the problem, and it
seems that the UserAgent I am setting with cURL seems to be causing
the problem (or part of the problem). If I change the UserAgent to
anything else, I get a normal response.
I remember reading in the documentation that Twitter expects a unique
UserAgent for the application, so that's what I did, but that seems to
be causing problems. This seems like it's likely some sort of
blacklist problem, but I can't figure out why it would work in this
manner (only blocking a small subset of my queries, and not IP-based).
Here are some sample queries I am trying to cURL:
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...
http://search.twitter.com/search.json?q=carnationbreakfastessentials
http://search.twitter.com/search.json?q=apple.comrpp=100since_id=s...
The first two don't work unless I change my UserAgent to something
else, but the last one works no matter what.
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API cURL strangeness
Tested it myself with :
tom-mbp:~ tom$ curl --user-agent PivotalVeracity/0.4
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrpp=100since_id=9431322892177408;
Result :
{results:[],max_id:14696108638863361,since_id:9431322892177408,refresh_url:?since_id=14696108638863361q=batteryoperatedcandles.net,results_per_page:100,page:1,completed_in:0.006856,since_id_str:9431322892177408,max_id_str:14696108638863361,query:batteryoperatedcandles.net}
Seems to work fine... Getting exactly the same results when using the
default User Agent.
Tom
On 12/14/10 5:15 PM, Brian Medendorp wrote:
UserAgent is 'PivotalVeracity/0.4'
Here's the test script that helped me track down the problem:
[code]
?php
$timeout = 30;
$useragent = 'PivotalVeracity/0.4';
#$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:
1.9.2.13) Gecko/20101203 Firefox/3.6.13';
$url = 'http://search.twitter.com/search.json?
q=batteryoperatedcandles.netrpp=100since_id=9431322892177408since=until=';
#$url = 'http://search.twitter.com/search.json?
q=carnationbreakfastessentials.comrpp=100since_id=since=2010-12-14until=';
#$url = 'http://search.twitter.com/search.json?
q=apple.comrpp=100since_id=since=2010-12-14until=';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
$content = curl_exec($ch);
if(curl_errno($ch))
{
print curl error: .curl_error($ch).\n;
print_r(curl_getinfo($ch));
}
print_r($content);
[/code]
On Dec 14, 11:03 am, Tom van der Woerdti...@tvdw.eu wrote:
And your UserAgent is?
Tom
On 12/14/10 5:02 PM, Brian Medendorp wrote:
I'm building an application that uses the search API to check for data
related to particular domains, and suddenly (within the last week or
so), I have started to experience a strange problem. Some of my
requests are coming back with a cURL error Empty reply from server,
but only when I am searching for a specific set of domains (all of the
other domains work fine).
I wrote a small test script to try and track down the problem, and it
seems that the UserAgent I am setting with cURL seems to be causing
the problem (or part of the problem). If I change the UserAgent to
anything else, I get a normal response.
I remember reading in the documentation that Twitter expects a unique
UserAgent for the application, so that's what I did, but that seems to
be causing problems. This seems like it's likely some sort of
blacklist problem, but I can't figure out why it would work in this
manner (only blocking a small subset of my queries, and not IP-based).
Here are some sample queries I am trying to cURL:
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...
http://search.twitter.com/search.json?q=carnationbreakfastessentials
http://search.twitter.com/search.json?q=apple.comrpp=100since_id=s...
The first two don't work unless I change my UserAgent to something
else, but the last one works no matter what.
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API Optional lang Has Problem.
Same here. On Nov 29, 1:50 am, Jeong Hoon Kim redi...@gmail.com wrote: About 5 days ago, Suddenly Search API Optional lang had no results..My optional lang is ko. Did anybody apply Search API lang option? Did the results come out correctly? -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API Optional lang Has Problem.
And same with some search operators (like source:xxx) Sorry for posting twice :) On Nov 29, 1:50 am, Jeong Hoon Kim redi...@gmail.com wrote: About 5 days ago, Suddenly Search API Optional lang had no results..My optional lang is ko. Did anybody apply Search API lang option? Did the results come out correctly? -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API Optional lang Has Problem.
Same problem here. When lang=all is used I am getting results. When a language is specified I get zero results most of the time, while in some cases I do get a result. Seems very strange. On Nov 29, 9:25 am, fbparis fbou...@gmail.com wrote: And same with some search operators (like source:xxx) Sorry for posting twice :) On Nov 29, 1:50 am, Jeong Hoon Kim redi...@gmail.com wrote: About 5 days ago, Suddenly Search API Optional lang had no results..My optional lang is ko. Did anybody apply Search API lang option? Did the results come out correctly?- Hide quoted text - - Show quoted text - -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API Optional lang Has Problem.
I'm seeing this as well. Including filter:links or setting that language causes the search to fail. I get an error message saying since_id has been adjusted due to a temporary error. I'm *not* including a since_id in the search parameters. Hayes On Mon, Nov 29, 2010 at 10:08 AM, Randomness randomness.bl...@gmail.comwrote: Same problem here. When lang=all is used I am getting results. When a language is specified I get zero results most of the time, while in some cases I do get a result. Seems very strange. On Nov 29, 9:25 am, fbparis fbou...@gmail.com wrote: And same with some search operators (like source:xxx) Sorry for posting twice :) On Nov 29, 1:50 am, Jeong Hoon Kim redi...@gmail.com wrote: About 5 days ago, Suddenly Search API Optional lang had no results..My optional lang is ko. Did anybody apply Search API lang option? Did the results come out correctly?- Hide quoted text - - Show quoted text - -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API Optional lang Has Problem.
This has happened before. Appending a since clause works around it, but limits your search results to only five days. Also last time this happened they fixed it within a few weeks. I just wish we could get an official comment on this. On Nov 28, 5:50 pm, Jeong Hoon Kim redi...@gmail.com wrote: About 5 days ago, Suddenly Search API Optionallanghad no results..My optionallangis ko. Did anybody apply Search APIlangoption? Did the results come out correctly? -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API Optional lang Has Problem.
There is an issue with Twitter's language detection. When specifying a language (lang=nl) , there is no result, when using lang=all, I do get results, in my language. Using lang=all gives us so many results, that we're hitting the rate limits with lots of stuff we're throwing away straight away after we've passed it past our own language detection Are there any plans to fix this? On Nov 29, 1:50 am, Jeong Hoon Kim redi...@gmail.com wrote: About 5 days ago, Suddenly Search API Optional lang had no results..My optional lang is ko. Did anybody apply Search API lang option? Did the results come out correctly? -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API is acting weird.
The adjusted since_id to xxx due to temporary error really means this: the since_id was not specified so I went back as far as I could. The earliest tweet is the database was 25075604044 so i used that. For users who tweet more often this error can still occur but is less likely. This is because the search results per page fit into the available index and so no adjustment of since_id is required. For less active accounts the error above occurs because search tries to get n results per page. If there aren't n tweets in the available index the since_id goes beyond what we have stored - so the message above is displayed. As for the other users not showing up when you search for them using from:. There are many reasons for users not to show up in Search. The most common one (and applicable to your account gena01) is there haven't been any tweets in the last 5 days. Other reasons are explained on our help site: http://support.twitter.com/groups/32-something-s-not-working/topics/118-search/articles/66018-my-tweets-or-hashtags-are-missing-from-search-known-issue Hope that's helpful, @themattharris On Sep 21, 8:22 am, Gena01 gen...@gmail.com wrote: So I've been messing with the search API and I am seeing some strange stuff going on. When I request: http://search.twitter.com/search.json?q=to:evI get normal results. If i requesthttp://search.twitter.com/search.json?q=to:aorhttp://search.twitter.com/search.json?q=to:gena01then I get a warning: adjusted since_id to 25075604044 due to temporary error. If I do from: instead of to: I also get these sort of discrepancies. For people like @ev I get feeds/tweets/etc for people not as popular I get nothing back. Is there something I am doing wrong or is search api broken? Gena01 -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
[twitter-dev] Re: search api - since date format
here is my problem I need to catch some tweets since yesterday 20pm until this morning 8am. the problem is that there is more than 1500 tweets that I need, and according to search api docs, I can get a max of roughly 1500 tweets per search query. [...] rpp The number of tweets to return per page, up to a max of 100. http://search.twitter.com/search.json?rpp=100 page The page number (starting at 1) to return, up to a max of roughly 1500 results (based on rpp * page). http://search.twitter.com/search.json?page=10 [...] from http://dev.twitter.com/doc/get/search i'm struggling to build a query where I can get my tweets since yesterday night. any idea? Arian On 9 set, 12:23, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Arian, A date string really is the only valid format for this function. If you want to cut the search off by certain times of day, you're best off post-processing your results for that kind of resolution. Thanks, Taylor On Thu, Sep 9, 2010 at 8:15 AM, arian arianpasqu...@gmail.com wrote: hi guys, I have a question about search api about 'until' parameter to be more exactly according to documentation until is Optional. Returns tweets generated before the given date. Date should be formatted as -MM- DD. example:http://search.twitter.com/search.json?until=2010-03-28; I need to know if its possible to set datetime, for example http://search.twitter.com/search.json?until=2010-09-08-19:00; or something like this. according to doc date should be formatted as -MM-DD, but I need inform time, if its possible what would be the string format in this case? is it possible? or how could I get a similar result? Arian -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
Re: [twitter-dev] Re: search api - since date format
This sort of scenario is better served with advance preparation, rather than relying on the Search API to excavate the tweets after the fact, it would be more advantageous to utilize the Streaming API, tracking and storing all relevant tweets during your period of interest. Is this a one-off task you're trying to accomplish or something more general you're looking to accomplish? Taylor On Thu, Sep 9, 2010 at 9:32 AM, arian arianpasqu...@gmail.com wrote: here is my problem I need to catch some tweets since yesterday 20pm until this morning 8am. the problem is that there is more than 1500 tweets that I need, and according to search api docs, I can get a max of roughly 1500 tweets per search query. [...] rpp The number of tweets to return per page, up to a max of 100. http://search.twitter.com/search.json?rpp=100 page The page number (starting at 1) to return, up to a max of roughly 1500 results (based on rpp * page). http://search.twitter.com/search.json?page=10 [...] from http://dev.twitter.com/doc/get/search i'm struggling to build a query where I can get my tweets since yesterday night. any idea? Arian On 9 set, 12:23, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Arian, A date string really is the only valid format for this function. If you want to cut the search off by certain times of day, you're best off post-processing your results for that kind of resolution. Thanks, Taylor On Thu, Sep 9, 2010 at 8:15 AM, arian arianpasqu...@gmail.com wrote: hi guys, I have a question about search api about 'until' parameter to be more exactly according to documentation until is Optional. Returns tweets generated before the given date. Date should be formatted as -MM- DD. example:http://search.twitter.com/search.json?until=2010-03-28; I need to know if its possible to set datetime, for example http://search.twitter.com/search.json?until=2010-09-08-19:00; or something like this. according to doc date should be formatted as -MM-DD, but I need inform time, if its possible what would be the string format in this case? is it possible? or how could I get a similar result? Arian -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
[twitter-dev] Re: Search API problems...
Matt, thanks for the quick response. After an evening of trying to figure out what's going on, it appears to be working again. I guess the problem must have been on my side. Thank you so much for replying so quickly though, and for the explanation on rates and error messages! Many thanks, ben On Aug 25, 1:02 am, Matt Harris thematthar...@twitter.com wrote: There are no known issues with search and running your query works for me. Hey Ben, The Search API does not use authentication and is rate limited differently to the 150 IP requests allowed on the REST API. If you are rate limited on the Search API we would return an error telling you rather than not reply. If the atom link is still not responding can you tryhttp://search.twitter.com/search.json?q=testand let us know the result? Thanks, Matt On Tue, Aug 24, 2010 at 4:28 PM, Ben goo...@iamben.co.uk wrote: Hey guys - I'm curious as to know whether there's any problems with the search API? I'm curling from a PHP script, and it keeps timing out with 'couldn't connect to host' errors when my URL is a search (eg: http://search.twitter.com/search.atom?q=test). Interestingly, if I curl either of the following: http://api.twitter.com/1/help/test.xml http://api.twitter.com/1/account/rate_limit_status.xml ...and it doesn't time out, I get a true, and my rate limit is 150/150. I'm not using any authentication, this is a straight request from a script. Could I be on an IP blacklist for search (can I check this?)? I've been pretty careful with my caching, I make nowhere near 150 requests an hour, although my site is on a shared server, so it's entirely plausible someone else has been hammering it. Although if that was the case, would something not show up on the odd times I actually get the rate limit to show something? If anyone can help, or point me in the direction of something I've missed, I'd be eternally grateful... ben -- Twitter developer documentation and resources:http://dev.twitter.com/doc API updates via Twitter:http://twitter.com/twitterapi Issues/Enhancements Tracker:http://code.google.com/p/twitter-api/issues/list Change your membership to this group:http://groups.google.com/group/twitter-development-talk?hl=en -- Matt Harris Developer Advocate, Twitterhttp://twitter.com/themattharris -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk?hl=en
[twitter-dev] Re: Search API bug when using lang parameter
Not sure if this matters, but I've tried using no authentication and oauth authentication for this search API call, and am receiving the same results for both. For a while now, I've been tweeting this bug at @twitterapi and @twitter searching for people who've come across it, and I'm coming up empty. It's a huge problem for me and my app development. Am I just overlooking something?
[twitter-dev] Re: Search API reporting temporary error
I'm going directly to the URL in question; it only seems to happen to me when I have lang=en anywhere in the url. On Jun 15, 1:35 pm, Mack D. Male master...@gmail.com wrote: There seems to be something wrong with thesearchAPI. It is only returning a tiny subset of what I would expect (after looking at the same query onsearch.twitter.com for instance) and is reporting the following: adjustedsince_idto 16201119561 due to temporary error Any word on what this temporary error is, or when it'll be fixed? I'm using the latest build of TweetSharp, if that makes any difference.
[twitter-dev] Re: Search API -Timezones problem
Can anyone answer my questions??
[twitter-dev] Re: Search API rate limit
Matt, What is exact limit..Whether I can write to twitter for whitelisting of the IP? Whether whitelisting of the IP would do any good? Shan On Jul 7, 12:16 am, Matt Harris thematthar...@twitter.com wrote: Hi Shan, The Search API is anonymous so authenticating makes no difference to the rate limit there. If you are requesting a lot of information from the search API you may want to look at the streaming API instead:http://dev.twitter.com/pages/streaming_api The majority of search cases can be handled by the default filter limits available through the streaming API. Best, Matt On Tue, Jul 6, 2010 at 7:31 AM, Ramanean shang...@gmail.com wrote: I am developing a normal twitter search application Inorder to beat the search rate limit if I ask a user to authenticate whether that would be helpful? Whether the calls made by the user for search api will be counted in the user's account ? or whether that would be still counted as a call from the IP address of the website? I am little bit confused here... Thanks Shan -- Matt Harris Developer Advocate, Twitterhttp://twitter.com/themattharris
Re: [twitter-dev] Re: Search API rate limit
Shan, as far as I know twitter has been reluctant to state definite numbers, so you'll have to experiment and implement a backoff mechanism in your app. Here is the relevant part of the docs: Search API Rate Limiting The Search API is rate limited by IP address. The number of search requests that originate from a given IP address are counted against the search rate limiter. The specific number of requests a client is able to make to the Search API for a given hour is not released. Note that the Search API is not limited by the same 150 requests per hour limit as the REST API. The number is quite a bit higher and we feel it is both liberal and sufficient for most applications. We do not give the exact number because we want to discourage unnecessary search usage. Search API usage requires that applications include a unique and identifying User Agent string. A HTTP Referrer is expected but is not required. Consumers using the Search API but failing to include a User Agent string will receive a lower rate limit. An application that exceeds the rate limitations of the Search API will receive HTTP 420 response codes to requests. It is a best practice to watch for this error condition and honor the Retry-After header that instructs the application when it is safe to continue. The Retry-After header's value is the number of seconds your application should wait before submitting another query (for example: Retry-After: 67). Cheers, Pascal On Jul 7, 2010, at 1:55 , Ramanean wrote: Matt, What is exact limit..Whether I can write to twitter for whitelisting of the IP? Whether whitelisting of the IP would do any good? Shan
[twitter-dev] Re: Search API returns only 15 results, even if rpp=100?
Thanks for that -- I just figured that out and was coming back to report my findings, but I guess you beat me to it. :) On Jun 22, 8:01 am, Jonathan Reichhold jonathan.reichh...@gmail.com wrote: There are plenty of results for this, but your url is encoded incorrectly http://search.twitter.com/search.atom?q=microsoft+OR+%23ms+OR+lnk.ms+... # is %23 in url-encoded form As the query exists it is microsoft OR with a page reference. Jonathan On Tue, Jun 22, 2010 at 6:34 AM, John Kalucki j...@twitter.com wrote: Try a less complex query, and you should get more results. On Mon, Jun 21, 2010 at 8:39 PM, Josh Santangelo j...@endquote.com wrote: For example, this query: https://search.twitter.com/search.atom?q=microsoft+OR+#ms+OR+lnk.ms+O... Is there any way to get a larger number of results per page? thanks, -josh- Hide quoted text - - Show quoted text -
[twitter-dev] Re: Search API: searching for Don and finding don't instead
Hello Twitter, Anyone home? j On Jun 2, 11:28 pm, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: We have a user that is causing us to create a search of the form: Don SomeLastName which is returning tweets containing don't and SomeLastName. Thats a no good! Is there a decent workaround for this by modifying the search? e.g. Don SomeLastName -don't but how do you escape the single quote? Like this? Don SomeLastName -don't
[twitter-dev] Re: Search API: searching for Don and finding don't instead
Hi Jeffrey, Thanks for bumping this to our attention. Some of the threads fall off our radar so a prompt is always welcome. Search treats separate words as an AND search meaning a search for: Don SomeLastName will translate to: Don AND SomeLastName. For a complete phrase search you would instead want to search for: Don SomeLastName. The problem you are experiencing with Don matching Don't is, as you suggested, managed by appending -don't to the query. You don't need to escape the apostrophe and the quotes are not necessary, making your search query: Don SomeLastName -don't You can read more about the supported advanced search operators on the search site [1]. Hope that helps, Matt Harris Developer Advocate, Twitter http://twitter.com/themattharris 1. http://search.twitter.com/operators On Jun 7, 9:09 am, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: Hello Twitter, Anyone home? j On Jun 2, 11:28 pm, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: We have a user that is causing us to create a search of the form: Don SomeLastName which is returning tweets containing don't and SomeLastName. Thats a no good! Is there a decent workaround for this by modifying the search? e.g. Don SomeLastName -don't but how do you escape the single quote? Like this? Don SomeLastName -don't
[twitter-dev] Re: Search API: searching for Don and finding don't instead
Thanks Matt, Unless they've been updated lately, the docs are not clear as to how to handle contractions, so thanks for the -don't example. Given that don't is regarded as a word, we believe that search should _not_ return don't in a search for don... It's a bug in our opinion. Further, I'm not sure whether this is a problem only with contractions (that is the handling of single-quote characters), or if search reacts in weird/inconsistent/buggy ways when other special characters (e.g. single quotes, utf-8 stuff, etc) are used. Can you check whether there is consistent handling and spec for these from the search team? Thanks, Jeffrey http://www.tweettronics.com On Jun 7, 10:50 am, themattharris thematthar...@twitter.com wrote: Hi Jeffrey, Thanks for bumping this to our attention. Some of the threads fall off our radar so a prompt is always welcome. Search treats separate words as an AND search meaning a search for: Don SomeLastName will translate to: Don AND SomeLastName. For a complete phrase search you would instead want to search for: Don SomeLastName. The problem you are experiencing with Don matching Don't is, as you suggested, managed by appending -don't to the query. You don't need to escape the apostrophe and the quotes are not necessary, making your search query: Don SomeLastName -don't You can read more about the supported advanced search operators on the search site [1]. Hope that helps, Matt Harris Developer Advocate, Twitterhttp://twitter.com/themattharris 1.http://search.twitter.com/operators On Jun 7, 9:09 am, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: Hello Twitter, Anyone home? j On Jun 2, 11:28 pm, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: We have a user that is causing us to create a search of the form: Don SomeLastName which is returning tweets containing don't and SomeLastName. Thats a no good! Is there a decent workaround for this by modifying the search? e.g. Don SomeLastName -don't but how do you escape the single quote? Like this? Don SomeLastName -don't
[twitter-dev] Re: Search api returning results based on walking shortened URLS: causing problems.
I've seen the same thing with some of my own searches, and I just figured the search algo was broken, because it returns results that have absolutely nothing to do with the phrase you searched for. On May 26, 6:24 pm, Jeffrey Greenberg jeffreygreenb...@gmail.com wrote: So we have customer that is searching, for example, for hotels.com. So we use the search api and we get from Twitter a tweet that has no such text in it, but it turns out that the shortened URL contains the string 'hotels.com': Here's the tweet: Siam Bayview Hotel Pattaya, Beach Rd. from THB 2,010 incl breakfast Special Ratehttp://bit.ly/295HOIThailand hotels He're the walked bit.ly url: http://www.r24.org/patong-beach-hotels.com/pattaya/siambayview/ In this case, this match isn't good. They don't want r24.org stuff, they want hotels.com stuff... On the other hand, it's great when it really shows hotels.com stuff.. I'm not sure what the 'right thing to do is at this moment, as I'm reacting to the customer's urgency and problem in getting unrelated stuff showing up in their search... I'm not sure how I should address this: 1. recommend that twitter do some sort of mod to the search api ( I don't have a good idea at the moment about what you should do: make such url walking optional? etc?) 2. do some sort of processing on our end, and communicating about better about what search does to our customers So: a. What's ya'll thoughts on this one? b. I believe that you (twitter) walk some shorteners but not all of them: e.g. bit.ly urls and your own shortener What is the current list that you do walk? This is related to entity parsing discussion here:http://groups.google.com/group/twitter-development-talk/browse_thread... Thanks, Jeffrey Greenberg tweettronics.com
[twitter-dev] Re: Search API or Streaming API?
Woops, my bad. I meant a meta search that would make use of all third party APIs to display the results. But I got your explanation. So if I intend to process the tweets and make sense of it, the Streaming API is what I would need to take a look at. But if I intend to get the search results and just display them on my site, then I guess the search API is what I should use! Pretty much clears everything, so cool! Thanks a lot! -Nischal On May 4, 3:27 am, John Kalucki j...@twitter.com wrote: If you are going to build a search engine, you'll need all of the Tweets to search over them. For this, you'll want to take the Firehose of all public statuses. http://apiwiki.twitter.com/Streaming-API-Documentation You'll need a commercial data license to do this. Email api to get started. GAE currently does not allow standing connections to the Streaming API. Also, you'll need considerably more resources than GAE to build a search engine. You'll need dozens of cores and hundreds of spindles just to get started. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sun, May 2, 2010 at 5:28 AM, nischalshetty nischalshett...@gmail.com wrote: I plan to build a search engine which would utilize the search APIs. Should I be using the Twitter Search API or the Streaming API to do the same? What is the difference between the two and would the Streaming API work on the Google App Engine?
Re: [twitter-dev] Re: Search API or Streaming API?
Note that from GAE, your search rate will be throttled significantly, as you are sharing the Search API with every other GAE project on a single IP. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, May 4, 2010 at 12:34 AM, nischalshetty nischalshett...@gmail.com wrote: Woops, my bad. I meant a meta search that would make use of all third party APIs to display the results. But I got your explanation. So if I intend to process the tweets and make sense of it, the Streaming API is what I would need to take a look at. But if I intend to get the search results and just display them on my site, then I guess the search API is what I should use! Pretty much clears everything, so cool! Thanks a lot! -Nischal On May 4, 3:27 am, John Kalucki j...@twitter.com wrote: If you are going to build a search engine, you'll need all of the Tweets to search over them. For this, you'll want to take the Firehose of all public statuses. http://apiwiki.twitter.com/Streaming-API-Documentation You'll need a commercial data license to do this. Email api to get started. GAE currently does not allow standing connections to the Streaming API. Also, you'll need considerably more resources than GAE to build a search engine. You'll need dozens of cores and hundreds of spindles just to get started. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sun, May 2, 2010 at 5:28 AM, nischalshetty nischalshett...@gmail.com wrote: I plan to build a search engine which would utilize the search APIs. Should I be using the Twitter Search API or the Streaming API to do the same? What is the difference between the two and would the Streaming API work on the Google App Engine?
[twitter-dev] Re: Search API or Streaming API?
Oh.. alright.. I thought GAE had multiple IP addresses... hmmm... then might have to look into Amazon Thanks a lot for the info :) -Nischal On May 4, 6:29 pm, John Kalucki j...@twitter.com wrote: Note that from GAE, your search rate will be throttled significantly, as you are sharing the Search API with every other GAE project on a single IP. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Tue, May 4, 2010 at 12:34 AM, nischalshetty nischalshett...@gmail.com wrote: Woops, my bad. I meant a meta search that would make use of all third party APIs to display the results. But I got your explanation. So if I intend to process the tweets and make sense of it, the Streaming API is what I would need to take a look at. But if I intend to get the search results and just display them on my site, then I guess the search API is what I should use! Pretty much clears everything, so cool! Thanks a lot! -Nischal On May 4, 3:27 am, John Kalucki j...@twitter.com wrote: If you are going to build a search engine, you'll need all of the Tweets to search over them. For this, you'll want to take the Firehose of all public statuses. http://apiwiki.twitter.com/Streaming-API-Documentation You'll need a commercial data license to do this. Email api to get started. GAE currently does not allow standing connections to the Streaming API. Also, you'll need considerably more resources than GAE to build a search engine. You'll need dozens of cores and hundreds of spindles just to get started. -John Kaluckihttp://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sun, May 2, 2010 at 5:28 AM, nischalshetty nischalshett...@gmail.com wrote: I plan to build a search engine which would utilize the search APIs. Should I be using the Twitter Search API or the Streaming API to do the same? What is the difference between the two and would the Streaming API work on the Google App Engine?
[twitter-dev] Re: Search API - from:xxx not returning @ replies
Comcastbonnie confirms this is not unusual: http://twitter.com/ComcastBonnie/statuses/13083585494 That this error happens for some and not others is not surprising. With new focus on the Search API this type of issue can be addressed :)
Re: [twitter-dev] Re: Search API - from:xxx not returning @ replies
Probably related to this: http://groups.google.com/group/twitter-development-talk/msg/3af17ba93d66abbf On Thu, Apr 29, 2010 at 11:52, mikawhite mikawh...@me.com wrote: Comcastbonnie confirms this is not unusual: http://twitter.com/ComcastBonnie/statuses/13083585494 That this error happens for some and not others is not surprising. With new focus on the Search API this type of issue can be addressed :) -- Abraham Williams | Developer for hire | http://abrah.am @abraham | http://projects.abrah.am | http://blog.abrah.am This email is: [ ] shareable [x] ask first [ ] private.
[twitter-dev] Re: Search API - 420 increase at 17:01 PDT
This issue is now fixed. -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
[twitter-dev] Re: Search API Changes: Popular Tweets vs. Recency
Thanks, good feedback. Yep, it is always preferable to be explicit about specifying the intent. API versioning and explicit options are both good ways of doing that. The kerfuffle around the popular searches being injected happened exactly because there was previously no way to specify intent. Thus, there was an implicit intent in the search API behavior that the developers came to trust. Now we feel as if a rug is somewhat being pulled from under us. To be fair, though, if popular tweets being included by default BREAKS anybody's app in the technical sense, then maybe it's time to look in the mirror or your code. My app won't be affected by it and will continue to operate just fine. If I want, I could just add extra value to my users by presenting the popular search somehow differently, but if not, it continues to be just a bunch of results, all the same. Be liberal in what you accept (http://en.wikipedia.org/wiki/ Robustness_principle) is a good rule to follow with Twitter API as with any external data. J -- To unsubscribe, reply using remove me as the subject.
Re: [twitter-dev] Re: Search API from:username performance issues?
Hi Doug, I'm getting reports of this from:user delay happening again, so here are some relevant request/response headers and screengrabs of the results. There are some cases where it can be out of sync for up to 8-10 minutes. This is for the search query from:resourcefulmom Request Headers Host: search.twitter.com User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.17) Gecko/2009122115 Firefox/3.0.17 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://search.twitter.com/ Cookie: __utma=43838368.580929392773971800.1239516392.1262479801.1267595157.514; __utmz=43838368.1267595157.514.159.utmcsr=push.ly|utmccn=(referral)|utmcmd=referral|utmcct=/home; __utmv=43838368.lang%3A%20en; __utma=110314503.2301945900846264600.1239516535.1262063897.1269652388.170; __utmz=110314503.1258388229.140.5.utmcsr=twitter.com|utmccn=(referral)|utmcmd=referral|utmcct=/; rpp=100; __qca=1239588110-79825009-53773698; lang=all; _twitter_sess=BAh7DToMY3NyZl9pZCIlZDY4ZTY2YjI5ZDRkODgxOGM2ZWZlMWUxM2Y2MDA5%250AYzQ6DnJldHVybl90byJeaHR0cDovL3R3aXR0ZXIuY29tL29hdXRoL2F1dGhv%250Acml6ZT9vYXV0aF90b2tlbj1CbTB6c1YwZGgxTGdRWXNIcGJjNG94bnV0SnRN%250AdzJXRG1nMUVXclR4ekU6E3Bhc3N3b3JkX3Rva2VuIi00OWU2MGRhMDdhZDBk%250AZWNlNzJjNGUwNjlkNjJhYmYyN2E5NmFhYzc4Ogl1c2VyaQNUOWUiCmZsYXNo%250ASUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhhc2h7AAY6CkB1%250Ac2VkewA6B2lkIiU3YTlmZjBlY2EzNTc2MTczMGZlNTFmMjYxZTJiZWJmZDoP%250AY3JlYXRlZF9hdGwrCH0QjyInAToRdHJhbnNfcHJvbXB0MA%253D%253D--265270e77f05570f78d081813c118b21eee077b9; __utmc=43838368; __utmb=110314503.1.10.1269652388; __utmc=110314503 Response Headers Date: Sat, 27 Mar 2010 01:13:06 GMT Server: hi Status: 200 OK X-Served-From: b022 X-Runtime: 2.41047 Content-Type: text/html; charset=utf-8 X-Served-By: c070.twitter.com X-Timeline-Cache-Hit: Miss Cache-Control: max-age=15, must-revalidate, max-age=300 Expires: Sat, 27 Mar 2010 01:18:03 GMT Content-Encoding: gzip Content-Length: 16677 Vary: Accept-Encoding X-Varnish: 3015348245 Age: 0 Via: 1.1 varnish X-Cache-Svr: c070.twitter.com X-Cache: MISS Set-Cookie: rpp=100; path=/; expires=Sun, 27 Mar 2011 01:13:03 GMT lang=all; path=/; expires=Sun, 27 Mar 2011 01:13:03 GMT Connection: close Screengrab of Twitter Search results: http://grab.by/3ln7 Screengrab of Twitter profile page: http://grab.by/3ln8 Please let me know if you need more info to help debug. Thanks, -Chad On Mon, Mar 15, 2010 at 2:55 PM, twitterdoug dc...@twitter.com wrote: Hi Chad, I didn't get there in time, the results looked fine to me. Should you be able to reproduce this, could you please send more information? dumps of results would be most useful, with complete HTTP requests/ responses... best, doug On Mar 12, 6:22 pm, Chad Etzel jazzyc...@gmail.com wrote: Hi dev team, I've gotten progressively more complaints from TweetGrid users about searches in the form of from:username not updating in a timely fashion. I haven't changed my code in a while, so after investigating it appears that the search index does lag behind a bit for from: searches as compared to just keywords. Is this a bug, or intentional? Example (if you read this in time):http://twitter.com/resourcefulmom compared tohttp://search.twitter.com/search?q=from:resourcefulmom Thanks, -Chad To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
[twitter-dev] Re: Search API fails with Chinese
Pretty odd, I am able to use curl to get http://search.twitter.com/search.json?q=^_^lang=zh but have the same problem as you fetching it through Firefox/Safari. On Mar 22, 12:51 pm, Irokez iro...@gmail.com wrote: http://search.twitter.com/search.json?q=^_^lang=en - works perfectlyhttp://search.twitter.com/search.json?q=^_^lang=zh - Twitter search has timed out Is there a way to solve the problem? To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
[twitter-dev] Re: Search API fails with Chinese
This is most likely because there are extremely few results in chinese that match the query. Right now Twitter Search handles lang queries in a relatively inefficient way, so that queries for common terms that match extremely few results may time out. We can (and will) make this better, but the point is that you probably wouldn't have gotten many, if any, results for this query in any case. d On Mar 22, 12:51 pm, Irokez iro...@gmail.com wrote: http://search.twitter.com/search.json?q=^_^lang=en - works perfectlyhttp://search.twitter.com/search.json?q=^_^lang=zh - Twitter search has timed out Is there a way to solve the problem? To unsubscribe from this group, send email to twitter-development-talk+unsubscribegooglegroups.com or reply to this email with the words REMOVE ME as the subject.
[twitter-dev] Re: Search API from:username performance issues?
Hi Chad, I didn't get there in time, the results looked fine to me. Should you be able to reproduce this, could you please send more information? dumps of results would be most useful, with complete HTTP requests/ responses... best, doug On Mar 12, 6:22 pm, Chad Etzel jazzyc...@gmail.com wrote: Hi dev team, I've gotten progressively more complaints from TweetGrid users about searches in the form of from:username not updating in a timely fashion. I haven't changed my code in a while, so after investigating it appears that the search index does lag behind a bit for from: searches as compared to just keywords. Is this a bug, or intentional? Example (if you read this in time):http://twitter.com/resourcefulmom compared tohttp://search.twitter.com/search?q=from:resourcefulmom Thanks, -Chad
[twitter-dev] Re: Search API : Pagination is out of order
Looking into this. On Mar 10, 1:36 am, Hrishi bakshi.hrishik...@gmail.com wrote: Hello all, I collecting location based tweets. I am using max_id and page parameters for pagination. The ids of the tweets returned seem to be out of order. For example : Go to:http://search.twitter.com/search.json?geocode=40.70771%2C-73.948974%2... Then go to page 2 using next_page value The ids of last results from page 1 are much lower than top results of page 2. In other words page 1 returns older results than page 2. Is this a bug or am I doing it wrong? Thanks
[twitter-dev] Re: Search API rate limit IP address question
Thank you for your reply! If this were true then sometimes your request works and other times it doesn't. Is that the case? Yes, each time I run my app, it makes ~80 calls to the Search API. I can only run a full test of the app 2 or 3 times before I get the Stream Error. But if I run a partial test of only 10 or so calls, I can run it a bunch more times before getting the error. If I wait 30 minutes or so, I can continue testing...but that really affects my workflow! There are multiple requests happening here. I assume the following, which may or may not be correct: - From your browser you call your app - Your app runs some call through the twitter API - Twitter servers process the call and send it back to your app - Your app returns processed code back to your browser Yes, this is correct. From the above processes your IP address is passed through by the Twitter API to the twitter service. I'd suggest try running your request from a completely different network and see what happens. I tried running it from a friend's computer. I get the same frequency of Error, but when he changes his computer's IP address, I'm suddenly able to run the app again... How can I shift the load to my webserver's IP (the one that's whitelisted) rather than each individual computer's IP? Is it possible with Search API? Thank you!
[twitter-dev] Re: Search API Irregularity
Yes, I am receiving limit messages. I will send an email once my college project is near completion and ready to go live. Thanks for Twitter's Dev Team Help! - Will Mulligan Worcester Polytechnic Institute
Re: [twitter-dev] Re: Search API Irregularity
Something just to keep in mind. Try not to postpone integration into a higher access level stream for last b'coz you might have to process A LOT (I really mean it!) of tweets. Unless well done it could be over whelming for the application to integrate with elevated access level stream. On Tue, Feb 16, 2010 at 4:15 PM, TimeSnag wmulli...@me.com wrote: Yes, I am receiving limit messages. I will send an email once my college project is near completion and ready to go live. Thanks for Twitter's Dev Team Help! - Will Mulligan Worcester Polytechnic Institute -- Regards, Atul Kulkarni
[twitter-dev] Re: Search API Irregularity
Are you able to tell me how many more? I know that current am getting about 28 tweets per second that contain 'rt' in them. I estimated from the twitter.com search, that there are about 40-50 per second, but that is assuming the twitter.com is not limited. Thanks
Re: [twitter-dev] Re: Search API Irregularity
If you are using just track predicate, then u should be fine. But in general gardenhose floods u depending on the time of the day and various other factors nearly a few thousand tweets per hour on an average or something similar. But again it depends what u r consuming. My numbers could certainly differ with other who have done that in the past. but it is certainly significantly more than normal access level. Regards, Atul. On Tue, Feb 16, 2010 at 6:04 PM, TimeSnag wmulli...@me.com wrote: Are you able to tell me how many more? I know that current am getting about 28 tweets per second that contain 'rt' in them. I estimated from the twitter.com search, that there are about 40-50 per second, but that is assuming the twitter.com is not limited. Thanks -- Regards, Atul Kulkarni
Re: [twitter-dev] Re: Search API Irregularity
Are you receiving limit messages? If not, than the issue isn't with your Streaming API role, but rather how you are defining your search terms. You may need a broader predicate set to catch more of them If you are receiving limit messages, you can request a higher access level at a...@twitter.com. Please send a brief company description and your use case. There is a retweet stream, but it only provides explicit retweets, not informal RT style retweets. Also, the retweet stream is generally unavailable until we announce our commercial license framework, which should be soon. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sun, Feb 14, 2010 at 11:11 AM, TimeSnag wmulli...@me.com wrote: Thanks for the advice. I switched over to streaming and am getting about 25-30 tweets/sec that contain 'rt'. Based on main website search, I estimate there are about 45-50 tweets/ sec that contain 'rt'. So, I am only getting about 50% of the actual tweets. If I applied for the retweet streaming api, would that give me a higher percentage of the retweets? Or is there another way to increase that percentage? Thanks for your help.
[twitter-dev] Re: Search API Irregularity
Thanks for the advice. I switched over to streaming and am getting about 25-30 tweets/sec that contain 'rt'. Based on main website search, I estimate there are about 45-50 tweets/ sec that contain 'rt'. So, I am only getting about 50% of the actual tweets. If I applied for the retweet streaming api, would that give me a higher percentage of the retweets? Or is there another way to increase that percentage? Thanks for your help.
Re: [twitter-dev] Re: Search API Irregularity
both of those are samples -- the streaming API is a sample and the search API does not return all tweets (not all tweets are indexed by search). these are the best two options for getting a sample of all the retweets, unfortunately. On Sun, Feb 14, 2010 at 11:11 AM, TimeSnag wmulli...@me.com wrote: Thanks for the advice. I switched over to streaming and am getting about 25-30 tweets/sec that contain 'rt'. Based on main website search, I estimate there are about 45-50 tweets/ sec that contain 'rt'. So, I am only getting about 50% of the actual tweets. If I applied for the retweet streaming api, would that give me a higher percentage of the retweets? Or is there another way to increase that percentage? Thanks for your help. -- Raffi Krikorian Twitter Platform Team http://twitter.com/raffi
[twitter-dev] Re: Search API domain
You should be using search.twitter.com for all search API calls. On Jan 30, 2:05 pm, Josh Roesslein jroessl...@gmail.com wrote: Hello, I have discovered that the search methods search and trends seem to work okay with the domain api.twitter.com. But the methods trends/current, trends/daily, and trends/weekly return 401's. They only appear to work correctly on the search.twitter.com. I have opened an issue here [1]. Will all search methods eventually work on the api.twitter.com domain? Thanks. Josh [1]http://code.google.com/p/twitter-api/issues/detail?id=1413
Re: [twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting starting 1/18/2010
1) Looks like the docs got updated. 2) 400 will eventually just be for API calls that are malformed: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1 Abraham On Sat, Jan 23, 2010 at 18:40, Andy Freeman ana...@earthlink.net wrote: (1) When will http://apiwiki.twitter.com/HTTP-Response-Codes-and-Errors be updated? (2) How does 420 differ from 400? On Dec 22 2009, 4:19 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: Eventually the REST API will return the same 420 response code to indicate rate limiting. We wanted to change as little as possible to get people comfortable with the new response code. On Dec 22, 4:07 pm, Marco Kaiser kaiser.ma...@gmail.com wrote: yeah, doesn't make much sense to have two different codes indicating that the limit is exceeded... 2009/12/23 DustyReagan dustyrea...@gmail.com Will you be changing the REST API error code to match the Search API? RE: 420 = rate limit exceeded. On Dec 22, 4:44 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. Please update your response your response handler to accommodate this new behavior. Apologies for the false start last time this change was announced. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!- Hide quoted text - - Show quoted text - -- Abraham Williams | Community Advocate | http://abrah.am Project | Out Loud | http://outloud.labs.poseurtech.com This email is: [ ] shareable [x] ask first [ ] private. Sent from Seattle, WA, United States
[twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting starting 1/18/2010
(1) When will http://apiwiki.twitter.com/HTTP-Response-Codes-and-Errors be updated? (2) How does 420 differ from 400? On Dec 22 2009, 4:19 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: Eventually the REST API will return the same 420 response code to indicate rate limiting. We wanted to change as little as possible to get people comfortable with the new response code. On Dec 22, 4:07 pm, Marco Kaiser kaiser.ma...@gmail.com wrote: yeah, doesn't make much sense to have two different codes indicating that the limit is exceeded... 2009/12/23 DustyReagan dustyrea...@gmail.com Will you be changing the REST API error code to match the Search API? RE: 420 = rate limit exceeded. On Dec 22, 4:44 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. Please update your response your response handler to accommodate this new behavior. Apologies for the false start last time this change was announced. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!- Hide quoted text - - Show quoted text -
[twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting now in effect
1) When will http://apiwiki.twitter.com/HTTP-Response-Codes-and-Errors be updated? (2) How does 420 differ from 400? On Jan 23, 4:21 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: In accordance with our previous announcement, we have completed the change to Search API rate limiting response code. This change allows downstream systems to more appropriately respond to rate limiting. The original announcement follows: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!
[twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting starting 1/18/2010
i am getting this response code in my twitter search application, how to resolve the error ? On Dec 23 2009, 3:44 am, Wilhelm Bierbaum wilh...@twitter.com wrote: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. Please update your response your response handler to accommodate this new behavior. Apologies for the false start last time this change was announced. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!
[twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting starting 1/18/2010
OK ... next question ... are the rate limit HTTP headers from the REST API now ported to Search and working / documented? 2. HTTP response headers included in all REST API responses which count against the rate limit: * X-RateLimit-Limit the current limit in effect * X-RateLimit-Remaining the number of hits remaining before you are rate limited * X-RateLimit-Reset the time the current rate limiting period ends in epoch time. On Dec 22 2009, 2:44 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. Please update your response your response handler to accommodate this new behavior. Apologies for the false start last time this change was announced. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!
Re: [twitter-dev] Re: search api results down by a factor of ten since Jan 15, 2010
So, any news on the matter. This probably means that the number of search results has deliberately been reduced to give people an incentive to move to the streaming api's? -M -- Dr. Mikio Braun, Beckerstr. 11, 12157 Berlin Privat: 030 / 42 10 56 42, Büro: 030 / 314 78627, Handy: 0172 / 97 45 676
Re: [twitter-dev] Re: search api results down by a factor of ten since Jan 15, 2010
Search results are altered to improve result quality. The Streaming API exists as a full-fidelity alternative for large-scale integrations. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Wed, Jan 20, 2010 at 3:20 AM, Mikio Braun mikiobr...@googlemail.comwrote: So, any news on the matter. This probably means that the number of search results has deliberately been reduced to give people an incentive to move to the streaming api's? -M -- Dr. Mikio Braun, Beckerstr. 11, 12157 Berlin Privat: 030 / 42 10 56 42, Büro: 030 / 314 78627, Handy: 0172 / 97 45 676
Re: [twitter-dev] Re: search api results down by a factor of ten since Jan 15, 2010
Dear John, thanks for the reply. We've already started to look into the migration to the streaming API. Looks very nice so far! -M On Wed, Jan 20, 2010 at 3:41 PM, John Kalucki j...@twitter.com wrote: Search results are altered to improve result quality. The Streaming API exists as a full-fidelity alternative for large-scale integrations. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. -- Dr. Mikio Braun, Beckerstr. 11, 12157 Berlin Privat: 030 / 42 10 56 42, Büro: 030 / 314 78627, Handy: 0172 / 97 45 676
[twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting starting 1/18/2010
Will you be changing the REST API error code to match the Search API? RE: 420 = rate limit exceeded. On Dec 22, 4:44 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. Please update your response your response handler to accommodate this new behavior. Apologies for the false start last time this change was announced. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!
Re: [twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting starting 1/18/2010
yeah, doesn't make much sense to have two different codes indicating that the limit is exceeded... 2009/12/23 DustyReagan dustyrea...@gmail.com Will you be changing the REST API error code to match the Search API? RE: 420 = rate limit exceeded. On Dec 22, 4:44 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. Please update your response your response handler to accommodate this new behavior. Apologies for the false start last time this change was announced. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!
[twitter-dev] Re: Search API: new HTTP response code 420 for rate limiting starting 1/18/2010
Eventually the REST API will return the same 420 response code to indicate rate limiting. We wanted to change as little as possible to get people comfortable with the new response code. On Dec 22, 4:07 pm, Marco Kaiser kaiser.ma...@gmail.com wrote: yeah, doesn't make much sense to have two different codes indicating that the limit is exceeded... 2009/12/23 DustyReagan dustyrea...@gmail.com Will you be changing the REST API error code to match the Search API? RE: 420 = rate limit exceeded. On Dec 22, 4:44 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: We're changing the response code sent back by the Search API when the rate limit has been exceeded. At present, it is impossible to distinguish rate limit responses from other error conditions in responses from the Search API -- this is what we're trying to fix. Starting Monday, January 18th, 2010 the Search API will respond with error code 420 in the event that the number of requests you have made exceeds the quota afforded by your assigned rate limit. Please update your response your response handler to accommodate this new behavior. Apologies for the false start last time this change was announced. If you have any questions, please feel free to post them on twitter-development-talk. Thanks!
[twitter-dev] Re: Search API questions
@AJ Chen You are 100% correct when you say that it’s the user’s responsibility to clean up duplicates in the search results. My issue is not so much about there being duplicates, but the fact that there are so many of them. My concept of search is that if there have been new tweets posted, say 30 odd since I last queried search, I ought to get the new tweets on my next query. What I shouldn’t be getting is, search results from say two hours ago whenever I query search. Maybe I am wrong here, but that’s how I expected the search API to work. On the issue of Rate Limiting, I am really not sure what the rate limit would be, since the documentation does not give a clear picture of what that limit is. The documentation (http://apiwiki.twitter.com/ Rate-limiting) merely hints that it is significantly higher than the 150 requests per hour limit for the REST API. Considering this, I don’t think my application / script should be exceeding that limit since I only make 4 requests per hour. Anyway, would really appreciate it if someone could point me in the right direction, or at least let me know if I am trying to the wrong thing with the search API. Regards, Elroy On Dec 3, 6:31 am, AJ Chen cano...@gmail.com wrote: unless I miss something, it's usually user's responsibility to dedup returned tweets on the client side. if you see duplicates between two feeds, just remove the duplicates. this is what client application should have in any case. if you see no fresh tweets but only old tweets, there may be a possibility that twitter returns only cashed results because you api calls exceed rate-limit. I'm not sure, though. does any one know about rate-limit for using search feedhttp://search.twitter.com/search.atomhttp://search.twitter.com/search.atom?geocode=19.017656%2C72.856178%2. ? -aj On Tue, Dec 1, 2009 at 8:49 PM, enygmatic enygma...@gmail.com wrote: Hi, Raffi Were you able to raise the cache issue with the search team? Seems the problem is worse than I thought. I have run my script (getting 25 results from search every 15 minutes, for Mumbai) for two days. The first day had 71% duplicate results due to the caching issue, while the second day fetched an amazing 90% duplicates. With these kind of results, I think it’s probably quite useless for me to even use the search API . So would appreciate if you could let me know if there is a chance that this issue may be resolved in the near future or if location specific streams would be available via the streaming API anytime soon. I understand that the twitter dev team has a lot on its hands, so it would be understandable if this isn’t anywhere in the list of features they intend to ship out in the near future. However, would definitely appreciate it if you could let me know if anything could be done or not. Thanks and Regards, Elroy Serrao On Nov 28, 7:45 pm, Raffi Krikorian ra...@twitter.com wrote: unfortunately, there is no (current) way to subscribe to the streaming API for a particular location. as for the caching issue on the search, that's unfortunate, and i'll try to raise the issue with the search team next week. @Abraham I actually use the geocode with the search api for my script, so using the search api isn't my problem. My problem is that I get stale results from the search cache, even when querying after a sufficient interval. Also the stale results seem hours old (at times, in fact yesterday at 23:00 hours I got a few results that were from 22:00-22:30 hours. Didn't have the problem when using twitter search from the browser). To overcome this Raffi Krikorian suggested using the streaming api instead of the search api. My question was - how do i get a location specific stream using the streaming api. From the streaming api docs, there doesn't seem a way to do this at the moment, which kind of defeats my purpose as I need to the deploy the script in the next one week or so. Guess I'll have to live with the stale results... Anyway thanks for the help. On Nov 28, 12:40 am, Abraham Williams 4bra...@gmail.com wrote: On Fri, Nov 27, 2009 at 12:38, enygmatic enygma...@gmail.com wrote: From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Check out the search operators:http://search.twitter.com/operators For example:http://search.twitter.com/search?q=near:NYC+within:15mi Abraham -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abraham Project | Awesome Lists |http://twitterli.st This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi-
[twitter-dev] Re: Search API: new HTTP response codes for rate limiting starting 12/16
I would have to agree with mat. But to each their own. The return codes frequently make no sense from twitter, so i guess the fact that it doesn't make sense it is irrelevant, so long as it is consistent. On Dec 3, 6:29 pm, mat mat.st...@gmail.com wrote: Given that 400 is bad request, and the client SHOULD NOT repeat the request without modifications (w3.org's emphasis), and 503 means service unavailable, try again later, and can include a retry-after header, would it not have made more sense to change the response code of the REST API to the more correct one? On Dec 3, 10:41 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: In an effort to simplify our APIs, we are standardizing the response codes returned by our various systems. Historically, the Search API has returned 503 for rate limiting whereas the REST API has returned 400. So, we are changing the response codes sent back from the Search API. Starting Wednesday, December 16th, 2009 the search API will respond with error code 400 in the event that the number of requests you have made exceeds the quota afforded by your rate limit. Please update your response handler accordingly. If you have any questions, please feel free to post them on twitter- development-talk.
[twitter-dev] Re: Search API: new HTTP response codes for rate limiting starting 12/16
Given that 400 is bad request, and the client SHOULD NOT repeat the request without modifications (w3.org's emphasis), and 503 means service unavailable, try again later, and can include a retry-after header, would it not have made more sense to change the response code of the REST API to the more correct one? On Dec 3, 10:41 pm, Wilhelm Bierbaum wilh...@twitter.com wrote: In an effort to simplify our APIs, we are standardizing the response codes returned by our various systems. Historically, the Search API has returned 503 for rate limiting whereas the REST API has returned 400. So, we are changing the response codes sent back from the Search API. Starting Wednesday, December 16th, 2009 the search API will respond with error code 400 in the event that the number of requests you have made exceeds the quota afforded by your rate limit. Please update your response handler accordingly. If you have any questions, please feel free to post them on twitter- development-talk.
Re: [twitter-dev] Re: Search API questions
unless I miss something, it's usually user's responsibility to dedup returned tweets on the client side. if you see duplicates between two feeds, just remove the duplicates. this is what client application should have in any case. if you see no fresh tweets but only old tweets, there may be a possibility that twitter returns only cashed results because you api calls exceed rate-limit. I'm not sure, though. does any one know about rate-limit for using search feed http://search.twitter.com/search.atomhttp://search.twitter.com/search.atom?geocode=19.017656%2C72.856178%2. ? -aj On Tue, Dec 1, 2009 at 8:49 PM, enygmatic enygma...@gmail.com wrote: Hi, Raffi Were you able to raise the cache issue with the search team? Seems the problem is worse than I thought. I have run my script (getting 25 results from search every 15 minutes, for Mumbai) for two days. The first day had 71% duplicate results due to the caching issue, while the second day fetched an amazing 90% duplicates. With these kind of results, I think it’s probably quite useless for me to even use the search API . So would appreciate if you could let me know if there is a chance that this issue may be resolved in the near future or if location specific streams would be available via the streaming API anytime soon. I understand that the twitter dev team has a lot on its hands, so it would be understandable if this isn’t anywhere in the list of features they intend to ship out in the near future. However, would definitely appreciate it if you could let me know if anything could be done or not. Thanks and Regards, Elroy Serrao On Nov 28, 7:45 pm, Raffi Krikorian ra...@twitter.com wrote: unfortunately, there is no (current) way to subscribe to the streaming API for a particular location. as for the caching issue on the search, that's unfortunate, and i'll try to raise the issue with the search team next week. @Abraham I actually use the geocode with the search api for my script, so using the search api isn't my problem. My problem is that I get stale results from the search cache, even when querying after a sufficient interval. Also the stale results seem hours old (at times, in fact yesterday at 23:00 hours I got a few results that were from 22:00-22:30 hours. Didn't have the problem when using twitter search from the browser). To overcome this Raffi Krikorian suggested using the streaming api instead of the search api. My question was - how do i get a location specific stream using the streaming api. From the streaming api docs, there doesn't seem a way to do this at the moment, which kind of defeats my purpose as I need to the deploy the script in the next one week or so. Guess I'll have to live with the stale results... Anyway thanks for the help. On Nov 28, 12:40 am, Abraham Williams 4bra...@gmail.com wrote: On Fri, Nov 27, 2009 at 12:38, enygmatic enygma...@gmail.com wrote: From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Check out the search operators:http://search.twitter.com/operators For example:http://search.twitter.com/search?q=near:NYC+within:15mi Abraham -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abraham Project | Awesome Lists |http://twitterli.st This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi- Hide quoted text - - Show quoted text - -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org @web2express on twitter Palo Alto, CA, USA 650-283-4091 *Monitor realtime web and follow trending topics with semantic intelligence*
[twitter-dev] Re: Search API questions
Hi, Raffi Were you able to raise the cache issue with the search team? Seems the problem is worse than I thought. I have run my script (getting 25 results from search every 15 minutes, for Mumbai) for two days. The first day had 71% duplicate results due to the caching issue, while the second day fetched an amazing 90% duplicates. With these kind of results, I think it’s probably quite useless for me to even use the search API . So would appreciate if you could let me know if there is a chance that this issue may be resolved in the near future or if location specific streams would be available via the streaming API anytime soon. I understand that the twitter dev team has a lot on its hands, so it would be understandable if this isn’t anywhere in the list of features they intend to ship out in the near future. However, would definitely appreciate it if you could let me know if anything could be done or not. Thanks and Regards, Elroy Serrao On Nov 28, 7:45 pm, Raffi Krikorian ra...@twitter.com wrote: unfortunately, there is no (current) way to subscribe to the streaming API for a particular location. as for the caching issue on the search, that's unfortunate, and i'll try to raise the issue with the search team next week. @Abraham I actually use the geocode with the search api for my script, so using the search api isn't my problem. My problem is that I get stale results from the search cache, even when querying after a sufficient interval. Also the stale results seem hours old (at times, in fact yesterday at 23:00 hours I got a few results that were from 22:00-22:30 hours. Didn't have the problem when using twitter search from the browser). To overcome this Raffi Krikorian suggested using the streaming api instead of the search api. My question was - how do i get a location specific stream using the streaming api. From the streaming api docs, there doesn't seem a way to do this at the moment, which kind of defeats my purpose as I need to the deploy the script in the next one week or so. Guess I'll have to live with the stale results... Anyway thanks for the help. On Nov 28, 12:40 am, Abraham Williams 4bra...@gmail.com wrote: On Fri, Nov 27, 2009 at 12:38, enygmatic enygma...@gmail.com wrote: From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Check out the search operators:http://search.twitter.com/operators For example:http://search.twitter.com/search?q=near:NYC+within:15mi Abraham -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abraham Project | Awesome Lists |http://twitterli.st This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi- Hide quoted text - - Show quoted text -
[twitter-dev] Re: Search API questions
@Abraham I actually use the geocode with the search api for my script, so using the search api isn't my problem. My problem is that I get stale results from the search cache, even when querying after a sufficient interval. Also the stale results seem hours old (at times, in fact yesterday at 23:00 hours I got a few results that were from 22:00-22:30 hours. Didn't have the problem when using twitter search from the browser). To overcome this Raffi Krikorian suggested using the streaming api instead of the search api. My question was - how do i get a location specific stream using the streaming api. From the streaming api docs, there doesn't seem a way to do this at the moment, which kind of defeats my purpose as I need to the deploy the script in the next one week or so. Guess I'll have to live with the stale results... Anyway thanks for the help. On Nov 28, 12:40 am, Abraham Williams 4bra...@gmail.com wrote: On Fri, Nov 27, 2009 at 12:38, enygmatic enygma...@gmail.com wrote: From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Check out the search operators:http://search.twitter.com/operators For example:http://search.twitter.com/search?q=near:NYC+within:15mi Abraham -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abraham Project | Awesome Lists |http://twitterli.st This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States
Re: [twitter-dev] Re: Search API questions
unfortunately, there is no (current) way to subscribe to the streaming API for a particular location. as for the caching issue on the search, that's unfortunate, and i'll try to raise the issue with the search team next week. @Abraham I actually use the geocode with the search api for my script, so using the search api isn't my problem. My problem is that I get stale results from the search cache, even when querying after a sufficient interval. Also the stale results seem hours old (at times, in fact yesterday at 23:00 hours I got a few results that were from 22:00-22:30 hours. Didn't have the problem when using twitter search from the browser). To overcome this Raffi Krikorian suggested using the streaming api instead of the search api. My question was - how do i get a location specific stream using the streaming api. From the streaming api docs, there doesn't seem a way to do this at the moment, which kind of defeats my purpose as I need to the deploy the script in the next one week or so. Guess I'll have to live with the stale results... Anyway thanks for the help. On Nov 28, 12:40 am, Abraham Williams 4bra...@gmail.com wrote: On Fri, Nov 27, 2009 at 12:38, enygmatic enygma...@gmail.com wrote: From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Check out the search operators:http://search.twitter.com/operators For example:http://search.twitter.com/search?q=near:NYC+within:15mi Abraham -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abraham Project | Awesome Lists |http://twitterli.st This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi
[twitter-dev] Re: Search API questions
the streaming API would be ideal for my purposes, so will eagerly wait and see what new features the twitter api dev team adds before the final release. Till then, search api is what I will use. Thanks a lot Raffi, for trying to raise the issue with the search team. Regards, Elroy On Nov 28, 7:45 pm, Raffi Krikorian ra...@twitter.com wrote: unfortunately, there is no (current) way to subscribe to the streaming API for a particular location. as for the caching issue on the search, that's unfortunate, and i'll try to raise the issue with the search team next week. @Abraham I actually use the geocode with the search api for my script, so using the search api isn't my problem. My problem is that I get stale results from the search cache, even when querying after a sufficient interval. Also the stale results seem hours old (at times, in fact yesterday at 23:00 hours I got a few results that were from 22:00-22:30 hours. Didn't have the problem when using twitter search from the browser). To overcome this Raffi Krikorian suggested using the streaming api instead of the search api. My question was - how do i get a location specific stream using the streaming api. From the streaming api docs, there doesn't seem a way to do this at the moment, which kind of defeats my purpose as I need to the deploy the script in the next one week or so. Guess I'll have to live with the stale results... Anyway thanks for the help. On Nov 28, 12:40 am, Abraham Williams 4bra...@gmail.com wrote: On Fri, Nov 27, 2009 at 12:38, enygmatic enygma...@gmail.com wrote: From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Check out the search operators:http://search.twitter.com/operators For example:http://search.twitter.com/search?q=near:NYC+within:15mi Abraham -- Abraham Williams | Community Evangelist |http://web608.org Hacker |http://abrah.am|http://twitter.com/abraham Project | Awesome Lists |http://twitterli.st This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi
[twitter-dev] Re: Search API questions
I got some requests to post the query that I am using: here is the query : http://search.twitter.com/search.atom?geocode=19.017656%2C72.856178%2C15.0mirpp=25 Do correct me if I am not querying or using the API correctly. (Should have been my first question actually :) ) Also here is a sample of the output from my ruby script. It will give you an idea of the stale results that I am getting. The script was run at approximately 21:37 IST. As you can see, I'm getting tweets all the way back to 14:00 hours in the afternoon. I'm pretty sure there are more tweets for my location. I'm querying for tweets originating out of Mumbai, and by querying through twitter search I have noticed that there are at least 40-50 tweets posted every 2 minutes or so. Output follows: Date-Day-Hour-Minute-Tweet-User-Hashtags(csv, if any)- source of tweet (All date/time info below is in IST) 2009-11-28 Saturday21 27 @Abhishek_Rai I too am huge fan of quizzing.. do let me kno if u find anythin interesting. ty Shakti_Shetty (Shakti Shetty) web 2009-11-28 Saturday21 21 @surubhi hallow darlin, 'm fine doin great...how about u?dacku87 (darshan thacker) mobile web 2009-11-28 Saturday20 40 powai mocha so full of people, smaloe conversations and music.. sumagambs (Sumit Singh Gambhir) web 2009-11-28 Saturday20 25 @thetruboy idk we'll see. Ari should be home by then ronniebaby010 (Princess)UberTwitter 2009-11-28 Saturday19 54 friends do look up www.clickthehorror.com - the website for my new film distirbuted by PNC has been launched - look 4ward to feedbacks sangeethsivan (sangeeth sivan) web 2009-11-28 Saturday19 54 I'm guessing @Netra and @prolificd are the two few Twitterers who've had multi-city tweetups. How cool is that. National figures! b50 (Bombay Addict) Tweetie 2009-11-28 Saturday19 36 RT: Trupti's Blog: What Commercial Floor Mats Offer: One of the best ways to keep any p.. http://bit.ly/6sZWJg #blog MishraNatty (Natasha Mishra)blogtwitterfeed 2009-11-28 Saturday19 09 @mattyza when launched back in 2005, the Xbox 360 was available in Core and Pro. Now it's Arcade and Elite. Same difference!aalaap (Aalaap Ghag)Tweetie 2009-11-28 Saturday19 05 Profit with Google, Twitter amp; affiliate marketing http://snipurl.com/tet1r Tiifani_Lurid (Tiifani Lurid) API 2009-11-28 Saturday18 35 Just voted OOiZiT.com for Best Online Music Label http://mashable.com/owa #openwebawardsankit_9oct (Ankit Khandelwal) openwebawards Mashable Connect 2009-11-28 Saturday18 35 @reginafetalvero HAHA. YUHH. Gift ko ah? :quot;gt; Jhoriiliee (Jorylie Cando) web 2009-11-28 Saturday18 24 @Tweet_Words JAGGERY PALM gannirules (gaanish) Snaptu 2009-11-28 Saturday17 34 @Karan_Talwar pls post that if you get an answer. champbox (champbox) Tweets60 2009-11-28 Saturday17 34 Just Got Home! :) Wee. Had FUN tonight! :) HBD kathy! Sayang wala si Beb, complete na sana.Jhoriiliee (Jorylie Cando) web 2009-11-28 Saturday17 34 I'm listening to Kurbaan: Kurbaan Hua (Soundtrack) - @Spinlet kmadvani (Kunal M Advani) API 2009-11-28 Saturday17 03 Eastern Province Under-19s 322/7 amp; 185/5 v South Western Districts Under-19s 92/10 amp; 152/10 *: Eastern Province.. http://bit.ly/4rS1iA venky888 (venkatesh iyer) twitterfeed 2009-11-28 Saturday16 52 Hey tweeps..Rocket Singh pics http://www.yashrajfilms.com/microsites/rocketsingh/fullpage.html check them out! ShazahnPadamsee (Shazahn Padamsee) web 2009-11-28 Saturday16 08 Started IE assignment jyotiswaroopr (Jyoti Swaroop Repaka) Digsby 2009-11-28 Saturday15 24 @PaulaAbdul Love you more than anything in this world. Thanks for being a huge part of my life. lt;3 LuvPaula (Anahita Abdul Cowell) web 2009-11-28 Saturday15 18 @richa_august84 fan of purane hindi gaane, hmm? me too!! sonali_k (sonali_k) web 2009-11-28 Saturday14 54 Fruits and Vegetables for energyzing the Solar Plexus Chakra: http://bit.ly/4NQV9M AnamikaS (Anamika S) web 2009-11-28 Saturday14 52 I'm off to read and then sleep. Don't dare disturb my slumber. eyemanut87 (Moo)Snaptu 2009-11-28 Saturday14 52 White House gate-crashers met Obama, PM: American couple Michaele and Tareq Salahi, who gate-crashed into a State D...
[twitter-dev] Re: Search API questions
Hi Elroy, I tried your query from python several times within the same minute. After running the query several times in a row I start getting fresh results and they remain fresh for a while. I tried changing the least significant decimal to make it a different query and I get stale results immediately. Switching back yields fresh results. This to me suggests that there may be two search tiers: one for low- frequency queries that probably searches a subset of tweets, and another one for frequent ones that searches everything and has an LRU cache of important queries. It seems that we can force queries into the LRU cache of the good tier by querying frequently enough. When I stop querying for three minutes or so I see the old results again. The question for the search team is how to have your query treated as an important one without abusing the API. Diego Diego On Nov 28, 1:18 pm, enygmatic enygma...@gmail.com wrote: I got some requests to post the query that I am using: here is the query :http://search.twitter.com/search.atom?geocode=19.017656%2C72.856178%2... Do correct me if I am not querying or using the API correctly. (Should have been my first question actually :) ) Also here is a sample of the output from my ruby script. It will give you an idea of the stale results that I am getting. The script was run at approximately 21:37 IST. As you can see, I'm getting tweets all the way back to 14:00 hours in the afternoon. I'm pretty sure there are more tweets for my location. I'm querying for tweets originating out of Mumbai, and by querying through twitter search I have noticed that there are at least 40-50 tweets posted every 2 minutes or so. Output follows: Date-Day-Hour-Minute-Tweet-User-Hashtags(csv, if any)- source of tweet (All date/time info below is in IST) 2009-11-28 Saturday 21 27 �...@abhishek_rai I too am huge fan of quizzing.. do let me kno if u find anythin interesting. ty Shakti_Shetty (Shakti Shetty) web 2009-11-28 Saturday 21 21 �...@surubhi hallow darlin, 'm fine doin great...how about u? dacku87 (darshan thacker) mobile web 2009-11-28 Saturday 20 40 powai mocha so full of people, smaloe conversations and music.. sumagambs (Sumit Singh Gambhir) web 2009-11-28 Saturday 20 25 �...@thetruboy idk we'll see. Ari should be home by then ronniebaby010 (Princess) UberTwitter 2009-11-28 Saturday 19 54 friends do look upwww.clickthehorror.com- the website for my new film distirbuted by PNC has been launched - look 4ward to feedbacks sangeethsivan (sangeeth sivan) web 2009-11-28 Saturday 19 54 I'm guessing @Netra and @prolificd are the two few Twitterers who've had multi-city tweetups. How cool is that. National figures! b50 (Bombay Addict) Tweetie 2009-11-28 Saturday 19 36 RT: Trupti's Blog: What Commercial Floor Mats Offer: One of the best ways to keep any p..http://bit.ly/6sZWJg #blog MishraNatty (Natasha Mishra) blog twitterfeed 2009-11-28 Saturday 19 09 �...@mattyza when launched back in 2005, the Xbox 360 was available in Core and Pro. Now it's Arcade and Elite. Same difference! aalaap (Aalaap Ghag) Tweetie 2009-11-28 Saturday 19 05 Profit with Google, Twitter amp; affiliate marketinghttp://snipurl.com/tet1r Tiifani_Lurid (Tiifani Lurid) API 2009-11-28 Saturday 18 35 Just voted OOiZiT.com for Best Online Music Labelhttp://mashable.com/owa#openwebawards ankit_9oct (Ankit Khandelwal) openwebawards Mashable Connect 2009-11-28 Saturday 18 35 �...@reginafetalvero HAHA. YUHH. Gift ko ah? :quot;gt; Jhoriiliee (Jorylie Cando) web 2009-11-28 Saturday 18 24 �...@tweet_words JAGGERY PALM gannirules (gaanish) Snaptu 2009-11-28 Saturday 17 34 �...@karan_talwar pls post that if you get an answer. champbox (champbox) Tweets60 2009-11-28 Saturday 17 34 Just Got Home! :) Wee. Had FUN tonight! :) HBD kathy! Sayang wala si Beb, complete na sana. Jhoriiliee (Jorylie Cando) web 2009-11-28 Saturday 17 34 I'm listening to Kurbaan: Kurbaan Hua (Soundtrack) - @Spinlet kmadvani (Kunal M Advani) API 2009-11-28 Saturday 17 03 Eastern Province Under-19s 322/7 amp; 185/5 v South Western Districts Under-19s 92/10 amp; 152/10 *: Eastern Province..http://bit.ly/4rS1iAvenky888 (venkatesh iyer) twitterfeed 2009-11-28 Saturday 16 52 Hey tweeps..Rocket Singh picshttp://www.yashrajfilms.com/microsites/rocketsingh/fullpage.html check them out! ShazahnPadamsee (Shazahn
[twitter-dev] Re: Search API questions
Hi Everyone, I've been running my script as a cron task (every 15 minutes) since last evening. So far I've got about 1375 results logged, out of which 973 are duplicates (meaning stale entries)...a staggering 70.7076% or approximately 71%. This is way more than expected..so a shout out to the development team - Is there anyway to solve this problem, get around it ? @Diego, thanks a lot for confirming what I found. Also I tried querying frequently like you suggested, and yes I do hit good results more frequently. I didn't get the idea of the least significant decimal - are u referring to the geocode? @twitter dev team I do agree with Diego, there is got to be a way of getting good search results without finding ways to trick the API. Even with a cache, I see no reason why I should be getting results from over 6 hours ago for my search query. Regards, Elroy On Nov 28, 10:16 pm, dbasch dba...@gmail.com wrote: Hi Elroy, I tried your query from python several times within the same minute. After running the query several times in a row I start getting fresh results and they remain fresh for a while. I tried changing the least significant decimal to make it a different query and I get stale results immediately. Switching back yields fresh results. This to me suggests that there may be two search tiers: one for low- frequency queries that probably searches a subset of tweets, and another one for frequent ones that searches everything and has an LRU cache of important queries. It seems that we can force queries into the LRU cache of the good tier by querying frequently enough. When I stop querying for three minutes or so I see the old results again. The question for the search team is how to have your query treated as an important one without abusing the API. Diego Diego On Nov 28, 1:18 pm, enygmatic enygma...@gmail.com wrote: I got some requests to post the query that I am using: here is the query :http://search.twitter.com/search.atom?geocode=19.017656%2C72.856178%2... Do correct me if I am not querying or using the API correctly. (Should have been my first question actually :) ) Also here is a sample of the output from my ruby script. It will give you an idea of the stale results that I am getting. The script was run at approximately 21:37 IST. As you can see, I'm getting tweets all the way back to 14:00 hours in the afternoon. I'm pretty sure there are more tweets for my location. I'm querying for tweets originating out of Mumbai, and by querying through twitter search I have noticed that there are at least 40-50 tweets posted every 2 minutes or so. Output follows: Date-Day-Hour-Minute-Tweet-User-Hashtags(csv, if any)- source of tweet (All date/time info below is in IST) 2009-11-28 Saturday 21 27 �...@abhishek_rai I too am huge fan of quizzing.. do let me kno if u find anythin interesting. ty Shakti_Shetty (Shakti Shetty) web 2009-11-28 Saturday 21 21 �...@surubhi hallow darlin, 'm fine doin great...how about u? dacku87 (darshan thacker) mobile web 2009-11-28 Saturday 20 40 powai mocha so full of people, smaloe conversations and music.. sumagambs (Sumit Singh Gambhir) web 2009-11-28 Saturday 20 25 �...@thetruboy idk we'll see. Ari should be home by then ronniebaby010 (Princess) UberTwitter 2009-11-28 Saturday 19 54 friends do look upwww.clickthehorror.com- the website for my new film distirbuted by PNC has been launched - look 4ward to feedbacks sangeethsivan (sangeeth sivan) web 2009-11-28 Saturday 19 54 I'm guessing @Netra and @prolificd are the two few Twitterers who've had multi-city tweetups. How cool is that. National figures! b50 (Bombay Addict) Tweetie 2009-11-28 Saturday 19 36 RT: Trupti's Blog: What Commercial Floor Mats Offer: One of the best ways to keep any p..http://bit.ly/6sZWJg #blog MishraNatty (Natasha Mishra) blog twitterfeed 2009-11-28 Saturday 19 09 �...@mattyza when launched back in 2005, the Xbox 360 was available in Core and Pro. Now it's Arcade and Elite. Same difference! aalaap (Aalaap Ghag) Tweetie 2009-11-28 Saturday 19 05 Profit with Google, Twitter amp; affiliate marketinghttp://snipurl.com/tet1r Tiifani_Lurid (Tiifani Lurid) API 2009-11-28 Saturday 18 35 Just voted OOiZiT.com for Best Online Music Labelhttp://mashable.com/owa#openwebawardsankit_9oct (Ankit Khandelwal) openwebawards Mashable Connect 2009-11-28 Saturday 18 35 �...@reginafetalvero HAHA. YUHH. Gift ko ah? :quot;gt; Jhoriiliee (Jorylie Cando) web 2009-11-28 Saturday 18 24
Re: [twitter-dev] Re: Search API questions
Just a couple of queries: I'm using the Atom format for search results (As mentioned on http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search) . I get the published date in the atom feed. So I am not sure what you mean by created_at:Fri, 27 Nov 2009 00:06:44 +. The format available in the atom feed is like this 2009-11-27T04:45:03Z. Do you mean the JSON format or are you referring to the search results returned by the streaming API ? Oddly though if I viewed the same feed in my browser, I could see the correct local times reported. Maybe a browser thing I guess...Anyway, converting the time reported to my timezone, shouldn't be that much of a problem I guess. time reported as 2009-11-27T04:45:03Z is in ISO8601 (http://en.wikipedia.org/wiki/ISO_8601 ), and the Z at the end means Zulu time (otherwise known as UTC). i wouldn't be all that surprised that if a browser, when encountering an atom feed, converts the time into local time. The streaming API seems like a good idea. Probably will consider shifting to it. In the meantime, does anyone have any ideas about my first problem? Any idea as to why I get some stale results (some times a couple of hours old) when I query with the API and the latest results when I query using Twitter advanced search? Or will switching to the feed generated for the advanced search results, instead of using the API solve my problem ? the search API does have a cache on it, specifically because there are a lot of applications which instead of using the streaming API are hammering the search API instead. you are probably seeing a cache hit as the search result. -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi
[twitter-dev] Re: Search API questions
@Raffi, thanks for the reply. I now convert the time from UTC to my local time zone, so my time zone problem is sorted out. On the issue of search, been going through the streaming api docs. From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Anyway, I guess I will have to live with the stale results from cache for now. Thanks for the help. On Nov 27, 7:44 pm, Raffi Krikorian ra...@twitter.com wrote: Just a couple of queries: I'm using the Atom format for search results (As mentioned onhttp://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search) . I get the published date in the atom feed. So I am not sure what you mean by created_at:Fri, 27 Nov 2009 00:06:44 +. The format available in the atom feed is like this 2009-11-27T04:45:03Z. Do you mean the JSON format or are you referring to the search results returned by the streaming API ? Oddly though if I viewed the same feed in my browser, I could see the correct local times reported. Maybe a browser thing I guess...Anyway, converting the time reported to my timezone, shouldn't be that much of a problem I guess. time reported as 2009-11-27T04:45:03Z is in ISO8601 (http://en.wikipedia.org/wiki/ISO_8601 ), and the Z at the end means Zulu time (otherwise known as UTC). i wouldn't be all that surprised that if a browser, when encountering an atom feed, converts the time into local time. The streaming API seems like a good idea. Probably will consider shifting to it. In the meantime, does anyone have any ideas about my first problem? Any idea as to why I get some stale results (some times a couple of hours old) when I query with the API and the latest results when I query using Twitter advanced search? Or will switching to the feed generated for the advanced search results, instead of using the API solve my problem ? the search API does have a cache on it, specifically because there are a lot of applications which instead of using the streaming API are hammering the search API instead. you are probably seeing a cache hit as the search result. -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi
Re: [twitter-dev] Re: Search API questions
On Fri, Nov 27, 2009 at 12:38, enygmatic enygma...@gmail.com wrote: From what I have gone through so far, there doesn't seem to be a way to query for status updates from a certain geographical location, say limited to a city. I may be mistaken here, so do correct me if I am wrong. Check out the search operators: http://search.twitter.com/operators For example: http://search.twitter.com/search?q=near:NYC+within:15mi Abraham -- Abraham Williams | Community Evangelist | http://web608.org Hacker | http://abrah.am | http://twitter.com/abraham Project | Awesome Lists | http://twitterli.st This email is: [ ] blogable [x] ask first [ ] private. Sent from Madison, WI, United States
[twitter-dev] Re: Search API questions
@Raffi, Thanks for the info. Just a couple of queries: I'm using the Atom format for search results (As mentioned on http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search). I get the published date in the atom feed. So I am not sure what you mean by created_at:Fri, 27 Nov 2009 00:06:44 +. The format available in the atom feed is like this 2009-11-27T04:45:03Z. Do you mean the JSON format or are you referring to the search results returned by the streaming API ? Oddly though if I viewed the same feed in my browser, I could see the correct local times reported. Maybe a browser thing I guess...Anyway, converting the time reported to my timezone, shouldn't be that much of a problem I guess. The streaming API seems like a good idea. Probably will consider shifting to it. In the meantime, does anyone have any ideas about my first problem? Any idea as to why I get some stale results (some times a couple of hours old) when I query with the API and the latest results when I query using Twitter advanced search? Or will switching to the feed generated for the advanced search results, instead of using the API solve my problem ? Regards, Elroy
[twitter-dev] Re: Search API geocode parameter dead
hi em. thanks for the error report, and we'll dig into it further -- but that seems like it was a transient error. i can, at this moment, hit that link and it seems to work. Dear all, from today Search API's geocode parameter (http://apiwiki.twitter.com/ Twitter-Search-API-Method%3A-search) does not work anymore. It only shows an error: hash errorCouldn't find Status with ID=2954291578/error /hash Example link: http://search.twitter.com/search.atom?geocode=40.757929%2C-73.985506%2C25km Did i miss an announcement? Is there a new alternative in the Geocode API? Thanks, Em -- Raffi Krikorian Twitter Platform Team ra...@twitter.com | @raffi
[twitter-dev] Re: Search API: max_id and page parameters giving very weird results
Except now that I look at it a day later, the results have completely changed, and seem to be in order. Why would the results change over time when the same max_id is set, and was valid at the time of the query? Are the ids of tweets not generated in ascending order? On Nov 3, 3:17 pm, TripleM stephenmerri...@gmail.com wrote: I've been trying to write a script to use the max_id parameter to loop through all 15 pages of results (with 100 results per page) without getting in troubles with grabbing the same tweet multiple times. Every time I do so, I find that not only are there a couple of duplicates on page 1 and 2, but also that the last tweet on page 1 is well further into the future, and has a lower ID, than a bunch of tweets on page 2. For example, consider these two, both with the same max_id but page = 1 and page = 2 respectively: http://search.twitter.com/search?rpp=100page=1geocode=-40.900557,17...http://search.twitter.com/search?rpp=100page=2geocode=-40.900557,17... (Or if you prefer json links which are what I am actually using, but I see the same thing on the above ones which are easier to describe:http://search.twitter.com/search.json?q=rpp=100geocode=-40.900557,1... Request:http://search.twitter.com/search.json?q=rpp=100geocode=-40.900557,1...) The first result on page 2 above was posted about 4 hours before the last tweet on page 1. There are also duplicates, eg AshleyGray00: Fireworks! I've been trying to figure this bug out for a while as I'm sure I'm missing something obvious but I'm completely stumped. Does anyone have any clue what is going on here? The only other threads I have found are about people trying to combine since_id and max_id which I know is not allowed, so I can't find anyone else having similar problems.
[twitter-dev] Re: Search API: max_id and page parameters giving very weird results
Apologies for the multiple posts, but as the above links no longer show the problem, you can replicate as follows: Go to http://search.twitter.com/search?rpp=100page=1geocode=-40.900557,174.885971,1000km Note how long ago the last tweet on that page was posted. Click 'Older' at the bottom. The first tweets on that page are much newer than the last ones on page 1. On Nov 3, 3:17 pm, TripleM stephenmerri...@gmail.com wrote: I've been trying to write a script to use the max_id parameter to loop through all 15 pages of results (with 100 results per page) without getting in troubles with grabbing the same tweet multiple times. Every time I do so, I find that not only are there a couple of duplicates on page 1 and 2, but also that the last tweet on page 1 is well further into the future, and has a lower ID, than a bunch of tweets on page 2. For example, consider these two, both with the same max_id but page = 1 and page = 2 respectively: http://search.twitter.com/search?rpp=100page=1geocode=-40.900557,17...http://search.twitter.com/search?rpp=100page=2geocode=-40.900557,17... (Or if you prefer json links which are what I am actually using, but I see the same thing on the above ones which are easier to describe:http://search.twitter.com/search.json?q=rpp=100geocode=-40.900557,1... Request:http://search.twitter.com/search.json?q=rpp=100geocode=-40.900557,1...) The first result on page 2 above was posted about 4 hours before the last tweet on page 1. There are also duplicates, eg AshleyGray00: Fireworks! I've been trying to figure this bug out for a while as I'm sure I'm missing something obvious but I'm completely stumped. Does anyone have any clue what is going on here? The only other threads I have found are about people trying to combine since_id and max_id which I know is not allowed, so I can't find anyone else having similar problems.
[twitter-dev] Re: Search API - 403 bursts and (maybe) a caching issue.
I'm experiencing the same. Empty results from the Search API when using the since_id parameter. This is really bad and my users are complaining about the Saved Searches tabs not updating. If you're lucky you end up at a caching server with up-to-date information, but it seems as if you can't force using that caching server. Please let us know if this can be fixed easily. Ole / Gravity Twitter Client for S60/Symbian s...@mobileways.de / @janole on Twitter On 26 Okt., 20:47, briantroy brian.cosin...@gmail.com wrote: Everything below ONLY PERTAINS TO THESEARCHAPI: 1) Since late last week I've noticed a significant number of 403 errors (403 Error from JSON: since_id too recent, poll less frequently). These usually indicate I'm hitting a server with an older view of thesearchindex - since it thinks the ID I sent in since_id is newer than the newest it has. These trouble me because when I get a 200 after the 403 sometimes I get everything back to my since_id, sometimes I don't. I appears some indexes have gaps until they catch up. QUESTION: Are there any ongoingsearchindexing issues that you are aware of? 2) Since late last week I've noticed that somesearchAPI requests appear to get stuck returning an empty json result (no new tweets). This can go on for HOURS (today one got stuck like this for 12 hours). When I restart my process sometimes this clears up (I get the backlog) - other times it does not (I continue to get 0 tweets in the json). All of the requests return HTTP 200 and valid json. QUESTION: Are they any ongoing caching issues with thesearchAPI? These issues are new in the last 7 days (since about last Thursday). My IP is whitelisted. I'm sending both a valid user agent and referrer header. My processes are throttled by the volume of tweets the receive. I've made no changes to my processing since late September. Any assistance would be appreciated. My user's are comparing what they see from my service tosearch.twitter.com and telling me we are broken. Regards, Brian Roy justSignal
