[twitter-dev] Re: Pushing Twitter data into the clients
Last I'd checked, the whole Twitter needs to rebuild as a messaging architecture conversation is many months old, and probably well on its way to being implemented. http://blog.twitter.com/2009/01/building-on-open-source.html On Thu, Mar 12, 2009 at 9:19 PM, Joshua Perry j...@6bit.com wrote: I actually hopped on to request a stopgap feature in a similar vein and saw this thread so I thought I'd add kind of a free-associative reply. You know, thinking out-loud. I think the biggest problem with push is that polling is too ingrained into Twitter's architecture to be able to bolt-on push tech without it being more expensive than the current pull methods. Picture the current architecture, back-end MySql, http server, and memory cache. When a user posts a tweet it gets written to cache and persisted to the DB, where is the hook for someone that wants to be pushed that tweet? An interested party needs to be constantly checking the cache/DB for changes in order to see any. In order to bolt push onto such an architecture would require some shim layer to do a sort of tight polling loop to see new updates and then push those updates to the waiting client. Until Twitter's architecture treats the data like what it actually is -- a stream of messages -- at all levels, it won't be able to provide efficiently provide push functionality without the current giant firehose with a filter method Once twitter has designed and implemented their tweet-router(tm) it will be quite easy for clients to register as endpoints and it will be much much cheaper infrastructure-wise than polling currently is. Really, this problem has been solved in software and in hardware by many companies since before Ethernet. Tell me why distributing a tweet is much different than a multicast TCP/IP message? The database is an endpoint connected to the tweet-router and just persists messages and still serves a pull REST api for the initial client cache-charge or web page display. An incoming message's from-user-id could be used as a multicast address and be routed quite efficiently to any connected client endpoint listening to that address. If the firehose is a hub with a giant broadcast domain, then tweet-router is a router behind a switch. Maybe thats silly but its what I've been thinking about. Josh Alex Payne wrote: We're going to be offering exactly that (a long-polling firehose) to select partners to build on. Please see our FAQ for more information. Providing a realtime solution for the rest of the API is a lot more challenging than just hooking up XMPP. If the technology was there, we'd do it. But we've investigated and benchmarked, and it isn't. We also want to provide something with a lower barrier to entry when we're ready to go down that road. We understand that power users are very excited about realtime, but the majority of users and developers are still getting up to speed with Twitter's basic offerings. Accordingly, that's where our energies are at right now. On Thu, Mar 12, 2009 at 16:08, Sam Sethi samkse...@googlemail.com wrote: Please can we get a Twitter xmpp feed. If Twitter are not going to offer this can they allow GNIP to go live. GNIP say Twitter are not allowing them to offer the firehose via xmpp to developers. Why? Pull based polling is last year. Maybe at the least Twitter could offer a Long polling option like friendfeed to give a psuedo realtime feed Thanks in advance Sam W: www.twitblogs.com Sent from my iPhone On 12 Mar 2009, at 21:21, Alex Payne a...@twitter.com wrote: POST requests are unlimited. We used to support XMPP as an experimental feature, but we don't currently. Delivering push features at our scale is a challenge. We're currently making our traditional REST request/response APIs the best they can be. In the future, maybe we'll tackle push as well. In the medium term, select partners will be able to have tweets pushed to them over HTTP via our firehose mechanism. As Andrew suggested, there's been quite a lot of discussion on these topics in this group and elsewhere on the web. On Thu, Mar 12, 2009 at 13:55, Adrian spiritpo...@gmail.com wrote: Hi there, I was wondering if it's possible to push data, rather than have the content pulled with JSON or XML fetches. You can poll after set amounts of time, but that only present the illusion of Push, and uses up bandwith. Also, is the API limit applied to POST requests? Lastly, has Twitter thought about implementing XMPP. -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] explore follower graph via api
is screen scraping the only way to explore the follower graph (multiple descendants) of a users followers? I can find no methods in the APIs that would allow one to create map diagrams... -Preston
[twitter-dev] explore follower graph via api
Is screen scraping the only way to map a tree of a user's followers (including multiple levels of descendants)? I could find no methods in the API that would provide this... -Preston
[twitter-dev] Re: explore follower graph via api
On Fri, Mar 13, 2009 at 4:03 PM, ptone pres...@ptone.com wrote: is screen scraping the only way to explore the follower graph (multiple descendants) of a users followers? I can find no methods in the APIs that would allow one to create map diagrams... -Preston Are you looking for this? http://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods -- simple is good http://brucewang.net http://twitter.com/number5
[twitter-dev] Re: explore follower graph via api
Preston, You have all of the follower data that you need from the API. Any follower data you can get from screen scraping is already exposed by the API. As Bruce indicated, the Social Graph APIs are a great place to start. If you give a more specific description of your needs, we can better guide you with implementation suggestions. Lastly, please don't screen scrape our site. We provide an API to ensure that everyone has efficient access to Twitter's data. Screen scraping is viewed as abuse will promptly be blocked. Thanks, Doug Williams Twitter API Support http://twitter.com/dougw On Fri, Mar 13, 2009 at 9:01 AM, Bruce Wang numb...@gmail.com wrote: On Fri, Mar 13, 2009 at 4:03 PM, ptone pres...@ptone.com wrote: is screen scraping the only way to explore the follower graph (multiple descendants) of a users followers? I can find no methods in the APIs that would allow one to create map diagrams... -Preston Are you looking for this? http://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods -- simple is good http://brucewang.net http://twitter.com/number5
[twitter-dev] Re: How long to keep a cache?
Jason, The answer depends on how your application is affected by a stale and possibly incorrect list. If the accuracy of the list is not critical and the application needs the API calls for other logic, you can cache for longer. If the list needs to be roughly accurate you should cache for a day. This vague answer is attempting to suggest that this is a design choice that is heavily based on the goals of your application. Thanks, Doug Williams Twitter API Support http://twitter.com/dougw On Mar 13, 9:22 am, Jason Levine jasonalev...@gmail.com wrote: I'm working on a Twitter application. As part of my application, to reduce calls to the Twitter API, I'm implementing a cache of the user's Friends list. My question is: How long should I keep this cache before I consider it out of date? 1 Day? 3 Days? 1 Week?
[twitter-dev] Re: Include $ as a searchable character
Just noticed a tweet from @ej from a couple days ago that this is working now. Just tried it out, works great! Thanks a lot for adding this. Now I can watch my portfolio tank with real-time commentary! -Chad On Tue, Feb 24, 2009 at 12:50 PM, Matt Sanford m...@twitter.com wrote: Hi there, This is behind a few other projects and I don't know the ETA for it. Thanks; — Matt On Feb 24, 2009, at 08:39 AM, Karthik wrote: Hi Matt, We are building an App which heavily depends on this new feature. Can you please let me know a tentative date for its release? Thank you. - Karthik On Feb 23, 7:52 pm, Matt Sanford m...@twitter.com wrote: Hi there, I added this info to the ticket. Thanks; — Matt On Feb 23, 2009, at 06:21 AM, Karthik wrote: Can you accept the request for the colon character too? I may use the prefix $IN for Indian stocks and would like to retrieve only the updates containing the term $IN:REL for example? Does that make sense? On Feb 23, 3:39 am, Matt Sanford m...@twitter.com wrote: Hi all, Requests for this have come up before so I opened a ticket for the person working on our tokenizer to add this. I'm not sure when he'll get to it but it's on the road map now. One bit of forewarning: if you try to query for every ticker symbol every five minutes you're building something unscalable and you can expect to hit the rate limit. Not that I'm sure this is what people have in mind but I thought I'd put this out there ahead of time just in case. — Matt On Feb 22, 2009, at 02:17 AM, Karthik Murugan wrote: No, they show status updates from unrelated profiles too. I guess, they are indexing updates containing stock names and filtering out posts that don't have a dollar sign before the stock name On Sun, Feb 22, 2009 at 12:40 PM, Chad Etzel jazzyc...@gmail.com wrote: My SWAG is that they are just parsing their follower/friend stream themselves and highlighting tokens beginning with $. -Chad On Sat, Feb 21, 2009 at 12:40 PM, Karthik fermis...@gmail.com wrote: I'm looking for a similar feature too. I wonder howhttp://stocktwits.com/streams/all could show statuses containing $ On Feb 21, 10:35 pm, Chad Etzel jazzyc...@gmail.com wrote: Yes, that's correct. Add $ as a token modifier, if you will. -Chad On Sat, Feb 21, 2009 at 10:43 AM, Nick Arnett nick.arn...@gmail.com wrote: On Fri, Feb 20, 2009 at 6:46 PM, Chad Etzel jazzyc...@gmail.com wrote: Sorry I must have been unclear. I don't want the $ by itself, I want it to be a searchable character in conjunction with other strings, so I want to search for $AAPL or $C much like # is with hashtags. FYI, in search engine language this means is that you want words to be tokenized with and without the $ (or other) similar characters. Right now, it sounds like $AAPL is tokenized as AAPL. If I understand correctly, you'd want the search engine to also add the token $AAPL. NIck
[twitter-dev] Re: Sequential API XML requests and cache
Yes, API responses will often be returned from cache. As such, there might be brief delays between an activity and the corresponding chain of cache invalidations. On Thu, Mar 12, 2009 at 11:41, mwm miguelwmonte...@gmail.com wrote: Hi, I'm the developer of TwitteReader and I have remade the architecture of the communication between the application's host and Twitter, via API. Example and objective: delete the status 123456 (that I own) and then load the last 20 friend_timeline messages. Before: 1. app calls host with statusID = 123456 2. host calls http://twitter.com/statuses/destroy/123456.xml 3. whether or not it is successful, it returns to the app with the output. 4. app calls host for last 20 messages 5. host calls http://twitter.com/statuses/friends_timeline.xml and might add page and count (max 20) 6. also returns with or without messages to the app with the call output. Now (testing): 1. app calls host with statusID = 123456 and reload request for last 20 messages 2. host calls http://twitter.com/statuses/destroy/123456.xml 3. host calls http://twitter.com/statuses/friends_timeline.xml 4. returns xml modified in the end : original xml + output from destroying Being this for destroying or updating. Thing is, I tested this and also with the version before and have been having inconclusive results of how the API and XML web service of Twitter works, so I can't know where, what I'm about to explain, fails. Using the new version (testing), if I'm at the TwitteReader interface reading and I click to delete a status (that I own), it follows the process I described above and returns a XML with the last 20 friend_timeline status where the one I just asked for deletion, should be gone. Thing is, most of the times, it happens to still be in the requested XML. After some more tests, I repeated the whole test but right after deleting the status message, I went to my twiiter.com/home (logged) with the same account and it wasn't there (the requested deleted status) and on TwittReader, the friends_timeline requested (XML) after deletion request, returns with that status (already deleted on twitter.com) visible. Only some time after, can't be precise but between 30 to 60 seconds (I think), the request of the XML (requested by a reload only), will be updated. I understand that it automatically disappears from the twitter.com/ home page as it accesses the DB directly and TwitteReader accesses it via API. What I want to know is if the API request via XML or other, sends cached versions so I can guarantee that it ain't my app design problem and can fix that problem on the client-side (doesn't displaying the request for deletion status message). As the destroy example, the update process, some times, suffers the same inconsistency when requesting the latest 20 messages after it. Long talk but I hope you understand it and also understand my English coming from a Portuguese. Thank you, mwm. -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Please Add Me to the List
Name: Matthew Arnold Twitter: twitter.com/mattarnold1977 e-mail: matt.arnold.1...@gmail.com blog: http://www.mattsblogsite.com app: http://www.populartweets.com Dev: ASP.NET; javascript; AJAX; SQL
[twitter-dev] Re: Batch Request
I like this a lot. Currently, I don't think we could deliver it with decent performance. In the future, we'll have the capability to do this, I think. In the meantime, you could always write a proxy to do it. That's worked for a couple of mobile Twitter clients. On Thu, Mar 12, 2009 at 22:27, atebits loren.brich...@gmail.com wrote: I'd like to second this request. It'd be more than reasonable for the example above to count as 3 marks against the rate limit, the only difference would be that there's a single round-trip to the twitter servers rather than 3 (would save a bit of overhead on your end too). This makes a ton of sense for mobile apps. Network latency is huge, but once a connection gets going it's best to transfer as much data in one chunk as possible. Not critical, just would be really nice to have :) Loren On Mar 12, 6:33 pm, Joshua Perry j...@6bit.com wrote: Well, I accidentally sent that message so let me finish here... batch_reply friends_timeline status.../status status.../status status.../status status.../status /friends_timeline replies status.../status status.../status /replies direct_messages direct_message.../direct_message direct_message.../direct_message direct_message.../direct_message /direct_messages /batch_reply This would probably mostly benefit client applications as it would allow us to consolidate 3 requests into one, allowing us to poll more often for these pieces of data that people are interested in as being the most real-time. I think this would be a great candidate for fast-track implementation as it could sit on top of whatever internal code serves the existing feeds just adding the wrapper before returning the data. Also, it would be at new URI so it would not be changing existing interfaces for current applications. Josh Joshua Perry wrote: And now something perhaps a little more sane and do-able. It would be very useful to have a batch request API that would allow requesting multiple datasets simultaneously. Something like this: http://twitter.com/batch_request.xml?friend_timeline_since_id=2345re... And this would return the data in the same format as the current REST api for the individual feeds but in a top-level container: batch_reply friends_timeline status.../status status.../status status.../status status.../status /friends_timeline direct_messages direct_message.../direct_message direct_message.../direct_message direct_message.../direct_message /direct_messages /batch_reply -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Archive
I am still encountering these issues. Wondering if I could get any help here. I feel bad bumping, but it's better than spawning another thread. Thanks, Kyle On Mar 10, 10:08 am, Kyle Tolle kyle.to...@gmail.com wrote: I first make a call tohttp://twitter.com/users/show/username.xmlto get the number of tweets. I then set the count to something like 500, and determine the number of pages. With these, I callhttp://twitter.com/statuses/user_timeline/username.xml?page=##count=## (Note: The API says you can have a count up to 3200, but any time I try that, I get an error.) So I believe I'm paging through the statuses/user_timeline corrects. I have the very first and lastest tweets I've made, but somehow there are 6 missing in between. Also, is there a way to callhttp://twitter.com/statuses/user_timeline/username.xml with authentication? I have my twitter account whitelisted with 20k api calls per hour, but my machines are on DHCP so I can't have an IP whitelisted. I got rate limited earlier with the aforementioned calls, so I suspect they aren't being authed. I appreciate the help! On Mar 10, 9:29 am, Doug Williams d...@twitter.com wrote: Kyle, Can you give an example of the calls you are using to get user tweets? It sounds like you need to page through the statuses/user_timeline method [1]. Where are you having problems? [1] -http://apiwiki.twitter.com/REST+API+Documentation#usertimeline Doug Williams Twitter API Supporthttp://twitter.com/dougw On Tue, Mar 10, 2009 at 7:20 AM, Kyle Tolle kyle.to...@gmail.com wrote: I've had trouble getting all the tweets for users while using an app to pull down what should be the entire list of tweets for that user. Is this the sort of restriction you mention? I'm not even really sure how to go about seeing which ones are missing to see if there is a pattern. I'd really like to make an archive/statistical utility, but it will not do well at either if it can't get all the tweets! I see these sorts of threads have been around for years... just wondering how much progress has been made on it. On Mar 9, 6:07 pm, Alex Payne a...@twitter.com wrote: We retain any and all tweets that aren't deleted by users. We have all the tweets ever :) We have some artificial limits in place restricting how far back in time/pages you retrieve tweets. As our data stores increase in reliability for large working sets, we'll be able to lift these restrictions and provide a reasonable quality of service to API clients. On Mon, Mar 9, 2009 at 13:07, shapper mdmo...@gmail.com wrote: Hello, When are tweets archived? It depends of how long they were created or twitter keeps always N tweets visible? Can I access older tweets? And can followers access my older tweets? Thanks, Miguel -- Alex Payne - API Lead, Twitter, Inc.http://twitter.com/al3x
[twitter-dev] Re: Archive
You could always get a static IP and have us whitelist that :) On Fri, Mar 13, 2009 at 11:35, Kyle Tolle kyle.to...@gmail.com wrote: I am still encountering these issues. Wondering if I could get any help here. I feel bad bumping, but it's better than spawning another thread. Thanks, Kyle On Mar 10, 10:08 am, Kyle Tolle kyle.to...@gmail.com wrote: I first make a call tohttp://twitter.com/users/show/username.xmlto get the number of tweets. I then set the count to something like 500, and determine the number of pages. With these, I callhttp://twitter.com/statuses/user_timeline/username.xml?page=##count=## (Note: The API says you can have a count up to 3200, but any time I try that, I get an error.) So I believe I'm paging through the statuses/user_timeline corrects. I have the very first and lastest tweets I've made, but somehow there are 6 missing in between. Also, is there a way to callhttp://twitter.com/statuses/user_timeline/username.xml with authentication? I have my twitter account whitelisted with 20k api calls per hour, but my machines are on DHCP so I can't have an IP whitelisted. I got rate limited earlier with the aforementioned calls, so I suspect they aren't being authed. I appreciate the help! On Mar 10, 9:29 am, Doug Williams d...@twitter.com wrote: Kyle, Can you give an example of the calls you are using to get user tweets? It sounds like you need to page through the statuses/user_timeline method [1]. Where are you having problems? [1] -http://apiwiki.twitter.com/REST+API+Documentation#usertimeline Doug Williams Twitter API Supporthttp://twitter.com/dougw On Tue, Mar 10, 2009 at 7:20 AM, Kyle Tolle kyle.to...@gmail.com wrote: I've had trouble getting all the tweets for users while using an app to pull down what should be the entire list of tweets for that user. Is this the sort of restriction you mention? I'm not even really sure how to go about seeing which ones are missing to see if there is a pattern. I'd really like to make an archive/statistical utility, but it will not do well at either if it can't get all the tweets! I see these sorts of threads have been around for years... just wondering how much progress has been made on it. On Mar 9, 6:07 pm, Alex Payne a...@twitter.com wrote: We retain any and all tweets that aren't deleted by users. We have all the tweets ever :) We have some artificial limits in place restricting how far back in time/pages you retrieve tweets. As our data stores increase in reliability for large working sets, we'll be able to lift these restrictions and provide a reasonable quality of service to API clients. On Mon, Mar 9, 2009 at 13:07, shapper mdmo...@gmail.com wrote: Hello, When are tweets archived? It depends of how long they were created or twitter keeps always N tweets visible? Can I access older tweets? And can followers access my older tweets? Thanks, Miguel -- Alex Payne - API Lead, Twitter, Inc.http://twitter.com/al3x -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Email Headers for Follower Request
Hi, I think that the idea of adding custom headers to your email notifications was genius and sure made my life a lot simpler. My question is - why there aren't such headers for the follower request (for blocked profiles)? Are there plans to add such headers to this messages too in the near future? Thanks, Arik Fraimovich (@arikfr)
[twitter-dev] Re: Email Headers for Follower Request
You can always request additional headers by filing an enhancement request: http://code.google.com/p/twitter-api/issues/entry On Fri, Mar 13, 2009 at 12:59, Arik Fraimovich arik...@gmail.com wrote: Hi, I think that the idea of adding custom headers to your email notifications was genius and sure made my life a lot simpler. My question is - why there aren't such headers for the follower request (for blocked profiles)? Are there plans to add such headers to this messages too in the near future? Thanks, Arik Fraimovich (@arikfr) -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Status link without user name
Say I have status_id 3244325. What is the URL to view that status on the web that doesn't include the username?
[twitter-dev] Re: Status link without user name
On Fri, Mar 13, 2009 at 5:10 PM, Joshua Perry j...@6bit.com wrote: Say I have status_id 3244325. What is the URL to view that status on the web that doesn't include the username? I'm guessing you mean an HTML version... I'm not sure that exists. Otherwise, you can use http://twitter.com/statuses/show/3244325.xmlhttp://twitter.com/statuses/show/1324853977.xml or http://twitter.com/statuses/show/3244325.http://twitter.com/statuses/show/1324853977.xml json Nick
[twitter-dev] Re: Status link without user name
Nick, If you only have a status id and you would like to construct a URL, you are going to have to use a call to statuses/show [1] to get the screen name for the user then construct the URL accordingly: http://twitter.com/screen_name/status/status_id [1] - http://apiwiki.twitter.com/REST+API+Documentation#show Doug Williams Twitter API Support http://twitter.com/dougw On Fri, Mar 13, 2009 at 8:24 PM, Nick Arnett nick.arn...@gmail.com wrote: On Fri, Mar 13, 2009 at 5:10 PM, Joshua Perry j...@6bit.com wrote: Say I have status_id 3244325. What is the URL to view that status on the web that doesn't include the username? I'm guessing you mean an HTML version... I'm not sure that exists. Otherwise, you can use http://twitter.com/statuses/show/3244325.xmlhttp://twitter.com/statuses/show/1324853977.xml or http://twitter.com/statuses/show/3244325.http://twitter.com/statuses/show/1324853977.xml json Nick
[twitter-dev] Re: Status link without user name
On Fri, Mar 13, 2009 at 5:32 PM, Doug Williams d...@twitter.com wrote: Nick, If you only have a status id and you would like to construct a URL, you are going to have to use a call to statuses/show [1] to get the screen name for the user then construct the URL accordingly: I think the original poster, whose question I was (flailing around) trying to answer, was probably hoping for an HTML page. Oh, I see what you're saying... use the API call to construct the link with the user name. Probably not the answer he wants, but we'll see. There is no link that returns HTML that doesn't have the screen name in it, I'm assuming. Nick
[twitter-dev] Re: Status link without user name
Thanks for the input guys, we do have the screen name also, I just assumed since the status_id was unique that there was an HTML URL that would go directly to it. Nick Arnett wrote: On Fri, Mar 13, 2009 at 5:32 PM, Doug Williams d...@twitter.com mailto:d...@twitter.com wrote: Nick, If you only have a status id and you would like to construct a URL, you are going to have to use a call to statuses/show [1] to get the screen name for the user then construct the URL accordingly: I think the original poster, whose question I was (flailing around) trying to answer, was probably hoping for an HTML page. Oh, I see what you're saying... use the API call to construct the link with the user name. Probably not the answer he wants, but we'll see. There is no link that returns HTML that doesn't have the screen name in it, I'm assuming. Nick
