Ed, Efficiency isn't the only goal of user streams, but it is an important one when considering systems at this scale. When optimizing such a system, you generally go looking for the biggest reward first.
We've identified applications that run on the desktop, such as Tweetdeck, Seesmic, and friends, as a larger consumer of resources on a per-user basis. They're always on and always polling at the maximum rate and the traffic comes directly from the desktop. So be it, as this is core to their utility and nature. Browser-based clients are service-based. The aggregate user streams demand from service-based applications, of all types, not just browser clients, far exceeds that of desktops. We cannot possibly launch into service-based demand. It's too vast. To ration access in this environment would be hard for everyone. Also most services do not place quite as large of a per-user load on the REST API. Browsers don't reload, or reload infrequently. Services mediate the load is various ways. Etc. Finally, services are not best served by user streams as they are today. They should have all of their requested traffic delivered over a small number of multiplexed streams. It's not ideal to open 10^5 or 10^6 connections from a given service to Twitter, only to have their aggregate traffic be far less than a Gardenhose feed. The service-side complexity to manage all these connections is large, and the resource utilization on our end is quite high. User streams, with a connection per user, would be an efficiency loss here, where multiplexed streams would be a tremendous efficiency gain. So, services, as a category, are too big to tackle first, they probably offer an efficiency loss, and, with a little more work, services can be supported with a huge efficiency gain for everyone. So we're initially going for the biggest efficiency win with the smallest feature set against the smallest reasonably partitionable market given the limited resources we have available. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sun, May 16, 2010 at 8:04 AM, M. Edward (Ed) Borasky <zn...@borasky-research.net> wrote: > On Sunday, May 16, 2010 12:58:13 am Michael Brunton-Spall wrote: >> I'll just briefly disagree with you about point 2. >> OAuth does not require that you have a browser available. With Twitter >> supporting xAuth your desktop or mobile application can take a persons >> username and password and exchange them for an Access Token for accessing >> protected resources via OAuth without ever firing up the browser. >> >> You can find out more about twitter and xauth at >> http://dev.twitter.com/pages/xauth > > I'm aware of xauth, but I'm likely going to stick with browser-based oAuth and > with a browser-based view component of the application. It's too hard to get a > "portable" "native" desktop UI any other way. People know how to use browsers. > > For example, think about the SugarCRM "Community Edition". If you run that as > a stand-alone desktop application on a Windows, Linux or Mac desktop, what > you've actually got is a whole PHP stack model/view/controller application > with your browser pointing to localhost. Using a browser for the UI and a > model/view/controller stack vastly increases the number of open source tools > I can bring to bear. > >> I think you are missing the point as to why twitter don't want web client >> to use the user streams right now. Much fewer people use desktop >> applications than use mobile or web applications. If you upgrade your >> desktop app to use the user streams then all of your customers have to >> upgrade their app, which will happen slowly over time. If you upgrade a >> web app all of your clients get upgraded all simultaneously. It sounds >> like it's purely a capacity issue at the moment. > > First of all, this is a new application, not an upgrade to an existing one. > Second, since it depends on Twitter and user streams, it's not going to be > deployed until Twitter has the capacity to support it. And third, it *is* > going to be a desktop application, not deployed on servers or mobiles. I > simply don't want to have technologies ruled out because they're "usually" > used on servers, like a web application stack. > > -- > M. Edward (Ed) Borasky > http://borasky-research.net/m-edward-ed-borasky/ @znmeb > > "A mathematician is a device for turning coffee into theorems." ~ Paul Erdős >