[twitter-dev] Re: Pushing Twitter data into the clients
Awesome. Right now, the onus is on the Tweet receiver to sort their friends into groups, and pick and choose those groups they wish read and check at any one time, like with Friendfeed. Would be great to have Sending groups too, where the sender of Tweet chooses who shall receive it. On Mar 15, 11:13 am, ledil wrote: > Hello, > > if you wait another two months, you can use XMPP within blummi.com. > Very are building a > location-based micro-blogging service based on XMPP with realtime > delivering of location > based information. > > greets, > > On Mar 14, 4:26 pm, Andrew Badera wrote: > > > Has Jaiku gone XMPP yet? We're having a discussion about jaiku on the > > GAE list now. I believe XMPP is almost-but-not-quite-ready? They're > > involved in a pubsubhub project on GAE to make it happen, if I paid > > enough attention to that convo. > > > On Sat, Mar 14, 2009 at 5:59 AM, Adrian wrote: > > > > Well, as much as I love Twitter, it's not on Google's infrastructure. > > > Speaking of which, Jaiku has gone open source and supports XMPP: > > > >http://code.google.com/p/jaikuengine/ > > >http://code.google.com/p/jaikuengine/source/browse/trunk/doc/design_f... > > > > Twitter Team: maybe there's something to learn here. > > > > On Mar 13, 6:07 pm, Joshua Perry wrote: > > >> Hey, thats an interesting read, as is Robey's commentary on the Kestrel > > >> message queuing system, but it brings up more questions for me. > > > >> Using a message queuing system is great for decoupling producers from > > >> consumers, but frankly I'm surprised that they're using a polling queue > > >> as recently as January -- however long the polling may be (250msec). > > >> When is the last time your network card polled the switch it is > > >> connected for the next packet in its queue? When was the last time your > > >> application polled the TCP stack for the next byte available in its > > >> buffer? Computers at their lowest level have been event driven since > > >> the early 60's. Any architecture with hopes of scaling should not be > > >> based on polling. Removing polling is usually one of the first > > >> optimizations in a system that finds itself needing to grow beyond its > > >> current capabilities. > > > >> Also, queuing systems don't lend themselves terribly well to the concept > > >> of a pub-sub model. In a multicast model the a message is placed on a > > >> queue and any interested parties can listen to that queue and work on > > >> messages from it; this is not much better than the firehose method, as > > >> the data is actually duplicated and blasted to the endpoints at the > > >> network layer. So while a multicast model is somewhat well-suited to > > >> the Twitter backbone I don't see it being useful outside of that > > >> backbone. In a unicast model a queue is not much more than a decoupling > > >> abstraction between the producer and the consumer. > > > >> I don't know what size their internal message data structure looks like, > > >> but assuming a message of around 1KB -- should be easy to get to this > > >> size passing around unicode -- their queue would exhaust its128MB of > > >> memory cache after only 128000 messages. I don't know if their servers > > >> are memory constrained, they have a large number of queues, this comes > > >> from throughput testing, or they are just optimizing early; with a > > >> "small" server these days able to be configured with upwards of 24GB of > > >> RAM, this number seems low. > > > >> Its easy to stand outside of their design -- and what I'm sure have been > > >> hundreds of hours of discussions and meetings -- and criticize their > > >> system but I hope that there is some constructive criticism in this > > >> discussion. With the reliance that Twitter puts on open-source software > > >> I believe they also value open discussion. > > > >> Josh > > > >> Andrew Badera wrote: > > >> > Last I'd checked, the whole "Twitter needs to rebuild as a messaging > > >> > architecture" conversation is many months old, and probably well on > > >> > its way to being implemented. > > > >> >http://blog.twitter.com/2009/01/building-on-open-source.html > > > >> > On Thu, Mar 12, 2009 at 9:19 PM, Joshua Perry wrote: > > > >> >> I actually hopped on to request a stopgap feature in a similar vein > > >> >> and saw > > >> >> this thread so I thought I'd add kind of a free-associative reply. > > >> >> You > > >> >> know, thinking out-loud. > > > >> >> I think the biggest problem with push is that polling is too > > >> >> ingrained into > > >> >> Twitter's architecture to be able to bolt-on push tech without it > > >> >> being more > > >> >> expensive than the current pull methods. > > > >> >> Picture the current architecture, back-end MySql, http server, and > > >> >> memory > > >> >> cache. When a user posts a tweet it gets written to cache and > > >> >> persisted to > > >> >> the DB, where is the hook for someone that wants to be pushed that > > >> >> tweet? > > >> >>
[twitter-dev] Re: Pushing Twitter data into the clients
Praise the lord, of course; but you can't say YouTube has not sped up upon integration into Google's tech stack? (Big Table, Google FS) http://www.techcrunchit.com/2008/07/16/google-where-companies-go-to-die/ I do think my comment may not have been wholly appropriate given Twitter's much, much better now. On Mar 14, 5:06 pm, Cameron Kaiser wrote: > > Well, as much as I love Twitter, it's not on Google's infrastructure. > > *cough*praisethelord*cough* > > -- > personal:http://www.cameronkaiser.com/-- > Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com > -- Please dispose of this message in the usual manner. -- Mission: Impossible > -
[twitter-dev] Re: Pushing Twitter data into the clients
Hello, if you wait another two months, you can use XMPP within blummi.com. Very are building a location-based micro-blogging service based on XMPP with realtime delivering of location based information. greets, On Mar 14, 4:26 pm, Andrew Badera wrote: > Has Jaiku gone XMPP yet? We're having a discussion about jaiku on the > GAE list now. I believe XMPP is almost-but-not-quite-ready? They're > involved in a pubsubhub project on GAE to make it happen, if I paid > enough attention to that convo. > > On Sat, Mar 14, 2009 at 5:59 AM, Adrian wrote: > > > Well, as much as I love Twitter, it's not on Google's infrastructure. > > Speaking of which, Jaiku has gone open source and supports XMPP: > > >http://code.google.com/p/jaikuengine/ > >http://code.google.com/p/jaikuengine/source/browse/trunk/doc/design_f... > > > Twitter Team: maybe there's something to learn here. > > > On Mar 13, 6:07 pm, Joshua Perry wrote: > >> Hey, thats an interesting read, as is Robey's commentary on the Kestrel > >> message queuing system, but it brings up more questions for me. > > >> Using a message queuing system is great for decoupling producers from > >> consumers, but frankly I'm surprised that they're using a polling queue > >> as recently as January -- however long the polling may be (250msec). > >> When is the last time your network card polled the switch it is > >> connected for the next packet in its queue? When was the last time your > >> application polled the TCP stack for the next byte available in its > >> buffer? Computers at their lowest level have been event driven since > >> the early 60's. Any architecture with hopes of scaling should not be > >> based on polling. Removing polling is usually one of the first > >> optimizations in a system that finds itself needing to grow beyond its > >> current capabilities. > > >> Also, queuing systems don't lend themselves terribly well to the concept > >> of a pub-sub model. In a multicast model the a message is placed on a > >> queue and any interested parties can listen to that queue and work on > >> messages from it; this is not much better than the firehose method, as > >> the data is actually duplicated and blasted to the endpoints at the > >> network layer. So while a multicast model is somewhat well-suited to > >> the Twitter backbone I don't see it being useful outside of that > >> backbone. In a unicast model a queue is not much more than a decoupling > >> abstraction between the producer and the consumer. > > >> I don't know what size their internal message data structure looks like, > >> but assuming a message of around 1KB -- should be easy to get to this > >> size passing around unicode -- their queue would exhaust its128MB of > >> memory cache after only 128000 messages. I don't know if their servers > >> are memory constrained, they have a large number of queues, this comes > >> from throughput testing, or they are just optimizing early; with a > >> "small" server these days able to be configured with upwards of 24GB of > >> RAM, this number seems low. > > >> Its easy to stand outside of their design -- and what I'm sure have been > >> hundreds of hours of discussions and meetings -- and criticize their > >> system but I hope that there is some constructive criticism in this > >> discussion. With the reliance that Twitter puts on open-source software > >> I believe they also value open discussion. > > >> Josh > > >> Andrew Badera wrote: > >> > Last I'd checked, the whole "Twitter needs to rebuild as a messaging > >> > architecture" conversation is many months old, and probably well on > >> > its way to being implemented. > > >> >http://blog.twitter.com/2009/01/building-on-open-source.html > > >> > On Thu, Mar 12, 2009 at 9:19 PM, Joshua Perry wrote: > > >> >> I actually hopped on to request a stopgap feature in a similar vein and > >> >> saw > >> >> this thread so I thought I'd add kind of a free-associative reply. You > >> >> know, thinking out-loud. > > >> >> I think the biggest problem with push is that polling is too ingrained > >> >> into > >> >> Twitter's architecture to be able to bolt-on push tech without it being > >> >> more > >> >> expensive than the current pull methods. > > >> >> Picture the current architecture, back-end MySql, http server, and > >> >> memory > >> >> cache. When a user posts a tweet it gets written to cache and > >> >> persisted to > >> >> the DB, where is the hook for someone that wants to be pushed that > >> >> tweet? > >> >> An interested party needs to be constantly checking the cache/DB for > >> >> changes > >> >> in order to see any. In order to bolt push onto such an architecture > >> >> would > >> >> require some shim layer to do a sort of tight polling loop to see new > >> >> updates and then push those updates to the waiting client. > > >> >> Until Twitter's architecture treats the data like what it actually is > >> >> -- a > >> >> stream of messages -- at all levels, it won't be able to provide
[twitter-dev] Re: Pushing Twitter data into the clients
Has Jaiku gone XMPP yet? We're having a discussion about jaiku on the GAE list now. I believe XMPP is almost-but-not-quite-ready? They're involved in a pubsubhub project on GAE to make it happen, if I paid enough attention to that convo. On Sat, Mar 14, 2009 at 5:59 AM, Adrian wrote: > > Well, as much as I love Twitter, it's not on Google's infrastructure. > Speaking of which, Jaiku has gone open source and supports XMPP: > > http://code.google.com/p/jaikuengine/ > http://code.google.com/p/jaikuengine/source/browse/trunk/doc/design_funument.txt > > Twitter Team: maybe there's something to learn here. > > > On Mar 13, 6:07 pm, Joshua Perry wrote: >> Hey, thats an interesting read, as is Robey's commentary on the Kestrel >> message queuing system, but it brings up more questions for me. >> >> Using a message queuing system is great for decoupling producers from >> consumers, but frankly I'm surprised that they're using a polling queue >> as recently as January -- however long the polling may be (250msec). >> When is the last time your network card polled the switch it is >> connected for the next packet in its queue? When was the last time your >> application polled the TCP stack for the next byte available in its >> buffer? Computers at their lowest level have been event driven since >> the early 60's. Any architecture with hopes of scaling should not be >> based on polling. Removing polling is usually one of the first >> optimizations in a system that finds itself needing to grow beyond its >> current capabilities. >> >> Also, queuing systems don't lend themselves terribly well to the concept >> of a pub-sub model. In a multicast model the a message is placed on a >> queue and any interested parties can listen to that queue and work on >> messages from it; this is not much better than the firehose method, as >> the data is actually duplicated and blasted to the endpoints at the >> network layer. So while a multicast model is somewhat well-suited to >> the Twitter backbone I don't see it being useful outside of that >> backbone. In a unicast model a queue is not much more than a decoupling >> abstraction between the producer and the consumer. >> >> I don't know what size their internal message data structure looks like, >> but assuming a message of around 1KB -- should be easy to get to this >> size passing around unicode -- their queue would exhaust its128MB of >> memory cache after only 128000 messages. I don't know if their servers >> are memory constrained, they have a large number of queues, this comes >> from throughput testing, or they are just optimizing early; with a >> "small" server these days able to be configured with upwards of 24GB of >> RAM, this number seems low. >> >> Its easy to stand outside of their design -- and what I'm sure have been >> hundreds of hours of discussions and meetings -- and criticize their >> system but I hope that there is some constructive criticism in this >> discussion. With the reliance that Twitter puts on open-source software >> I believe they also value open discussion. >> >> Josh >> >> Andrew Badera wrote: >> > Last I'd checked, the whole "Twitter needs to rebuild as a messaging >> > architecture" conversation is many months old, and probably well on >> > its way to being implemented. >> >> >http://blog.twitter.com/2009/01/building-on-open-source.html >> >> > On Thu, Mar 12, 2009 at 9:19 PM, Joshua Perry wrote: >> >> >> I actually hopped on to request a stopgap feature in a similar vein and >> >> saw >> >> this thread so I thought I'd add kind of a free-associative reply. You >> >> know, thinking out-loud. >> >> >> I think the biggest problem with push is that polling is too ingrained >> >> into >> >> Twitter's architecture to be able to bolt-on push tech without it being >> >> more >> >> expensive than the current pull methods. >> >> >> Picture the current architecture, back-end MySql, http server, and memory >> >> cache. When a user posts a tweet it gets written to cache and persisted >> >> to >> >> the DB, where is the hook for someone that wants to be pushed that tweet? >> >> An interested party needs to be constantly checking the cache/DB for >> >> changes >> >> in order to see any. In order to bolt push onto such an architecture >> >> would >> >> require some shim layer to do a sort of tight polling loop to see new >> >> updates and then push those updates to the waiting client. >> >> >> Until Twitter's architecture treats the data like what it actually is -- a >> >> stream of messages -- at all levels, it won't be able to provide >> >> efficiently >> >> provide push functionality without the current giant firehose with a >> >> filter >> >> method >> >> >> Once twitter has designed and implemented their tweet-router(tm) it will >> >> be >> >> quite easy for clients to register as endpoints and it will be much much >> >> cheaper infrastructure-wise than polling currently is. Really, this >> >> problem >> >> has been solved in
[twitter-dev] Re: Pushing Twitter data into the clients
> Well, as much as I love Twitter, it's not on Google's infrastructure. *cough*praisethelord*cough* -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- Please dispose of this message in the usual manner. -- Mission: Impossible -
[twitter-dev] Re: Pushing Twitter data into the clients
Well, as much as I love Twitter, it's not on Google's infrastructure. Speaking of which, Jaiku has gone open source and supports XMPP: http://code.google.com/p/jaikuengine/ http://code.google.com/p/jaikuengine/source/browse/trunk/doc/design_funument.txt Twitter Team: maybe there's something to learn here. On Mar 13, 6:07 pm, Joshua Perry wrote: > Hey, thats an interesting read, as is Robey's commentary on the Kestrel > message queuing system, but it brings up more questions for me. > > Using a message queuing system is great for decoupling producers from > consumers, but frankly I'm surprised that they're using a polling queue > as recently as January -- however long the polling may be (250msec). > When is the last time your network card polled the switch it is > connected for the next packet in its queue? When was the last time your > application polled the TCP stack for the next byte available in its > buffer? Computers at their lowest level have been event driven since > the early 60's. Any architecture with hopes of scaling should not be > based on polling. Removing polling is usually one of the first > optimizations in a system that finds itself needing to grow beyond its > current capabilities. > > Also, queuing systems don't lend themselves terribly well to the concept > of a pub-sub model. In a multicast model the a message is placed on a > queue and any interested parties can listen to that queue and work on > messages from it; this is not much better than the firehose method, as > the data is actually duplicated and blasted to the endpoints at the > network layer. So while a multicast model is somewhat well-suited to > the Twitter backbone I don't see it being useful outside of that > backbone. In a unicast model a queue is not much more than a decoupling > abstraction between the producer and the consumer. > > I don't know what size their internal message data structure looks like, > but assuming a message of around 1KB -- should be easy to get to this > size passing around unicode -- their queue would exhaust its128MB of > memory cache after only 128000 messages. I don't know if their servers > are memory constrained, they have a large number of queues, this comes > from throughput testing, or they are just optimizing early; with a > "small" server these days able to be configured with upwards of 24GB of > RAM, this number seems low. > > Its easy to stand outside of their design -- and what I'm sure have been > hundreds of hours of discussions and meetings -- and criticize their > system but I hope that there is some constructive criticism in this > discussion. With the reliance that Twitter puts on open-source software > I believe they also value open discussion. > > Josh > > Andrew Badera wrote: > > Last I'd checked, the whole "Twitter needs to rebuild as a messaging > > architecture" conversation is many months old, and probably well on > > its way to being implemented. > > >http://blog.twitter.com/2009/01/building-on-open-source.html > > > On Thu, Mar 12, 2009 at 9:19 PM, Joshua Perry wrote: > > >> I actually hopped on to request a stopgap feature in a similar vein and saw > >> this thread so I thought I'd add kind of a free-associative reply. You > >> know, thinking out-loud. > > >> I think the biggest problem with push is that polling is too ingrained into > >> Twitter's architecture to be able to bolt-on push tech without it being > >> more > >> expensive than the current pull methods. > > >> Picture the current architecture, back-end MySql, http server, and memory > >> cache. When a user posts a tweet it gets written to cache and persisted to > >> the DB, where is the hook for someone that wants to be pushed that tweet? > >> An interested party needs to be constantly checking the cache/DB for > >> changes > >> in order to see any. In order to bolt push onto such an architecture would > >> require some shim layer to do a sort of tight polling loop to see new > >> updates and then push those updates to the waiting client. > > >> Until Twitter's architecture treats the data like what it actually is -- a > >> stream of messages -- at all levels, it won't be able to provide > >> efficiently > >> provide push functionality without the current giant firehose with a filter > >> method > > >> Once twitter has designed and implemented their tweet-router(tm) it will be > >> quite easy for clients to register as endpoints and it will be much much > >> cheaper infrastructure-wise than polling currently is. Really, this > >> problem > >> has been solved in software and in hardware by many companies since before > >> Ethernet. Tell me why distributing a tweet is much different than a > >> multicast TCP/IP message? The database is an endpoint connected to the > >> tweet-router and just persists messages and still serves a pull REST api > >> for > >> the initial client cache-charge or web page display. An incoming message's > >> from-user-id could be used as a multicast add
[twitter-dev] Re: Pushing Twitter data into the clients
Hey, thats an interesting read, as is Robey's commentary on the Kestrel message queuing system, but it brings up more questions for me. Using a message queuing system is great for decoupling producers from consumers, but frankly I'm surprised that they're using a polling queue as recently as January -- however long the polling may be (250msec). When is the last time your network card polled the switch it is connected for the next packet in its queue? When was the last time your application polled the TCP stack for the next byte available in its buffer? Computers at their lowest level have been event driven since the early 60's. Any architecture with hopes of scaling should not be based on polling. Removing polling is usually one of the first optimizations in a system that finds itself needing to grow beyond its current capabilities. Also, queuing systems don't lend themselves terribly well to the concept of a pub-sub model. In a multicast model the a message is placed on a queue and any interested parties can listen to that queue and work on messages from it; this is not much better than the firehose method, as the data is actually duplicated and blasted to the endpoints at the network layer. So while a multicast model is somewhat well-suited to the Twitter backbone I don't see it being useful outside of that backbone. In a unicast model a queue is not much more than a decoupling abstraction between the producer and the consumer. I don't know what size their internal message data structure looks like, but assuming a message of around 1KB -- should be easy to get to this size passing around unicode -- their queue would exhaust its128MB of memory cache after only 128000 messages. I don't know if their servers are memory constrained, they have a large number of queues, this comes from throughput testing, or they are just optimizing early; with a "small" server these days able to be configured with upwards of 24GB of RAM, this number seems low. Its easy to stand outside of their design -- and what I'm sure have been hundreds of hours of discussions and meetings -- and criticize their system but I hope that there is some constructive criticism in this discussion. With the reliance that Twitter puts on open-source software I believe they also value open discussion. Josh Andrew Badera wrote: Last I'd checked, the whole "Twitter needs to rebuild as a messaging architecture" conversation is many months old, and probably well on its way to being implemented. http://blog.twitter.com/2009/01/building-on-open-source.html On Thu, Mar 12, 2009 at 9:19 PM, Joshua Perry wrote: I actually hopped on to request a stopgap feature in a similar vein and saw this thread so I thought I'd add kind of a free-associative reply. You know, thinking out-loud. I think the biggest problem with push is that polling is too ingrained into Twitter's architecture to be able to bolt-on push tech without it being more expensive than the current pull methods. Picture the current architecture, back-end MySql, http server, and memory cache. When a user posts a tweet it gets written to cache and persisted to the DB, where is the hook for someone that wants to be pushed that tweet? An interested party needs to be constantly checking the cache/DB for changes in order to see any. In order to bolt push onto such an architecture would require some shim layer to do a sort of tight polling loop to see new updates and then push those updates to the waiting client. Until Twitter's architecture treats the data like what it actually is -- a stream of messages -- at all levels, it won't be able to provide efficiently provide push functionality without the current giant firehose with a filter method Once twitter has designed and implemented their tweet-router(tm) it will be quite easy for clients to register as endpoints and it will be much much cheaper infrastructure-wise than polling currently is. Really, this problem has been solved in software and in hardware by many companies since before Ethernet. Tell me why distributing a tweet is much different than a multicast TCP/IP message? The database is an endpoint connected to the tweet-router and just persists messages and still serves a pull REST api for the initial client cache-charge or web page display. An incoming message's from-user-id could be used as a multicast address and be routed quite efficiently to any connected client endpoint "listening" to that address. If the firehose is a hub with a giant broadcast domain, then tweet-router is a router behind a switch. Maybe thats silly but its what I've been thinking about. Josh Alex Payne wrote: We're going to be offering exactly that (a long-polling firehose) to select partners to build on. Please see our FAQ for more information. Providing a realtime solution for the rest of the API is a lot more challenging than just hooking up XMPP. If the technology was there, we'd do it. Bu
[twitter-dev] Re: Pushing Twitter data into the clients
Last I'd checked, the whole "Twitter needs to rebuild as a messaging architecture" conversation is many months old, and probably well on its way to being implemented. http://blog.twitter.com/2009/01/building-on-open-source.html On Thu, Mar 12, 2009 at 9:19 PM, Joshua Perry wrote: > I actually hopped on to request a stopgap feature in a similar vein and saw > this thread so I thought I'd add kind of a free-associative reply. You > know, thinking out-loud. > > I think the biggest problem with push is that polling is too ingrained into > Twitter's architecture to be able to bolt-on push tech without it being more > expensive than the current pull methods. > > Picture the current architecture, back-end MySql, http server, and memory > cache. When a user posts a tweet it gets written to cache and persisted to > the DB, where is the hook for someone that wants to be pushed that tweet? > An interested party needs to be constantly checking the cache/DB for changes > in order to see any. In order to bolt push onto such an architecture would > require some shim layer to do a sort of tight polling loop to see new > updates and then push those updates to the waiting client. > > Until Twitter's architecture treats the data like what it actually is -- a > stream of messages -- at all levels, it won't be able to provide efficiently > provide push functionality without the current giant firehose with a filter > method > > Once twitter has designed and implemented their tweet-router(tm) it will be > quite easy for clients to register as endpoints and it will be much much > cheaper infrastructure-wise than polling currently is. Really, this problem > has been solved in software and in hardware by many companies since before > Ethernet. Tell me why distributing a tweet is much different than a > multicast TCP/IP message? The database is an endpoint connected to the > tweet-router and just persists messages and still serves a pull REST api for > the initial client cache-charge or web page display. An incoming message's > from-user-id could be used as a multicast address and be routed quite > efficiently to any connected client endpoint "listening" to that address. > > If the firehose is a hub with a giant broadcast domain, then tweet-router is > a router behind a switch. > > Maybe thats silly but its what I've been thinking about. > > Josh > > Alex Payne wrote: > > We're going to be offering exactly that (a long-polling firehose) to > select partners to build on. Please see our FAQ for more information. > > Providing a realtime solution for the rest of the API is a lot more > challenging than just hooking up XMPP. If the technology was there, > we'd do it. But we've investigated and benchmarked, and it isn't. We > also want to provide something with a lower barrier to entry when > we're ready to go down that road. > > We understand that power users are very excited about "realtime", but > the majority of users and developers are still getting up to speed > with Twitter's basic offerings. Accordingly, that's where our energies > are at right now. > > On Thu, Mar 12, 2009 at 16:08, Sam Sethi wrote: > > > Please can we get a Twitter xmpp feed. If Twitter are not going to offer > this can they allow GNIP to go live. GNIP say Twitter are not allowing them > to offer the firehose via xmpp to developers. Why? Pull based polling is > last year. Maybe at the least Twitter could offer a Long polling option like > friendfeed to give a psuedo realtime feed > > Thanks in advance > > Sam > > W: www.twitblogs.com > > Sent from my iPhone > > On 12 Mar 2009, at 21:21, Alex Payne wrote: > > > > POST requests are unlimited. > > We used to support XMPP as an experimental feature, but we don't > currently. > > Delivering push features at our scale is a challenge. We're currently > making our traditional REST request/response APIs the best they can > be. In the future, maybe we'll tackle push as well. In the medium > term, select partners will be able to have tweets pushed to them over > HTTP via our "firehose" mechanism. > > As Andrew suggested, there's been quite a lot of discussion on these > topics in this group and elsewhere on the web. > > On Thu, Mar 12, 2009 at 13:55, Adrian wrote: > > > Hi there, I was wondering if it's possible to push data, rather than > have the content pulled with JSON or XML fetches. You can poll after > set amounts of time, but that only present the illusion of Push, and > uses up bandwith. > > Also, is the API limit applied to POST requests? > > Lastly, has Twitter thought about implementing XMPP. > > > > -- > Alex Payne - API Lead, Twitter, Inc. > http://twitter.com/al3x > > > >
[twitter-dev] Re: Pushing Twitter data into the clients
I actually hopped on to request a stopgap feature in a similar vein and saw this thread so I thought I'd add kind of a free-associative reply. You know, thinking out-loud. I think the biggest problem with push is that polling is too ingrained into Twitter's architecture to be able to bolt-on push tech without it being more expensive than the current pull methods. Picture the current architecture, back-end MySql, http server, and memory cache. When a user posts a tweet it gets written to cache and persisted to the DB, where is the hook for someone that wants to be pushed that tweet? An interested party needs to be constantly checking the cache/DB for changes in order to see any. In order to bolt push onto such an architecture would require some shim layer to do a sort of tight polling loop to see new updates and then push those updates to the waiting client. Until Twitter's architecture treats the data like what it actually is -- a stream of messages -- at all levels, it won't be able to provide efficiently provide push functionality without the current giant firehose with a filter method Once twitter has designed and implemented their tweet-router(tm) it will be quite easy for clients to register as endpoints and it will be much much cheaper infrastructure-wise than polling currently is. Really, this problem has been solved in software and in hardware by many companies since before Ethernet. Tell me why distributing a tweet is much different than a multicast TCP/IP message? The database is an endpoint connected to the tweet-router and just persists messages and still serves a pull REST api for the initial client cache-charge or web page display. An incoming message's from-user-id could be used as a multicast address and be routed quite efficiently to any connected client endpoint "listening" to that address. If the firehose is a hub with a giant broadcast domain, then tweet-router is a router behind a switch. Maybe thats silly but its what I've been thinking about. Josh Alex Payne wrote: We're going to be offering exactly that (a long-polling firehose) to select partners to build on. Please see our FAQ for more information. Providing a realtime solution for the rest of the API is a lot more challenging than just hooking up XMPP. If the technology was there, we'd do it. But we've investigated and benchmarked, and it isn't. We also want to provide something with a lower barrier to entry when we're ready to go down that road. We understand that power users are very excited about "realtime", but the majority of users and developers are still getting up to speed with Twitter's basic offerings. Accordingly, that's where our energies are at right now. On Thu, Mar 12, 2009 at 16:08, Sam Sethi wrote: Please can we get a Twitter xmpp feed. If Twitter are not going to offer this can they allow GNIP to go live. GNIP say Twitter are not allowing them to offer the firehose via xmpp to developers. Why? Pull based polling is last year. Maybe at the least Twitter could offer a Long polling option like friendfeed to give a psuedo realtime feed Thanks in advance Sam W: www.twitblogs.com Sent from my iPhone On 12 Mar 2009, at 21:21, Alex Payne wrote: POST requests are unlimited. We used to support XMPP as an experimental feature, but we don't currently. Delivering push features at our scale is a challenge. We're currently making our traditional REST request/response APIs the best they can be. In the future, maybe we'll tackle push as well. In the medium term, select partners will be able to have tweets pushed to them over HTTP via our "firehose" mechanism. As Andrew suggested, there's been quite a lot of discussion on these topics in this group and elsewhere on the web. On Thu, Mar 12, 2009 at 13:55, Adrian wrote: Hi there, I was wondering if it's possible to push data, rather than have the content pulled with JSON or XML fetches. You can poll after set amounts of time, but that only present the illusion of Push, and uses up bandwith. Also, is the API limit applied to POST requests? Lastly, has Twitter thought about implementing XMPP. -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Pushing Twitter data into the clients
We're going to be offering exactly that (a long-polling firehose) to select partners to build on. Please see our FAQ for more information. Providing a realtime solution for the rest of the API is a lot more challenging than just hooking up XMPP. If the technology was there, we'd do it. But we've investigated and benchmarked, and it isn't. We also want to provide something with a lower barrier to entry when we're ready to go down that road. We understand that power users are very excited about "realtime", but the majority of users and developers are still getting up to speed with Twitter's basic offerings. Accordingly, that's where our energies are at right now. On Thu, Mar 12, 2009 at 16:08, Sam Sethi wrote: > > Please can we get a Twitter xmpp feed. If Twitter are not going to offer > this can they allow GNIP to go live. GNIP say Twitter are not allowing them > to offer the firehose via xmpp to developers. Why? Pull based polling is > last year. Maybe at the least Twitter could offer a Long polling option like > friendfeed to give a psuedo realtime feed > > Thanks in advance > > Sam > > W: www.twitblogs.com > > Sent from my iPhone > > On 12 Mar 2009, at 21:21, Alex Payne wrote: > >> >> POST requests are unlimited. >> >> We used to support XMPP as an experimental feature, but we don't >> currently. >> >> Delivering push features at our scale is a challenge. We're currently >> making our traditional REST request/response APIs the best they can >> be. In the future, maybe we'll tackle push as well. In the medium >> term, select partners will be able to have tweets pushed to them over >> HTTP via our "firehose" mechanism. >> >> As Andrew suggested, there's been quite a lot of discussion on these >> topics in this group and elsewhere on the web. >> >> On Thu, Mar 12, 2009 at 13:55, Adrian wrote: >>> >>> Hi there, I was wondering if it's possible to push data, rather than >>> have the content pulled with JSON or XML fetches. You can poll after >>> set amounts of time, but that only present the illusion of Push, and >>> uses up bandwith. >>> >>> Also, is the API limit applied to POST requests? >>> >>> Lastly, has Twitter thought about implementing XMPP. >>> >> >> >> >> -- >> Alex Payne - API Lead, Twitter, Inc. >> http://twitter.com/al3x > -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Pushing Twitter data into the clients
Please can we get a Twitter xmpp feed. If Twitter are not going to offer this can they allow GNIP to go live. GNIP say Twitter are not allowing them to offer the firehose via xmpp to developers. Why? Pull based polling is last year. Maybe at the least Twitter could offer a Long polling option like friendfeed to give a psuedo realtime feed Thanks in advance Sam W: www.twitblogs.com Sent from my iPhone On 12 Mar 2009, at 21:21, Alex Payne wrote: POST requests are unlimited. We used to support XMPP as an experimental feature, but we don't currently. Delivering push features at our scale is a challenge. We're currently making our traditional REST request/response APIs the best they can be. In the future, maybe we'll tackle push as well. In the medium term, select partners will be able to have tweets pushed to them over HTTP via our "firehose" mechanism. As Andrew suggested, there's been quite a lot of discussion on these topics in this group and elsewhere on the web. On Thu, Mar 12, 2009 at 13:55, Adrian wrote: Hi there, I was wondering if it's possible to push data, rather than have the content pulled with JSON or XML fetches. You can poll after set amounts of time, but that only present the illusion of Push, and uses up bandwith. Also, is the API limit applied to POST requests? Lastly, has Twitter thought about implementing XMPP. -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Pushing Twitter data into the clients
Well, here are some good links behind XMPP: http://xmpp.org/ http://metajack.im/ (key XMPP advocate and founder of http://www.chesspark.com/ ) On Mar 12, 11:05 pm, Andrew Badera wrote: > What is this "XMPP" of which you speak? Tell us more! > > PS: Read the docs? FAQs? This list's archive? Used Google? Ever? > > On Thu, Mar 12, 2009 at 3:55 PM, Adrian wrote: > > > Hi there, I was wondering if it's possible to push data, rather than > > have the content pulled with JSON or XML fetches. You can poll after > > set amounts of time, but that only present the illusion of Push, and > > uses up bandwith. > > > Also, is the API limit applied to POST requests? > > > Lastly, has Twitter thought about implementing XMPP.
[twitter-dev] Re: Pushing Twitter data into the clients
Great, well that's a start, and obviously the goal is to have IM like performance in Twitter. Maybe there's an engineer out there that can help with the problem. On Mar 12, 11:21 pm, Alex Payne wrote: > POST requests are unlimited. > > We used to support XMPP as an experimental feature, but we don't currently. > > Delivering push features at our scale is a challenge. We're currently > making our traditional REST request/response APIs the best they can > be. In the future, maybe we'll tackle push as well. In the medium > term, select partners will be able to have tweets pushed to them over > HTTP via our "firehose" mechanism. > > As Andrew suggested, there's been quite a lot of discussion on these > topics in this group and elsewhere on the web. > > On Thu, Mar 12, 2009 at 13:55, Adrian wrote: > > > Hi there, I was wondering if it's possible to push data, rather than > > have the content pulled with JSON or XML fetches. You can poll after > > set amounts of time, but that only present the illusion of Push, and > > uses up bandwith. > > > Also, is the API limit applied to POST requests? > > > Lastly, has Twitter thought about implementing XMPP. > > -- > Alex Payne - API Lead, Twitter, Inc.http://twitter.com/al3x
[twitter-dev] Re: Pushing Twitter data into the clients
POST requests are unlimited. We used to support XMPP as an experimental feature, but we don't currently. Delivering push features at our scale is a challenge. We're currently making our traditional REST request/response APIs the best they can be. In the future, maybe we'll tackle push as well. In the medium term, select partners will be able to have tweets pushed to them over HTTP via our "firehose" mechanism. As Andrew suggested, there's been quite a lot of discussion on these topics in this group and elsewhere on the web. On Thu, Mar 12, 2009 at 13:55, Adrian wrote: > > Hi there, I was wondering if it's possible to push data, rather than > have the content pulled with JSON or XML fetches. You can poll after > set amounts of time, but that only present the illusion of Push, and > uses up bandwith. > > Also, is the API limit applied to POST requests? > > Lastly, has Twitter thought about implementing XMPP. > -- Alex Payne - API Lead, Twitter, Inc. http://twitter.com/al3x
[twitter-dev] Re: Pushing Twitter data into the clients
What is this "XMPP" of which you speak? Tell us more! PS: Read the docs? FAQs? This list's archive? Used Google? Ever? On Thu, Mar 12, 2009 at 3:55 PM, Adrian wrote: > > Hi there, I was wondering if it's possible to push data, rather than > have the content pulled with JSON or XML fetches. You can poll after > set amounts of time, but that only present the illusion of Push, and > uses up bandwith. > > Also, is the API limit applied to POST requests? > > Lastly, has Twitter thought about implementing XMPP. >
