As an aside, could Twitter release the streaming client they use under some open source license, so we can use it as a prototype? I took a look at the one Tom May of Gist wrote using Apache HttpClient and it didn't make much sense to me - it was importing a bunch of Java libraries and I'm not a Java programmer.
On Jan 16, 10:18 pm, John Kalucki <j...@twitter.com> wrote: > Given a reasonable stack, it shouldn't be all that hard to build something > robust. Our internal streaming client, which transits every tweet that you > see on the streaming api, seems to work just fine through various forms of > abuse, and it's, roughly, a few hundred lines wrapped around Apache > httpclient. > > On the other hand, I suspect that dependability is all but impossible on > some stacks, or will require some heroism on the part of a library > developer. > > As a community, we need clients that trivially allow robustness in a variety > of stacks. We'll get there soon enough. > > On Sat, Jan 16, 2010 at 10:05 PM, M. Edward (Ed) Borasky > <zzn...@gmail.com>wrote: > > > > > On Jan 16, 7:28 pm, John Kalucki <j...@twitter.com> wrote: > > > I'd strongly suggest consuming the Streaming API only from persistent > > > processes that write into some form of durable asynchronous queue (of any > > > type) for your application to consume. Running curl periodically is > > unlikely > > > to be a robust solution. > > > > Select one of the existing Streaming API clients out there and wrap it in > > a > > > durable process. Write to rotated log files, a message queue, or whatever > > > other mechanism that you choose, to buffer the arrival of new statuses > > > before consumption by your application. This will allow you to restart > > your > > > application at will without data loss. > > > I don't know that there are any open source libraries out there yet > > that are robust enough to do that. At the moment, I'm working > > exclusively in Perl, and "AnyEvent::Twitter::Stream" seems to be the > > only Perl Streaming API consumer with any kind of mileage on it. As > > you point out, real-time programming for robustness is a non-trivial > > exercise. It would be nice if someone would build a C library and SWIG > > ".i" files. ;-) > > > -- > > M. Edward (Ed) Borasky > >http://borasky-research.net/smart-at-znmeb > > > "A mathematician is a device for turning coffee into theorems." ~ Paul > > Erdős