Our client would make even less sense to you then. It's written in Scala!
On Sun, Jan 17, 2010 at 9:56 PM, M. Edward (Ed) Borasky <zzn...@gmail.com>wrote:
> As an aside, could Twitter release the streaming client they use under
> some open source license, so we can use it as a prototype? I took a
> look at the one Tom May of Gist wrote using Apache HttpClient and it
> didn't make much sense to me - it was importing a bunch of Java
> libraries and I'm not a Java programmer.
> On Jan 16, 10:18 pm, John Kalucki <j...@twitter.com> wrote:
> > Given a reasonable stack, it shouldn't be all that hard to build
> > robust. Our internal streaming client, which transits every tweet that
> > see on the streaming api, seems to work just fine through various forms
> > abuse, and it's, roughly, a few hundred lines wrapped around Apache
> > httpclient.
> > On the other hand, I suspect that dependability is all but impossible on
> > some stacks, or will require some heroism on the part of a library
> > developer.
> > As a community, we need clients that trivially allow robustness in a
> > of stacks. We'll get there soon enough.
> > On Sat, Jan 16, 2010 at 10:05 PM, M. Edward (Ed) Borasky
> > <zzn...@gmail.com>wrote:
> > > On Jan 16, 7:28 pm, John Kalucki <j...@twitter.com> wrote:
> > > > I'd strongly suggest consuming the Streaming API only from persistent
> > > > processes that write into some form of durable asynchronous queue (of
> > > > type) for your application to consume. Running curl periodically is
> > > unlikely
> > > > to be a robust solution.
> > > > Select one of the existing Streaming API clients out there and wrap
> it in
> > > a
> > > > durable process. Write to rotated log files, a message queue, or
> > > > other mechanism that you choose, to buffer the arrival of new
> > > > before consumption by your application. This will allow you to
> > > your
> > > > application at will without data loss.
> > > I don't know that there are any open source libraries out there yet
> > > that are robust enough to do that. At the moment, I'm working
> > > exclusively in Perl, and "AnyEvent::Twitter::Stream" seems to be the
> > > only Perl Streaming API consumer with any kind of mileage on it. As
> > > you point out, real-time programming for robustness is a non-trivial
> > > exercise. It would be nice if someone would build a C library and SWIG
> > > ".i" files. ;-)
> > > --
> > > M. Edward (Ed) Borasky
> > >http://borasky-research.net/smart-at-znmeb
> > > "A mathematician is a device for turning coffee into theorems." ~ Paul
> > > Erdős