I re-read the section on updating filter predicates in the api doc and
it seems to say that reconnecting every 2 min is acceptable. If thats
the case that would have an impact on the design I'm kicking around.
On Nov 16, 10:55 pm, John Kalucki <jkalu...@gmail.com> wrote:
> If you have a fairly stable set of tags, you should absolutely obtain
> those via the Streaming API. You can migrate dynamic queries that are
> made against the REST API to the Streaming API several times an hour.
> This will cut your Search velocity to everyone's benefit, and will
> improve your result set fidelity.
> Few tags generate more than a handful of updates per second, and most
> generate much less traffic.
> Benjamin Darfler wrote:
> > It might not be the best use of the Search API but we have made a
> > pretty successful system that polls for, on average a few hundred
> > tags, in serial, at a rate that keeps us below the 20k/hr limit. We
> > also respect 503 errors and back off accordingly.
> > I think what might work the best is some sort of hybrid system. We are
> > at a point where some tags/users are known a priori and need a quicker
> > and more guaranteed delivery. We could set up a stream for that and
> > keep the rest as search requests.
> > I'll give the sampled stream some thought. Its possible that we could
> > ingest the sampled stream and pull out messages that match one of our
> > terms and dump the rest on the floor. I'd have to prototype that a bit
> > to see if we end up with a satisfactory amount of messages.
> > Its quite possible that setting up a stream with a few hundred tags
> > would be an overwhelming and unnecessary amount of messages flowing
> > into our system since we use twitter messages mostly as filler and
> > don't require specific messages or even a lot of messages most of the
> > time.
> > Ben
> > On Nov 16, 10:04 pm, John Kalucki <jkalu...@gmail.com> wrote:
> > > Benjamin,
> > > I don't think we have a good solution for this use case. Calling the
> > > Search REST API or the Streaming API aggressively aren't acceptable
> > > techniques. I'd suggest rethinking your app to be less demanding.
> > > Perhaps if you could compute your predicates in advance and keep your
> > > update rate to well below once per minute, you could be successful.
> > > Would it be possible to base some of your work off of a sampled
> > > stream?
> > > -John Kaluckihttp://twitter.com/jkalucki
> > > Services, Twitter Inc.
> > > Benjamin Darfler wrote:
> > > > I have the use case where I want to ingest twitter messages based on a
> > > > set of tags that will changes frequently (possibly a few times a min
> > > > or more). I'm polling the search api and can vary the terms I'm
> > > > searching for with each request. I would love to move over to the
> > > > streaming api but I'm not sure if I can dynamically change the terms
> > > > I'm filtering on without tearing down and restarting the stream which
> > > > I imagine is not efficient nor particularly friendly to twitter. Is
> > > > there a well defined way to use the stream api in this manner?