It might not be the best use of the Search API but we have made a
pretty successful system that polls for, on average a few hundred
tags, in serial, at a rate that keeps us below the 20k/hr limit. We
also respect 503 errors and back off accordingly.

I think what might work the best is some sort of hybrid system. We are
at a point where some tags/users are known a priori and need a quicker
and more guaranteed delivery. We could set up a stream for that and
keep the rest as search requests.

I'll give the sampled stream some thought. Its possible that we could
ingest the sampled stream and pull out messages that match one of our
terms and dump the rest on the floor. I'd have to prototype that a bit
to see if we end up with a satisfactory amount of messages.

Its quite possible that setting up a stream with a few hundred tags
would be an overwhelming and unnecessary amount of messages flowing
into our system since we use twitter messages mostly as filler and
don't require specific messages or even a lot of messages most of the


On Nov 16, 10:04 pm, John Kalucki <> wrote:
> Benjamin,
> I don't think we have a good solution for this use case. Calling the
> Search REST API or the Streaming API aggressively aren't acceptable
> techniques. I'd suggest rethinking your app to be less demanding.
> Perhaps if you could compute your predicates in advance and keep your
> update rate to well below once per minute, you could be successful.
> Would it be possible to  base some of your work off of a sampled
> stream?
> -John Kalucki
> Services, Twitter Inc.
> Benjamin Darfler wrote:
> > I have the use case where I want to ingest twitter messages based on a
> > set of tags that will changes frequently (possibly a few times a min
> > or more). I'm polling the search api and can vary the terms I'm
> > searching for with each request. I would love to move over to the
> > streaming api but I'm not sure if I can dynamically change the terms
> > I'm filtering on without tearing down and restarting the stream which
> > I imagine is not efficient nor particularly friendly to twitter. Is
> > there a well defined way to use the stream api in this manner?

Reply via email to