It might not be the best use of the Search API but we have made a pretty successful system that polls for, on average a few hundred tags, in serial, at a rate that keeps us below the 20k/hr limit. We also respect 503 errors and back off accordingly.
I think what might work the best is some sort of hybrid system. We are at a point where some tags/users are known a priori and need a quicker and more guaranteed delivery. We could set up a stream for that and keep the rest as search requests. I'll give the sampled stream some thought. Its possible that we could ingest the sampled stream and pull out messages that match one of our terms and dump the rest on the floor. I'd have to prototype that a bit to see if we end up with a satisfactory amount of messages. Its quite possible that setting up a stream with a few hundred tags would be an overwhelming and unnecessary amount of messages flowing into our system since we use twitter messages mostly as filler and don't require specific messages or even a lot of messages most of the time. Ben On Nov 16, 10:04 pm, John Kalucki <jkalu...@gmail.com> wrote: > Benjamin, > > I don't think we have a good solution for this use case. Calling the > Search REST API or the Streaming API aggressively aren't acceptable > techniques. I'd suggest rethinking your app to be less demanding. > > Perhaps if you could compute your predicates in advance and keep your > update rate to well below once per minute, you could be successful. > Would it be possible to base some of your work off of a sampled > stream? > > -John Kaluckihttp://twitter.com/jkalucki > Services, Twitter Inc. > > > > Benjamin Darfler wrote: > > I have the use case where I want to ingest twitter messages based on a > > set of tags that will changes frequently (possibly a few times a min > > or more). I'm polling the search api and can vary the terms I'm > > searching for with each request. I would love to move over to the > > streaming api but I'm not sure if I can dynamically change the terms > > I'm filtering on without tearing down and restarting the stream which > > I imagine is not efficient nor particularly friendly to twitter. Is > > there a well defined way to use the stream api in this manner?
