I re-read the section on updating filter predicates in the api doc and it seems to say that reconnecting every 2 min is acceptable. If thats the case that would have an impact on the design I'm kicking around.
Ben On Nov 16, 10:55 pm, John Kalucki <[email protected]> wrote: > If you have a fairly stable set of tags, you should absolutely obtain > those via the Streaming API. You can migrate dynamic queries that are > made against the REST API to the Streaming API several times an hour. > This will cut your Search velocity to everyone's benefit, and will > improve your result set fidelity. > > Few tags generate more than a handful of updates per second, and most > generate much less traffic. > > -John > > > > Benjamin Darfler wrote: > > It might not be the best use of the Search API but we have made a > > pretty successful system that polls for, on average a few hundred > > tags, in serial, at a rate that keeps us below the 20k/hr limit. We > > also respect 503 errors and back off accordingly. > > > I think what might work the best is some sort of hybrid system. We are > > at a point where some tags/users are known a priori and need a quicker > > and more guaranteed delivery. We could set up a stream for that and > > keep the rest as search requests. > > > I'll give the sampled stream some thought. Its possible that we could > > ingest the sampled stream and pull out messages that match one of our > > terms and dump the rest on the floor. I'd have to prototype that a bit > > to see if we end up with a satisfactory amount of messages. > > > Its quite possible that setting up a stream with a few hundred tags > > would be an overwhelming and unnecessary amount of messages flowing > > into our system since we use twitter messages mostly as filler and > > don't require specific messages or even a lot of messages most of the > > time. > > > Ben > > > On Nov 16, 10:04 pm, John Kalucki <[email protected]> wrote: > > > Benjamin, > > > > I don't think we have a good solution for this use case. Calling the > > > Search REST API or the Streaming API aggressively aren't acceptable > > > techniques. I'd suggest rethinking your app to be less demanding. > > > > Perhaps if you could compute your predicates in advance and keep your > > > update rate to well below once per minute, you could be successful. > > > Would it be possible to base some of your work off of a sampled > > > stream? > > > > -John Kaluckihttp://twitter.com/jkalucki > > > Services, Twitter Inc. > > > > Benjamin Darfler wrote: > > > > I have the use case where I want to ingest twitter messages based on a > > > > set of tags that will changes frequently (possibly a few times a min > > > > or more). I'm polling the search api and can vary the terms I'm > > > > searching for with each request. I would love to move over to the > > > > streaming api but I'm not sure if I can dynamically change the terms > > > > I'm filtering on without tearing down and restarting the stream which > > > > I imagine is not efficient nor particularly friendly to twitter. Is > > > > there a well defined way to use the stream api in this manner?
