Thanks thats great to know. I'm sure with this information some sort of new design that can be found. I'll be giving it some thought at the end of the week.
Ben On Nov 16, 10:55 pm, John Kalucki <[email protected]> wrote: > If you have a fairly stable set of tags, you should absolutely obtain > those via the Streaming API. You can migrate dynamic queries that are > made against the REST API to the Streaming API several times an hour. > This will cut your Search velocity to everyone's benefit, and will > improve your result set fidelity. > > Few tags generate more than a handful of updates per second, and most > generate much less traffic. > > -John > > > > Benjamin Darfler wrote: > > It might not be the best use of the Search API but we have made a > > pretty successful system that polls for, on average a few hundred > > tags, in serial, at a rate that keeps us below the 20k/hr limit. We > > also respect 503 errors and back off accordingly. > > > I think what might work the best is some sort of hybrid system. We are > > at a point where some tags/users are known a priori and need a quicker > > and more guaranteed delivery. We could set up a stream for that and > > keep the rest as search requests. > > > I'll give the sampled stream some thought. Its possible that we could > > ingest the sampled stream and pull out messages that match one of our > > terms and dump the rest on the floor. I'd have to prototype that a bit > > to see if we end up with a satisfactory amount of messages. > > > Its quite possible that setting up a stream with a few hundred tags > > would be an overwhelming and unnecessary amount of messages flowing > > into our system since we use twitter messages mostly as filler and > > don't require specific messages or even a lot of messages most of the > > time. > > > Ben > > > On Nov 16, 10:04 pm, John Kalucki <[email protected]> wrote: > > > Benjamin, > > > > I don't think we have a good solution for this use case. Calling the > > > Search REST API or the Streaming API aggressively aren't acceptable > > > techniques. I'd suggest rethinking your app to be less demanding. > > > > Perhaps if you could compute your predicates in advance and keep your > > > update rate to well below once per minute, you could be successful. > > > Would it be possible to base some of your work off of a sampled > > > stream? > > > > -John Kaluckihttp://twitter.com/jkalucki > > > Services, Twitter Inc. > > > > Benjamin Darfler wrote: > > > > I have the use case where I want to ingest twitter messages based on a > > > > set of tags that will changes frequently (possibly a few times a min > > > > or more). I'm polling the search api and can vary the terms I'm > > > > searching for with each request. I would love to move over to the > > > > streaming api but I'm not sure if I can dynamically change the terms > > > > I'm filtering on without tearing down and restarting the stream which > > > > I imagine is not efficient nor particularly friendly to twitter. Is > > > > there a well defined way to use the stream api in this manner?
