John, great information. thanks a lot. I'll put in a proper wait time before next re-connection. -aj
On Sun, Jun 14, 2009 at 8:14 PM, John Kalucki <[email protected]> wrote: > > AJ, > > If you had a valid connection and the connection drops, reconnect > immediately. This is encouraged! > > If you attempt a connection and get a TCP or IP level error, back off > linearly, but cap the backoff to something fairly short. Perhaps start > at 20 milliseconds, double, and cap at 15 seconds. There's probably a > transitory network problem and it will probably clear up quickly. > > If you get a HTTP error (4XX), backoff linearly, but cap the backoff > at something longer, perhaps start at 250 milliseconds, double, and > cap at 120 seconds. Whatever has caused the issue isn't going away > anytime soon. There's not much point in polling any faster and you are > just more likely to run afoul of some rate limit. > > The service is fairly lenient. You aren't going to get banned for a > few dozen bungled connections here and there. But, if you do anything > in a while loop that also doesn't have a sleep, you'll eventually get > the hatchet for some small number of minutes. If you get the hatchet > repeatedly, you'll be cut off for an indeterminate period of time. > > There are four main reasons to have your connection closed: > * Duplicate clients logins (earlier connections terminated) > * Hosebird server restarts (code deploys) > * Lagging connection getting thrown off (client too slow, or > insufficient bandwidth) > * General Twitter network maintenance (Load balancer restarts, network > reconfigurations, other very very rare events) > > We plan to have enough spare capacity on the surviving servers to > absorb the load from server restarts. You must ensure that your client > is fast enough and that you have sufficient bandwidth and a stable > enough connection to consume your stream. I usually see connections > that survive for a few days before mysteriously being dropped. Just > reconnect in these cases. > > -John Kalucki > Services, Twitter Inc. > > > On Jun 14, 3:31 pm, AJ <[email protected]> wrote: > > The streaming api is great, but it sometimes closes the connection for > > whatever reason. my realtime system must figure out when to reconnect > > automatically. the auto-reconnection can't blindly request a > > connection whenever it is not connected, otherwise it will floor the > > api and may cause the api to ban or refuse the user's request. it's > > bad to bombard the api server with repeated connection requests. > > Could the api team recommend some best practice for dealing with auto- > > reconnection? > > > > maybe certain error code or error message can indicate the cause of > > dropping connection and wait time for next connection request. I just > > a long list of exceptions from streaming api as a result of repeated > > connection, and the different messages are: > > > > twitter4j.TwitterException: Address already in use: connect > > twitter4j.TwitterException: Authentication credentials were missing or > > incorrect. > > twitter4j.TwitterException: Connection refused: connect > > twitter4j.TwitterException: No route to host: connect > > twitter4j.TwitterException: Stream closed. > > twitter4j.TwitterException: The request is understood, but it has been > > refused. An accompanying error message will explain why. > > twitter4j.TwitterException: connect timed out > > > > How to prevent such situation of repeated connections requests? > > > > thanks, > > aj > -- AJ Chen, PhD Co-Chair, Semantic Web SIG, sdforum.org Technical Architect, healthline.com http://web2express.org Palo Alto, CA
