Re: [twitter-dev] Streaming API connection abandonment & separation anxiety issues
connection issue around 15:56 UTC, see our log below: 22-02-2010 16:55:41 - Consume rate: 13 status/sec (764 total), avg enqueueStatus(): 1.51ms, avg checkFilterPredicates(): 349.46ms (12 total) over 60 seconds. 22-02-2010 16:56:18 - Phirehose connection error occured: 22-02-2010 16:56:18 - Closing Phirehose connection. 22-02-2010 16:56:18 - Connecting to twitter stream: http://stream.twitter.com/1/statuses/filter.json with params: array ( 'delimit 22-02-2010 16:56:18 - Resolved host stream.twitter.com to 168.143.162.55 22-02-2010 16:56:18 - Connecting to 168.143.162.55 22-02-2010 17:01:20 - Idle timeout: No statuses received for > 300 seconds. Reconnecting. 22-02-2010 17:01:20 - Closing Phirehose connection. Kim
Re: [twitter-dev] Streaming API connection abandonment & separation anxiety issues
* John Kalucki [100222 10:51]: > If you had a correlative experience within a minute or so of 15:55 UTC, > please respond to this message. Indeed, I did. From the application log: 2010/02/22 07:56:14 9481340618: RT @whitehouse: The Presi... 2010/02/22 07:57:14 timeout 2010/02/22 07:57:14 Waiting 0 seconds before reconnecting... 2010/02/22 07:57:14 connecting -Marc
[twitter-dev] Streaming API connection abandonment & separation anxiety issues
A number of developers have reported abandoned connection issues on the Streaming API starting, perhaps, about two weeks ago. The symptoms include a long-established TCP connection to stream.twitter.com going quiet, with the connection mysteriously held open for perhaps hours afterward. After sorting through a lot of conflicting data and chasing a few wild geese, I finally reproduced this problem at Feb 22 15:55 UTC (7:55am PST). I'd imagine that a number of streams were abandoned at this time. If you had a correlative experience within a minute or so of 15:55 UTC, please respond to this message. We currently suspect an infrequent hardware load balancer issue, perhaps related to a recent configuration change. The appearance is that the load balancer is, for whatever reason, dropping valid connections, closing the connection to the Streaming API servers, but not sending a TCP FIN or TCP RST to the client. This is bad. We're treating this as a critical production issue and working through the details with network operations. I'll follow up as we learn more. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc.
