I don't doubt that this is happening, but, whenever I've looked into this in the past, the issue boiled down to a poorly written client that didn't detect a TCP Close or TCP Reset when the server, for whatever reason, ended the connection.
However, it appears that something different is happening now. There are reports across multiple client library of this behavior occurring -- low velocity streams where the server and the client think that the connection is open, but no data is read by the client. Now these reports are coincident in time, and do not correlate with server maintenance. This morning, I drilled down with the Phirehose PHP client people on their mailing list. ( http://groups.google.com/group/phirehose-users/browse_frm/thread/6a29c7b9f50d0368) They've provided some good data so far. After I do a little more experimentation, I'll start a fresh thread on this list to address this issue. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sat, Feb 20, 2010 at 6:06 PM, Sami <sami.ben.romdh...@gmail.com> wrote: > Sorry, John but this is really happening and I am having it on a daily > basis in the last 2 weeks on both dev machine and production. I am > using Apache httpClient and this code was working fine for months > before this started to happen 2 weeks ago. I have a java call "String > line =reader.readLine(); " and this call at least once a day just > does not return (100% sure since I am logging all the call chain in > debug mode). > > So please take a look at whatever data you are monitoring to see if > anything can explain this. In the meantime, I am building a monitoring > thread to force a restart on the connection which is really saddening > > Thanks > > > On Feb 19, 2:55 pm, John Kalucki <j...@twitter.com> wrote: > > This shouldn't be happening, and having developers build these sorts of > > workarounds saddens me. > > > > It is possible that the server side is holding dead connections open, but > I > > doubt it -- as I've a considerable amount of data to the contrary. I > suspect > > that the socket code does not detect a close, driven by either a TCP > Close > > or a TCP Reset. I've run connections over the public internet with close > > monitoring and rarely noticed a timeout. > > > > If you point the same client at a file of streaming data on a web server, > > does the client detect the end of file at the correct point? > > > > -John Kaluckihttp://twitter.com/jkalucki > > Infrastructure, Twitter Inc. > > > > > > > > On Fri, Feb 19, 2010 at 9:16 AM, Dima Brodsky <ddbrod...@gmail.com> > wrote: > > > Hi, > > > > > Yup, I saw it the last couple of weeks, this week has been considerably > > > better. I use the delimited stream so I do read(entry size), > read(entry), > > > repeat ... I just put a 30 second timeout on the read operation (this > is all > > > in python) and if a read fails I close the stream and reconnect. This > seems > > > to work quite well ... depending on the chunk size you are reading you > could > > > probably lower the timeout if you are afraid of losing data. > > > > > ttyl > > > Dima > > > > > On 19-Feb-10, at 7:36 AM, rob wrote: > > > > > Has anyone else ran into an issue where over time the Streaming API > > >> just stops sending results? > > > > >> We are using a Ruby library to connect (twitter-stream) which uses > > >> EventMachine to open a persistent connection to the API (we are > > >> tracking and following). > > > > >> The library properly handles reconnection (from dropped connections) > > >> and the various error conditions. > > > > >> All works well for a period of time (8+ hours in some cases, sometimes > > >> a full day) after that the connection does no get dropped but no data > > >> gets sent. (At least that's what is seems as EventMachine feels its > > >> still connected) > > > > >> If we just drop the connection and reconnect all is well and the data > > >> starts to flow again. (Which we could do but that seems like a hack) > > > > >> Anyone else have this issue or should I dive into the EventMachine > > >> code and see if there is an issue in there? > > > > >> Thanks in advance, > > > > >> Rob > > > > > -- > > > ddbrod...@gmail.com > > > > > "The price of reliability is the pursuit of the utmost simplicity. > > > It is a price which the very rich find the most hard to pay." > > > (Sir > > > Antony Hoare, 1980) >
