Are you using multilang protocol? I know that after upgrading to 0.9.4 it seemed like I was being affected by this bug - https://issues.apache.org/jira/browse/STORM-738 and rolled back to previous stable version of 0.8.2. I did not verify this thoroughly on my cluster though.
From: Nick R. Katsipoulakis [mailto:[email protected]] Sent: Thursday, September 03, 2015 9:08 AM To: [email protected] Subject: Re: Netty reconnect Hello again, I read STORM-404 and I saw that is resolved on version 0.9.4. However, I have version 0.9.4 installed in my cluster, and I have seen similar behavior in my workers. In fact, at random times I would see that some workers were considered dead (Netty was dropping messages) and they would be restarted by the nimbus. Currently, I only see dropped messages but not restarted workers. FYI, my cluster has the following information * 3X AWS m4.xlarge instances for ZooKeeper and Nimbus * 4X AWS m4.xlarge instances for Supervisors (each one with 2 workers) Thanks, Nick 2015-09-03 8:38 GMT-04:00 Ganesh Chandrasekaran <[email protected]<mailto:[email protected]>>: Agreed with Jitendra. We were using 0.9.3 version and facing the same issue of netty reconnects which was the issue 404. Upgrading to 0.9.4 fixed the issue. Thanks, Ganesh From: Jitendra Yadav [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, September 03, 2015 8:20 AM To: [email protected]<mailto:[email protected]> Subject: Re: Netty reconnect I don't know your storm version, but it's worth to check these Jira's and see if similar scenario occurring. https://issues.apache.org/jira/browse/STORM-404 https://issues.apache.org/jira/browse/STORM-450 Thanks Jitendra On Thu, Sep 3, 2015 at 5:22 PM, John Yost <[email protected]<mailto:[email protected]>> wrote: Hi Everyone, When I see this, it is evidence that one or more of the workers are not starting up, which results in connections either not occuring or reconnecting occuring when supervisors kill workers that don't start up properly. I recommend checking the supervisor and nimbus logs to see if there are any root causes other than network issues causing the connect/reconnect. --John On Thu, Sep 3, 2015 at 7:32 AM, Nick R. Katsipoulakis <[email protected]<mailto:[email protected]>> wrote: Hello Kashyap, I have been having the same issue for some time now on my AWS cluster. To be honest, I do not know how to resolve it. Regards, Nick 2015-09-03 0:07 GMT-04:00 Kashyap Mhaisekar <[email protected]<mailto:[email protected]>>: Hi, Has anyone experienced Netty reconnects repeatedly? My workers seem to be eternally in reconnect state and topology doesn't serve messages at all. It gets connected once in a while and then goes back to getting reconnecting. Any fixes for this? "Reconnect started for Netty-Client" Thanks Kashyap -- Nikolaos Romanos Katsipoulakis, University of Pittsburgh, PhD candidate -- Nikolaos Romanos Katsipoulakis, University of Pittsburgh, PhD candidate
