I ran tcpdump on all the three nodes. It looks like that for every [PSH, ACK] there is a missing [ACK] from the other nodes to this 2nd node on port 3888.
On Thu, Apr 7, 2016 at 1:29 PM, s influxdb <[email protected]> wrote: > Thanks Flavio for your quick replies. > The zookeeper version is 3.4.6 > > > > On Thu, Apr 7, 2016 at 1:23 PM, Flavio P JUNQUEIRA <[email protected]> wrote: > >> You need to determine why it is not receiving notification messages. From >> the information you've given, it doesn't look like a zookeeper code issue. >> >> BTW, which version are you using? >> >> -Flavio >> On 7 Apr 2016 21:20, "s influxdb" <[email protected]> wrote: >> >> > nothin on the iptables firewall . >> > >> > What options do i have to reconnect this node to the cluster ? >> > >> > >> > On Thu, Apr 7, 2016 at 10:14 AM, s influxdb <[email protected]> >> wrote: >> > >> > > telnet works on 2888 and 3888 to the other nodes. Now i see >> > > java.net.SocketTimeoutException: connect timed out messages in the >> logs >> > for >> > > node 2 >> > > >> > > On Thu, Apr 7, 2016 at 3:05 AM, Flavio Junqueira <[email protected]> >> wrote: >> > > >> > >> I only see notifications from the node to itself. It says that it is >> > >> connected to 1, but it doesn't seem to be receiving the notification >> > from >> > >> 1. It also doesn't seem to be receiving the connection request from >> 3. >> > >> >> > >> Last time I've seen something like this was due to iptables rules, >> but >> > if >> > >> it was working before and no configuration has changed, then I don't >> > know >> > >> what it could be. >> > >> >> > >> -Flavio >> > >> >> > >> > On 07 Apr 2016, at 05:43, s influxdb <[email protected]> >> wrote: >> > >> > >> > >> > this is the pastie >> > >> > http://pastie.org/10788301 >> > >> > >> > >> > On Wed, Apr 6, 2016 at 9:41 PM, s influxdb <[email protected]> >> > >> wrote: >> > >> > >> > >> >> We had one of the node giving OOM java.lang.OutOfMemoryError: >> unable >> > to >> > >> >> create new native thread and then being unresponsive. >> > >> >> >> > >> >> We tried to add the node back to the cluster but with no luck. >> > >> >> >> > >> >> It doesn't seem to "Receive any notification " messages from the >> > other >> > >> >> nodes. >> > >> >> Keeps "Sending notifications " in loop >> > >> >> >> > >> >> Please see attached the logs of the node that is out of rotation. >> > >> >> >> > >> >> Any inputs appreciated. >> > >> >> >> > >> >> Thanks >> > >> >> >> > >> >> > >> >> > > >> > >> > >
