Flavio Junqueira commented on ZOOKEEPER-880:

I think we agree that monitoring alone was not causing the issue. But, your 
logs indicate that there were some orphan threads due to the monitoring, and we 
can see it from excerpts of your logs like the one I posted above. Without the 
monitoring, the same problem is being triggered, though, but apparently in a 
different way and it is not clear why. You can see it from all the "Channel 
eof" messages on the log. 

To solve this issue, we need to understand the following:

# What's causing those IOExceptions?
# Why are we even starting a new connection if there is no leader election 
going on? 

Do you folks have any idea if there is anything in your environment that could 
be causing those TCP connections to break? 

> QuorumCnxManager$SendWorker grows without bounds
> ------------------------------------------------
>                 Key: ZOOKEEPER-880
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-880
>             Project: Zookeeper
>          Issue Type: Bug
>    Affects Versions: 3.2.2
>            Reporter: Jean-Daniel Cryans
>            Priority: Critical
>         Attachments: hbase-hadoop-zookeeper-sv4borg12.log.gz, 
> hbase-hadoop-zookeeper-sv4borg9.log.gz, jstack, 
> TRACE-hbase-hadoop-zookeeper-sv4borg9.log.gz
> We're seeing an issue where one server in the ensemble has a steady growing 
> number of QuorumCnxManager$SendWorker threads up to a point where the OS runs 
> out of native threads, and at the same time we see a lot of exceptions in the 
> logs.  This is on 3.2.2 and our config looks like:
> {noformat}
> tickTime=3000
> dataDir=/somewhere_thats_not_tmp
> clientPort=2181
> initLimit=10
> syncLimit=5
> server.0=sv4borg9:2888:3888
> server.1=sv4borg10:2888:3888
> server.2=sv4borg11:2888:3888
> server.3=sv4borg12:2888:3888
> server.4=sv4borg13:2888:3888
> {noformat}
> The issue is on the first server. I'm going to attach threads dumps and logs 
> in moment.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to