[
https://issues.apache.org/jira/browse/ZOOKEEPER-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar resolved ZOOKEEPER-939.
-------------------------------------
Resolution: Duplicate
Resolving as duplicate of ZOOKEEPER-880. thanks for pointing out Thomas!
> the threads number of a zookeeper is increased all the time
> -----------------------------------------------------------
>
> Key: ZOOKEEPER-939
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-939
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.3.0
> Environment: Linux 2.6.9-52bs #2 SMP Fri Jan 26 13:34:38 CST 2007
> x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Qian Ye
>
> I have a group of zookeeper servers, there are three servers in this group.
> server.0=10.81.4.11:2888:3888
> server.1=10.23.240.93:2888:3888
> server.2=10.23.244.224:2888:3888
> At first, the cluster ran well. About several days ago, I shut down the
> zookeeper process on one of servers(server.2)., and today, I find that the
> other two servers run in wired status(the network is fine). The zookeeper
> process take pretty much resource on the two servers:
> on server.1 (it's the leader)
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 26836 work 18 0 12.8g 803m 8724 S 3.7 10.1 195:56.56 java
> $ ll /proc/26836/fd/ | wc -l
> 3586
> [[email protected] conf]$ ll /proc/26836/task/ | wc -l
> 10510
> some warning log:
> 2010-11-24 15:37:48,705 - WARN
> [Thread-37409:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:39:48,626 - WARN
> [Thread-37414:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:39:48,656 - WARN
> [Thread-37413:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:39:48,657 - WARN
> [Thread-37413:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:41:48,614 - WARN
> [Thread-37417:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:41:48,643 - WARN
> [Thread-37418:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:41:48,662 - WARN
> [Thread-37417:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:43:48,627 - WARN
> [Thread-37421:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:43:48,627 - WARN
> [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:43:48,654 - WARN
> [Thread-37421:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:44:48,622 - WARN
> [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:44:48,652 - WARN
> [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:44:48,653 - WARN
> [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:45:48,668 - WARN
> [Thread-37426:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:46:48,647 - WARN
> [Thread-37427:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:46:48,722 - WARN
> [Thread-37428:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:46:48,736 - WARN
> [Thread-37427:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:47:48,687 - WARN
> [Thread-37430:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> on server.0
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 27322 work 19 0 15.2g 943m 9140 S 38.6 11.8 1396:51 java
> $ ll /proc/27322/fd/ | wc -l
> 3587
> $ ll /proc/27322/task/ | wc -l
> 12938
> 2010-11-24 15:37:49,269 - WARN
> [Thread-37407:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:39:49,235 - WARN
> [Thread-37412:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:39:49,410 - WARN
> [Thread-37411:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:39:49,411 - WARN
> [Thread-37411:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:41:49,314 - WARN
> [Thread-37416:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:41:49,383 - WARN
> [Thread-37415:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:41:49,405 - WARN
> [Thread-37415:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:43:49,372 - WARN
> [Thread-37420:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:43:49,512 - WARN
> [Thread-37419:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:43:49,513 - WARN
> [Thread-37419:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:44:49,407 - WARN
> [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:45:49,645 - WARN
> [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:45:49,781 - WARN
> [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:45:49,799 - WARN
> [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:46:49,495 - WARN
> [Thread-37427:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:47:49,541 - WARN
> [Thread-37429:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:47:49,622 - WARN
> [Thread-37428:QuorumCnxManager$SendWorker@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:47:49,622 - WARN
> [Thread-37428:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
> 2010-11-24 15:48:48,827 - WARN
> [Thread-37431:QuorumCnxManager$RecvWorker@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> What's more, the number of threads under the zookeeper process is still
> increasing time by time. It seems that , something is wrong in communication
> of the two servers. Have anyone met such problem before?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
