[
https://issues.apache.org/jira/browse/ZOOKEEPER-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935253#action_12935253
]
Flavio Junqueira commented on ZOOKEEPER-939:
--------------------------------------------
Hi Qian, Please have a look at ZOOKEEPER-880. Also, please consider uploading
logs and snapshots separately instead of posting them on the description of the
jira, just so that we don't get long descriptions with every update to the jira.
> the threads number of a zookeeper is increased all the time
> -----------------------------------------------------------
>
> Key: ZOOKEEPER-939
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-939
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.3.0
> Environment: Linux 2.6.9-52bs #2 SMP Fri Jan 26 13:34:38 CST 2007
> x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Qian Ye
>
> I have a group of zookeeper servers, there are three servers in this group.
> server.0=10.81.4.11:2888:3888
> server.1=10.23.240.93:2888:3888
> server.2=10.23.244.224:2888:3888
> At first, the cluster ran well. About several days ago, I shut down the
> zookeeper process on one of servers(server.2)., and today, I find that the
> other two servers run in wired status(the network is fine). The zookeeper
> process take pretty much resource on the two servers:
> on server.1 (it's the leader)
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 26836 work 18 0 12.8g 803m 8724 S 3.7 10.1 195:56.56 java
> $ ll /proc/26836/fd/ | wc -l
> 3586
> [[email protected] conf]$ ll /proc/26836/task/ | wc -l
> 10510
> some warning log:
> 2010-11-24 15:37:48,705 - WARN
> [Thread-37409:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:39:48,626 - WARN
> [Thread-37414:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:39:48,656 - WARN
> [Thread-37413:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:39:48,657 - WARN
> [Thread-37413:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:41:48,614 - WARN
> [Thread-37417:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:41:48,643 - WARN
> [Thread-37418:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:41:48,662 - WARN
> [Thread-37417:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:43:48,627 - WARN
> [Thread-37421:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:43:48,627 - WARN
> [Thread-37422:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:43:48,654 - WARN
> [Thread-37421:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:44:48,622 - WARN
> [Thread-37424:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:44:48,652 - WARN
> [Thread-37423:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:44:48,653 - WARN
> [Thread-37423:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:45:48,668 - WARN
> [Thread-37426:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:46:48,647 - WARN
> [Thread-37427:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:46:48,722 - WARN
> [Thread-37428:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:46:48,736 - WARN
> [Thread-37427:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:47:48,687 - WARN
> [Thread-37430:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> on server.0
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 27322 work 19 0 15.2g 943m 9140 S 38.6 11.8 1396:51 java
> $ ll /proc/27322/fd/ | wc -l
> 3587
> $ ll /proc/27322/task/ | wc -l
> 12938
> 2010-11-24 15:37:49,269 - WARN
> [Thread-37407:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:39:49,235 - WARN
> [Thread-37412:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:39:49,410 - WARN
> [Thread-37411:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:39:49,411 - WARN
> [Thread-37411:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:41:49,314 - WARN
> [Thread-37416:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:41:49,383 - WARN
> [Thread-37415:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:41:49,405 - WARN
> [Thread-37415:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:43:49,372 - WARN
> [Thread-37420:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:43:49,512 - WARN
> [Thread-37419:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:43:49,513 - WARN
> [Thread-37419:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:44:49,407 - WARN
> [Thread-37422:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:45:49,645 - WARN
> [Thread-37424:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:45:49,781 - WARN
> [Thread-37423:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:45:49,799 - WARN
> [Thread-37423:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:46:49,495 - WARN
> [Thread-37427:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> 2010-11-24 15:47:49,541 - WARN
> [Thread-37429:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.nio.channels.AsynchronousCloseException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
> 2010-11-24 15:47:49,622 - WARN
> [Thread-37428:quorumcnxmanager$sendwor...@581] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
> 2010-11-24 15:47:49,622 - WARN
> [Thread-37428:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
> 2010-11-24 15:48:48,827 - WARN
> [Thread-37431:quorumcnxmanager$recvwor...@658] - Connection broken:
> java.io.IOException: Channel eof
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
> What's more, the number of threads under the zookeeper process is still
> increasing time by time. It seems that , something is wrong in communication
> of the two servers. Have anyone met such problem before?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
