the threads number of a zookeeper is increased all the time
-----------------------------------------------------------
Key: ZOOKEEPER-939
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-939
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.3.0
Environment: Linux 2.6.9-52bs #2 SMP Fri Jan 26 13:34:38 CST 2007
x86_64 x86_64 x86_64 GNU/Linux
Reporter: Qian Ye
I have a group of zookeeper servers, there are three servers in this group.
server.0=10.81.4.11:2888:3888
server.1=10.23.240.93:2888:3888
server.2=10.23.244.224:2888:3888
At first, the cluster ran well. About several days ago, I shut down the
zookeeper process on one of servers(server.2)., and today, I find that the
other two servers run in wired status(the network is fine). The zookeeper
process take pretty much resource on the two servers:
on server.1 (it's the leader)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26836 work 18 0 12.8g 803m 8724 S 3.7 10.1 195:56.56 java
$ ll /proc/26836/fd/ | wc -l
3586
[[email protected] conf]$ ll /proc/26836/task/ | wc -l
10510
some warning log:
2010-11-24 15:37:48,705 - WARN [Thread-37409:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:39:48,626 - WARN [Thread-37414:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:39:48,656 - WARN [Thread-37413:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:39:48,657 - WARN [Thread-37413:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:41:48,614 - WARN [Thread-37417:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:41:48,643 - WARN [Thread-37418:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:41:48,662 - WARN [Thread-37417:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:43:48,627 - WARN [Thread-37421:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:43:48,627 - WARN [Thread-37422:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:43:48,654 - WARN [Thread-37421:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:44:48,622 - WARN [Thread-37424:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:44:48,652 - WARN [Thread-37423:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:44:48,653 - WARN [Thread-37423:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:45:48,668 - WARN [Thread-37426:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:46:48,647 - WARN [Thread-37427:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:46:48,722 - WARN [Thread-37428:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:46:48,736 - WARN [Thread-37427:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:47:48,687 - WARN [Thread-37430:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
on server.0
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27322 work 19 0 15.2g 943m 9140 S 38.6 11.8 1396:51 java
$ ll /proc/27322/fd/ | wc -l
3587
$ ll /proc/27322/task/ | wc -l
12938
2010-11-24 15:37:49,269 - WARN [Thread-37407:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:39:49,235 - WARN [Thread-37412:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:39:49,410 - WARN [Thread-37411:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:39:49,411 - WARN [Thread-37411:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:41:49,314 - WARN [Thread-37416:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:41:49,383 - WARN [Thread-37415:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:41:49,405 - WARN [Thread-37415:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:43:49,372 - WARN [Thread-37420:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:43:49,512 - WARN [Thread-37419:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:43:49,513 - WARN [Thread-37419:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:44:49,407 - WARN [Thread-37422:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:45:49,645 - WARN [Thread-37424:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:45:49,781 - WARN [Thread-37423:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:45:49,799 - WARN [Thread-37423:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:46:49,495 - WARN [Thread-37427:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:47:49,541 - WARN [Thread-37429:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:47:49,622 - WARN [Thread-37428:quorumcnxmanager$sendwor...@581]
- Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:47:49,622 - WARN [Thread-37428:quorumcnxmanager$sendwor...@589]
- Send worker leaving thread
2010-11-24 15:48:48,827 - WARN [Thread-37431:quorumcnxmanager$recvwor...@658]
- Connection broken:
java.io.IOException: Channel eof
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
What's more, the number of threads under the zookeeper process is still
increasing time by time. It seems that , something is wrong in communication of
the two servers. Have anyone met such problem before?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
