So shouldn't all servers in another DC just have one session? So even if I
have 50 observers in another DC that should be 50 sessions established since
the IP doesn't change correct? Am I missing something? In some ZK clients I
see the following exception even though they are in the same DC.
WARN - Session 0x0 for server msgzkapp013.abc.com/10.138.43.219:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
WARN - Session 0x0 for server msgzkapp012.abc.com/10.138.42.219:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:851)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
What might be happening in this case?
Cheers
Avinash
On Tue, Oct 5, 2010 at 9:47 AM, Patrick Hunt ph...@apache.org wrote:
A (/10.30.84.207 a zookeeper client) is attempting to establish more
then 10 sessions to the ZooKeeper server where you got the log. This can be
caused by a bug in user code (we've seen bugs where incorrectly implemented
ZK clients attempt to create an infinite number of sessions, which
essentially DOS the service, so we added the maxClientCnxn default limit of
10).
Often users see this problem when they are trying to simulate a real
environment - they run a simulated set of clients sessions (10) from a
single host (ip) hitting the servers. However in your case I'm guessing that
it has something to do with this
The observers are running in a different data center from where the ZK
non-observers are running.
Could you have a NAT or some other networking configuration that makes all
the observers seem to be coming from the same IP address?
Patrick
On Tue, Oct 5, 2010 at 9:33 AM, Avinash Lakshman
avinash.laksh...@gmail.com wrote:
Thanks Patrick. But what does this mean? I see the log on server A telling
me Too many connections from A - default is 10. Too many connection from
A
to whom? I do not see who the other end of the connection is.
Cheers
Avinash
On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt ph...@apache.org wrote:
See this configuration param in the docs maxClientCnxns:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
Patrick
On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman
avinash.laksh...@gmail.com
wrote:
I find this happening in my observers node in the logs. The observers
are
running in a different data center from where the ZK non-observers are
running. The only way to fix this seems to be restarting. How can I
start
addressing this? Here is the stack trace.
Too many connections from /10.30.84.207 - max is 10
WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
Please advice.
Cheers
Avinash
