So shouldn't all servers in another DC just have one session? So even if I
have 50 observers in another DC that should be 50 sessions established since
the IP doesn't change correct? Am I missing something? In some ZK clients I
see the following exception even though they are in the same DC.
WARN - Session 0x0 for server msgzkapp013.abc.com/10.138.43.219:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
WARN - Session 0x0 for server msgzkapp012.abc.com/10.138.42.219:5001,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:851)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
What might be happening in this case?
Cheers
Avinash
On Tue, Oct 5, 2010 at 9:47 AM, Patrick Hunt <ph...@apache.org> wrote:
> "A" (/10.30.84.207 a zookeeper client) is attempting to establish more
> then 10 sessions to the ZooKeeper server where you got the log. This can be
> caused by a bug in user code (we've seen bugs where incorrectly implemented
> ZK clients attempt to create an infinite number of sessions, which
> essentially DOS the service, so we added the maxClientCnxn default limit of
> 10).
>
> Often users see this problem when they are trying to simulate a real
> environment - they run a simulated set of clients sessions (>10) from a
> single host (ip) hitting the servers. However in your case I'm guessing that
> it has something to do with this
>
> "The observers are running in a different data center from where the ZK
> non-observers are running."
>
> Could you have a NAT or some other networking configuration that makes all
> the observers seem to be coming from the same IP address?
>
> Patrick
>
> On Tue, Oct 5, 2010 at 9:33 AM, Avinash Lakshman <
> avinash.laksh...@gmail.com> wrote:
>
>> Thanks Patrick. But what does this mean? I see the log on server A telling
>> me "Too many connections from A - default is 10". Too many connection from
>> A
>> to whom? I do not see who the other end of the connection is.
>>
>> Cheers
>> Avinash
>>
>> On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt <ph...@apache.org> wrote:
>>
>> > See this configuration param in the docs "maxClientCnxns":
>> >
>> >
>> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
>> >
>> > <
>> >
>> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
>> > >
>> > Patrick
>> >
>> > On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman <
>> > avinash.laksh...@gmail.com
>> > > wrote:
>> >
>> > > I find this happening in my observers node in the logs. The observers
>> are
>> > > running in a different data center from where the ZK non-observers are
>> > > running. The only way to fix this seems to be restarting. How can I
>> start
>> > > addressing this? Here is the stack trace.
>> > >
>> > > Too many connections from /10.30.84.207 - max is 10
>> > > WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
>> > > unexpected error, closing socket connection and attempting reconnect
>> > > java.io.IOException: Connection reset by peer
>> > > at sun.nio.ch.FileDispatcher.read0(Native Method)
>> > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>> > > at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>> > > at
>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> > > at
>> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
>> > > at
>> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
>> > >
>> > > Please advice.
>> > >
>> > > Cheers
>> > > Avinash
>> > >
>> >
>>
>
>
