We are using
zookeeper-3.4.6
more zookeeper logs here:
2014-11-14 09:42:28,233 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for
client /10.93.83.43:43487 which had sessionid 0x149a4cc1b581b60
2014-11-14 09:42:28,234 [myid:1] - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x149a4cc1b581b5f, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:780)
2014-11-14 09:42:28,234 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for
client /10.93.83.43:43486 which had sessionid 0x149a4cc1b581b5f
2014-11-14 09:42:48,689 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@197] - Accepted socket connection
from /10.93.83.50:46935
2014-11-14 09:42:49,280 [myid:1] - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:ZooKeeperServer@822] - Connection request from old
client /10.93.83.50:46935; will be dropped if server is in r-o mode
2014-11-14 09:42:49,280 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:ZooKeeperServer@868] - Client attempting to establish
new session at /10.93.83.50:46935
2014-11-14 09:42:49,311 [myid:1] - INFO
[CommitProcessor:1:ZooKeeperServer@617] - Established session
0x149a4cc1b581b62 with negotiated timeout 40000 for client /
10.93.83.50:46935
2014-11-14 09:45:19,746 [myid:1] - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x149a4cc1b581b61, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:780)
2014-11-14 09:45:19,747 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for
client /10.93.83.50:46331 which had sessionid 0x149a4cc1b581b61
2014-11-14 09:47:25,995 [myid:1] - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x149a4cc1b581b5b, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:780)
2014-11-14 09:47:25,996 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for
client /10.93.83.50:44094 which had sessionid 0x149a4cc1b581b5b
Chen
On Thu, Nov 13, 2014 at 5:25 PM, Jun Rao <jun...@gmail.com> wrote:
> Which version of ZK are you using?
>
> Thanks,
>
> Jun
>
> On Thu, Nov 13, 2014 at 10:15 AM, Chen Wang <chen.apache.s...@gmail.com>
> wrote:
>
> > Thanks for the info.
> > It makes sense, however, I didn't see any "session timeout"/"expired"
> > entries in consumer log..
> > but do see lots of connection closed entry in zookeeper log:
> >
> > 2014-11-13 10:07:53,132 [myid:1] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for
> > client /10.93.83.50:37180 which had sessionid 0x149a4cc1b580e7d
> > 2014-11-13 10:08:04,499 [myid:1] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@197] - Accepted socket
> > connection
> > from /10.93.80.121:38437
> > 2014-11-13 10:08:04,503 [myid:1] - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2182:ZooKeeperServer@822] - Connection request from old
> > client /10.93.80.121:38437; will be dropped if server is in r-o mode
> > 2014-11-13 10:08:04,503 [myid:1] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2182:ZooKeeperServer@868] - Client attempting to
> establish
> > new session at /10.93.80.121:38437
> > 2014-11-13 10:08:04,538 [myid:1] - INFO
> > [CommitProcessor:1:ZooKeeperServer@617] - Established session
> > 0x149a4cc1b580e7e with negotiated timeout 40000 for client /
> > 10.93.80.121:38437
> > 2014-11-13 10:08:08,746 [myid:1] - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for
> > client /10.93.80.121:38437 which had sessionid 0x149a4cc1b580e7e
> >
> > We are using -Xmx2048m for consumer, and I didn't see any GC related
> > exceptions
> >
> > Chen
> >
> >
> >
> > On Thu, Nov 13, 2014 at 9:13 AM, Guozhang Wang <wangg...@gmail.com>
> wrote:
> >
> > > Hey Chen,
> > >
> > > As Neha suggested, typical reason of too many rebalances is that your
> > > consumers kept being timed out from ZK, and you can verify this by
> > checking
> > > in your consumer logs for sth. like "session timeout" entries (these
> are
> > > not ERROR entries).
> > >
> > > Guozhang
> > >
> > > Guozhang
> > >
> > > On Wed, Nov 12, 2014 at 5:31 PM, Neha Narkhede <
> neha.narkh...@gmail.com>
> > > wrote:
> > >
> > > > Does this help?
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > > > ?
> > > >
> > > > On Wed, Nov 12, 2014 at 3:53 PM, Chen Wang <
> chen.apache.s...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi there,
> > > > > My kafka client is reading a 3 partition topic from kafka with 3
> > > threads
> > > > > distributed on different machines. I am seeing frequent owner
> changes
> > > on
> > > > > the topics when running:
> > > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> > > > > my_test_group --topic mytopic -zkconnect localhost:2181
> > > > >
> > > > > The owner kept changing once a while, but I didn't see any
> exceptions
> > > > > thrown from the consumer side. When checking broker log, its full
> of
> > > > > INFO Closing socket connection to /IP. (kafka.network.Processor)
> > > > >
> > > > > Is this expected behavior? If so, how can I tell when the leader
> is
> > > > > imbalanced, and rebalance is triggered?
> > > > > Thanks,
> > > > > Chen
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
