I am using Kazoo client and see the following error:
Aug 9 07:07:20 a2s1 python[2085]: File
"/usr/lib/python2.7/site-packages/kazoo/client.py", line 894, in set
Aug 9 07:07:20 a2s1 python[2085]: return self.set_async(path, data,
version).get()
Aug 9 07:07:20 a2s1 python[2085]: File
"/usr/lib64/python2.7/site-packages/gevent/event.py", line 223, in get
Aug 9 07:07:20 a2s1 python[2085]: raise self._exception
Aug 9 07:07:20 a2s1 python[2085]: OperationTimeoutException: operation timeout
zookeeper logs around the error time:
2013-08-09 07:07:06,580 [myid:] - WARN [SyncThread:0:FileTxnLog@321] -
fsync-ing the write ahead log in SyncThread:0 took 2291ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
…
2013-08-09 07:07:19,827 [myid:] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of
stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x1406356c3bc0001, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:19,828 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1000] - Closed socket
connection for client /127.0.0.1:54698 which had sessionid 0x1406356c3bc0001
2013-08-09 07:07:20,246 [myid:] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of
stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x1406356c3bc0000, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:722)
2013-08-09 07:07:20,246 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1000] - Closed socket
connection for client /127.0.0.1:54579 which had sessionid 0x1406356c3bc0000
2013-08-09 07:07:21,583 [myid:] - WARN [SyncThread:0:FileTxnLog@321] -
fsync-ing the write ahead log in SyncThread:0 took 8001ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:21,680 [myid:] - ERROR [SyncThread:0:NIOServerCnxn@180] -
Unexpected Exception:
java.nio.channels.CancelledKeyException
[root@a2s1 zookeeper]# grep fsync /var/log/zookeeper/zookeeper.out
2013-08-09 07:06:52,280 [myid:] - WARN [SyncThread:0:FileTxnLog@321] -
fsync-ing the write ahead log in SyncThread:0 took 1724ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:06:58,315 [myid:] - WARN [SyncThread:0:FileTxnLog@321] -
fsync-ing the write ahead log in SyncThread:0 took 2378ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:01,389 [myid:] - WARN [SyncThread:0:FileTxnLog@321] -
fsync-ing the write ahead log in SyncThread:0 took 1113ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:06,580 [myid:] - WARN [SyncThread:0:FileTxnLog@321] -
fsync-ing the write ahead log in SyncThread:0 took 2291ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
2013-08-09 07:07:21,583 [myid:] - WARN [SyncThread:0:FileTxnLog@321] -
fsync-ing the write ahead log in SyncThread:0 took 8001ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
[
Could the client (Kazoo) be timing out because of fsync delay? What parameter
would control duration for OperationTimeoutException that I can perhaps
increase to verify? There is only ZooKeeper client and the load isn't much - 1
read/sec and 2 writes/sec roughly. Zookeeper configuration is default. Kazoo
client params are also default.
Thanks
Deepinder
