|
Dear all,
I am a new user of storm. I am writing a proof of concept with a very simple topology.
I am using the local mode. When I run my application during 5 minutes, it is OK. But I am not able to run it during hours. Indeed, it always fails and stops with the following error (I enclosed the full stack strace):
WARN org.apache.storm.zookeeper.server.persistence.FileTxnLog - fsync-ing the write ahead log in SyncThread:0 took 4702ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
(...)
org.apache.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x14a526c861a0003, likely client has closed socket
(...)
java.nio.channels.CancelledKeyException: null
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
(...)
There is a reference to ZooKeeper troubleshooting guide. But I do not find any solution/workaround in it.
It looks like a problem in the logger of ZooKeeper. Is-it possible de disable it?
Any help is welcome.
Denis DEBARBIEUX
norsysgroupe |
729284 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4086ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
1062238 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4387ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
1218137 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4525ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
2319825 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5151ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
2560025 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5099ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
2718738 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read
additional data from client sessionid 0x14a526c861a0001, likely client has
closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
2718797 [xx.sandbox.demo.Main.main()-EventThread] INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
2718798 [ConnectionStateManager-0] WARN
org.apache.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
2718821 [xx.sandbox.demo.Main.main()-EventThread] WARN backtype.storm.cluster
- Received event :disconnected::none: with disconnected Zookeeper.
2723949 [xx.sandbox.demo.Main.main()-EventThread] INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
RECONNECTED
2723951 [ConnectionStateManager-0] WARN
org.apache.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
2966611 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4737ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
3127154 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4980ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
3877848 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4773ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
4440014 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4899ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
4563468 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5352ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
4691353 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5655ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
5235255 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5138ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
5406851 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5232ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
5695086 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4708ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
5880302 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5144ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6081737 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4911ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6179087 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4989ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6184969 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5791ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6195729 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5272ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6266091 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5018ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6381007 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5275ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6386987 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5932ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6401795 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4904ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6411925 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5081ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6509602 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4995ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6565154 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4796ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6586246 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5780ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6596480 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5149ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6636575 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5097ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6651415 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4823ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6686454 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4936ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6697348 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5919ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6733126 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5019ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6773542 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4924ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6809180 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4913ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6884528 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4996ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6900001 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4747ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6960289 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5012ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6975278 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5183ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6985015 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4872ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6989911 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4869ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
6995142 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5219ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7010804 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5026ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7021278 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4950ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7031999 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5513ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7037150 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5145ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7041923 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read
additional data from client sessionid 0x14a526c861a0003, likely client has
closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
7041925 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read
additional data from client sessionid 0x14a526c861a0005, likely client has
closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
7041942 [SyncThread:0] ERROR org.apache.zookeeper.server.NIOServerCnxn -
Unexpected Exception:
java.nio.channels.CancelledKeyException: null
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
~[na:1.7.0_72]
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
~[na:1.7.0_72]
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
[zookeeper-3.4.5.jar:3.4.5-1392090]
7041957 [SyncThread:0] ERROR org.apache.zookeeper.server.NIOServerCnxn -
Unexpected Exception:
java.nio.channels.CancelledKeyException: null
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
~[na:1.7.0_72]
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
~[na:1.7.0_72]
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)
[zookeeper-3.4.5.jar:3.4.5-1392090]
7047163 [xx.sandbox.demo.Main.main()-EventThread] INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
7047164 [xx.sandbox.demo.Main.main()-EventThread] INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
SUSPENDED
7047164 [xx.sandbox.demo.Main.main()-EventThread] WARN backtype.storm.cluster
- Received event :disconnected::none: with disconnected Zookeeper.
7047164 [ConnectionStateManager-0] WARN
org.apache.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
7047164 [ConnectionStateManager-0] WARN
org.apache.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
7047167 [xx.sandbox.demo.Main.main()-EventThread] WARN backtype.storm.cluster
- Received event :disconnected::none: with disconnected Zookeeper.
7052301 [xx.sandbox.demo.Main.main()-EventThread] INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
RECONNECTED
7052302 [ConnectionStateManager-0] WARN
org.apache.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
7057810 [xx.sandbox.demo.Main.main()-EventThread] INFO
org.apache.curator.framework.state.ConnectionStateManager - State change:
RECONNECTED
7057810 [ConnectionStateManager-0] WARN
org.apache.curator.framework.state.ConnectionStateManager - There are no
ConnectionStateListeners registered.
7057817 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5452ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7115617 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5430ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7147510 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5248ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7152446 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4920ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7173076 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 4873ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7193550 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5077ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7240456 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5089ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7245975 [SyncThread:0] WARN org.apache.zookeeper.server.persistence.FileTxnLog
- fsync-ing the write ahead log in SyncThread:0 took 5471ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting guide
7251050 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN
org.apache.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read
additional data from client sessionid 0x14a526c861a0003, likely client has
closed socket
