I think the "unreasonable length" in your case means that the integer
describing the size of the next transaction event contains an "unreasonable
large" value (2186882 bytes), so I would assume this is a corrupted
transaction log file.
If this is a distributed cluster, then you should be able to restore the
failing node by copying the data from an other server (e.g. from the
current leader). Make sure you copy both the latest snapshot, plus all the
transaction logs written since the start of the last snapshot creation. (it
never hurts if you copy more files, ZooKeeper will only use the files that
are necessary)
Kind regards,
Máté
On Tue, Dec 14, 2021 at 5:15 PM Joe (Joseph) Marrero <jmarr...@drfirst.com>
wrote:
> Hi.
>
> Yesterday, our ZooKeeper instances failed to recover and we saw this error
> in the logs:
>
>
> 2021-12-14 03:01:43,296 [myid:2] - ERROR [main:QuorumPeer@1139] - Unable
> to load database on disk
>
> java.io.IOException: Unreasonable length = 2186882
>
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
>
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
>
> at
> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:768)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:352)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:258)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:303)
>
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:285)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1093)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1078)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:227)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:136)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:90)
>
> 2021-12-14 03:01:43,304 [myid:2] - INFO [main:AbstractConnector@380] -
> Stopped ServerConnector@7526515b{HTTP/1.1,[http/1.1]}{0.0.0.0:9141}
>
> 2021-12-14 03:01:43,305 [myid:2] - INFO [main:ContextHandler@1016] -
> Stopped o.e.j.s.ServletContextHandler@2f953efd{/,null,UNAVAILABLE}
>
> 2021-12-14 03:01:43,306 [myid:2] - ERROR [main:QuorumPeerMain@113] -
> Unexpected exception, exiting abnormally
>
> java.lang.RuntimeException: Unable to run quorum server
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1140)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1078)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:227)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:136)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:90)
>
> Caused by: java.io.IOException: Unreasonable length = 2186882
>
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:166)
>
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:127)
>
> at
> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:159)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:768)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:352)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:258)
>
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:303)
>
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:285)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1093)
>
> ... 4 more
>
>
> The hard disk was at 50% capacity when this failure occurred. Any ideas
> what would cause this kind of unrecoverable error?
>
>
>
> *[image: DrFirst Logo]*
>
> *Joe Marrero*
>
> Senior Software Engineer
>
> jmarr...@drfirst.com <jdel...@drfirst.com>
>
> DrFirst.com <http://www.drfirst.com/> | Twitter
> <https://twitter.com/drfirst> | Facebook
> <http://www.facebook.com/DrFirstInc> | LinkedIn
> <https://www.linkedin.com/company/drfirst/>
>
> [image: HIMSS Banner]
>
>
> Notice of Confidentiality: The information included and/or attached in
> this electronic mail transmission may contain confidential or privileged
> information and is intended for the addressee. Any unauthorized disclosure,
> reproduction, distribution or the taking of action in reliance on the
> contents of the information is prohibited. If you believe that you have
> received the message in error, please notify the sender by reply
> transmission and delete the message without copying or disclosing it.
>
