Mohammad Arshad created ZOOKEEPER-3496:
------------------------------------------
Summary: Transaction larger than jute.maxbuffer makes ZooKeeper
unavailable
Key: ZOOKEEPER-3496
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
Project: ZooKeeper
Issue Type: Bug
Affects Versions: 3.4.14, 3.5.5
Reporter: Mohammad Arshad
Assignee: Mohammad Arshad
Fix For: 3.6.0, 3.4.15, 3.5.6
*Problem:*
ZooKeeper server fails to start, logs following error
{code:java}
Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
at
org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
at
org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
{code}
This indicates that one of the transactions size is more than the configured
jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
allowed to write?
*Analysis:*
At ZooKeeper server jute.maxbuffer specifies the maximum size of a transaction.
By default it is 1 MB at the server
jute.maxbuffer is used for following:
# Size sanity check of incoming request. Incoming requests size must not be
more than jute.maxbuffer
# Size sanity check of the transaction while reading from transaction or
snapshot file. Transaction size must not be more than jute.maxbuffer+1024
# Size sanity check of transaction while reading data from the leader.
Transaction size must not be more than jute.maxbuffer+1024
Request size sanity check is done in the beginning of a request processing but
later request processing adds additional information into request then writes
to transaction file. This additional information size is not considered in
sanity check. This is how transaction larger than jute.maxbuffer are accepted
into ZooKeeper.
If this additional information size is less than 1024 Bytes then it is OK as
ZooKeeper already takes care of it.
But if this additional information size is more than 1024 bytes it allows the
request, But while reading from transaction/snapshot file and while reading
from leader it fails and make the ZooKeeper service unavailable
+Example:+
Suppose incoming request size is 1000000 Bytes
Configured jute.maxbuffer is 1000000
After processing the request ZooKeeper server adds 1025 more bytes
In this case, request will be processed successfully, and 1000000+1025 bytes
will be written to transaction file
But while reading from the transaction log 1000000+1025 bytes cannot be read as
max allowed length is 1000000(effectively 1000000+1024).
*Solutions:*
If incoming request size sanity check is done after populating all additional
information then this problem is solved. But doing sanity check in the later
stage of request processing will defeat the purpose of sanity check itself. So
this we can not do
Currently additional information size is constant 1024 Bytes [Code
Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
We should increase this value and make it more reasonable. I propose to make
this additional information size to same as the jute.maxbuffer. Also make
additional information size configurable.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
