[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938471#comment-16938471
]
Hudson commented on ZOOKEEPER-3496:
---
SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #711 (See
[https://builds.apache.org/job/ZooKeeper-trunk/711/])
ZOOKEEPER-3496: Transaction larger than jute.maxbuffer makes ZooKeeper
(enrico.olivelli: rev 4279758ead655aff34cdff21c9f2c71d66030d14)
* (edit)
zookeeper-jute/src/test/java/org/apache/jute/BinaryInputArchiveTest.java
* (edit) zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java
* (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.5.7
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938401#comment-16938401
]
Hudson commented on ZOOKEEPER-3496:
---
SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #548 (See
[https://builds.apache.org/job/Zookeeper-trunk-single-thread/548/])
ZOOKEEPER-3496: Transaction larger than jute.maxbuffer makes ZooKeeper
(enrico.olivelli: rev 4279758ead655aff34cdff21c9f2c71d66030d14)
* (edit)
zookeeper-jute/src/test/java/org/apache/jute/BinaryInputArchiveTest.java
* (edit) zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java
* (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.5.7
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938373#comment-16938373
]
Mohammad Arshad commented on ZOOKEEPER-3496:
Thanks [~eolivelli], [~hanm], [~lvfangmin] for the reviews.
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.5.7
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938371#comment-16938371
]
Mohammad Arshad commented on ZOOKEEPER-3496:
[~hanm] had committed to branch-3.5. This issue is committed to branch-3.5 and
master
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.5.7
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937045#comment-16937045
]
Hadoop QA commented on ZOOKEEPER-3496:
--
+1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version )
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4152//console
This message is automatically generated.
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.4.15, 3.5.7
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936131#comment-16936131
]
Hadoop QA commented on ZOOKEEPER-3496:
--
+1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version )
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4150//console
This message is automatically generated.
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.4.15, 3.5.7
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934358#comment-16934358
]
Hadoop QA commented on ZOOKEEPER-3496:
--
+1 overall. GitHub Pull Request Build
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version )
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/4148//console
This message is automatically generated.
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.4.15, 3.5.7
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923151#comment-16923151
]
Mohammad Arshad commented on ZOOKEEPER-3496:
PR Created
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.6.0, 3.4.15, 3.5.6
>
> Attachments: ZOOKEEPER-3496-001.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (ZOOKEEPER-3496) Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922851#comment-16922851
]
Fangmin Lv commented on ZOOKEEPER-3496:
---
[~arshad.mohammad] thanks for reporting the issue and providing the patch, it
looks good to me, do you mind to create a PR for this.
> Transaction larger than jute.maxbuffer makes ZooKeeper unavailable
> --
>
> Key: ZOOKEEPER-3496
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3496
> Project: ZooKeeper
> Issue Type: Bug
>Affects Versions: 3.5.5, 3.4.14
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Critical
> Fix For: 3.6.0, 3.4.15, 3.5.6
>
> Attachments: ZOOKEEPER-3496-001.patch
>
>
> *Problem:*
> ZooKeeper server fails to start, logs following error
> {code:java}
> Exception in thread "main" java.io.IOException: Unreasonable length = 1001025
> at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
> at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
> {code}
> This indicates that one of the transactions size is more than the configured
> jute.maxbuffer values. But how transaction more than jute.maxbuffer size is
> allowed to write?
> *Analysis:*
> At ZooKeeper server jute.maxbuffer specifies the maximum size of a
> transaction. By default it is 1 MB at the server
> jute.maxbuffer is used for following:
> # Size sanity check of incoming request. Incoming requests size must not be
> more than jute.maxbuffer
> # Size sanity check of the transaction while reading from transaction or
> snapshot file. Transaction size must not be more than jute.maxbuffer+1024
> # Size sanity check of transaction while reading data from the leader.
> Transaction size must not be more than jute.maxbuffer+1024
> Request size sanity check is done in the beginning of a request processing
> but later request processing adds additional information into request then
> writes to transaction file. This additional information size is not
> considered in sanity check. This is how transaction larger than
> jute.maxbuffer are accepted into ZooKeeper.
> If this additional information size is less than 1024 Bytes then it is OK as
> ZooKeeper already takes care of it.
> But if this additional information size is more than 1024 bytes it allows the
> request, But while reading from transaction/snapshot file and while reading
> from leader it fails and make the ZooKeeper service unavailable
> +Example:+
> Suppose incoming request size is 100 Bytes
> Configured jute.maxbuffer is 100
> After processing the request ZooKeeper server adds 1025 more bytes
> In this case, request will be processed successfully, and 100+1025 bytes
> will be written to transaction file
> But while reading from the transaction log 100+1025 bytes cannot be read
> as max allowed length is 100(effectively 100+1024).
> *Solutions:*
> If incoming request size sanity check is done after populating all additional
> information then this problem is solved. But doing sanity check in the later
> stage of request processing will defeat the purpose of sanity check itself.
> So this we can not do
> Currently additional information size is constant 1024 Bytes [Code
> Reference|https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-jute/src/main/java/org/apache/jute/BinaryInputArchive.java#L126].
> We should increase this value and make it more reasonable. I propose to make
> this additional information size to same as the jute.maxbuffer. Also make
> additional information size configurable.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
