[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254228#comment-16254228 ] Hudson commented on ZOOKEEPER-2574: --- FAILURE: Integrated in Jenkins build ZooKeeper-trunk #3609 (See [https://builds.apache.org/job/ZooKeeper-trunk/3609/]) ZOOKEEPER-2690: Update documentation source for ZOOKEEPER-2574 (phunt: rev efeebd0a1ba7f6be3b4ffe262ab4045656bb1eb3) * (edit) docs/index.pdf * (edit) docs/zookeeperOver.pdf * (edit) docs/zookeeperStarted.pdf * (edit) docs/linkmap.pdf * (edit) docs/zookeeperJMX.pdf * (edit) docs/zookeeperHierarchicalQuorums.pdf * (edit) src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml * (edit) docs/javaExample.pdf * (edit) docs/zookeeperProgrammers.pdf * (edit) docs/zookeeperTutorial.pdf * (edit) docs/zookeeperObservers.pdf * (edit) docs/zookeeperAdmin.html * (edit) docs/zookeeperReconfig.pdf * (edit) docs/zookeeperInternals.pdf * (edit) docs/zookeeperQuotas.pdf * (edit) docs/recipes.pdf * (edit) docs/zookeeperAdmin.pdf > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218935#comment-16218935 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1119//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1119//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/1119//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860051#comment-15860051 ] Michael Han commented on ZOOKEEPER-2574: Good catch [~arshad.mohammad] - created ZOOKEEPER-2690 to track the work. Obviously I missed this during review. Mark it as minor priority because the doc change was cosmetic. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859266#comment-15859266 ] Mohammad Arshad commented on ZOOKEEPER-2574: Document changes are done in generated html, docs/zookeeperAdmin.html. When the document is generated again these changes will be overridden. Changes should have been done in src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml. May be we can raise new jira an port the changes > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833883#comment-15833883 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- Github user abhishekrai commented on the issue: https://github.com/apache/zookeeper/pull/111 Thanks @hanm! > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833799#comment-15833799 ] Hudson commented on ZOOKEEPER-2574: --- FAILURE: Integrated in Jenkins build ZooKeeper-trunk #3247 (See [https://builds.apache.org/job/ZooKeeper-trunk/3247/]) ZOOKEEPER-2574: PurgeTxnLog can inadvertently delete required txn log (hanm: rev 762f4af65bb1056a582a6f36183a9e28fe0ccab8) * (edit) src/java/main/org/apache/zookeeper/server/persistence/FileSnap.java * (edit) src/java/main/org/apache/zookeeper/server/PurgeTxnLog.java * (edit) docs/zookeeperAdmin.html * (edit) src/java/main/org/apache/zookeeper/server/persistence/FileTxnSnapLog.java * (edit) src/java/test/org/apache/zookeeper/server/PurgeTxnTest.java > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833792#comment-15833792 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/111 Merged to 3.4, 3.5, and master. Thanks for your contribution @abhishekrai ! > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833791#comment-15833791 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12833002/ZOOKEEPER-2574.6.patch against trunk revision 762f4af65bb1056a582a6f36183a9e28fe0ccab8. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3565//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3, 3.6.0 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833782#comment-15833782 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/111 > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745037#comment-15745037 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/113//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/113//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/113//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745012#comment-15745012 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- Github user abhishekrai commented on the issue: https://github.com/apache/zookeeper/pull/111 Thanks for the review, I've updated the commit. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745011#comment-15745011
]
ASF GitHub Bot commented on ZOOKEEPER-2574:
---
Github user abhishekrai commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/111#discussion_r92156972
--- Diff: src/java/main/org/apache/zookeeper/server/PurgeTxnLog.java ---
@@ -108,9 +141,11 @@ public boolean accept(File f){
// remove the old files
for(File f: files)
{
-System.out.println("Removing file: "+
+final String msg = "Removing file: "+
DateFormat.getDateTimeInstance().format(f.lastModified())+
-"\t"+f.getPath());
+"\t"+f.getPath();
+LOG.info(msg);
+System.out.println(msg);
--- End diff --
It's not ideal but they both serve a purpose that the other cannot as far
as I can tell.
System.out.println is useful when this is invoked directly through CLI.
The old behavior was to provide just this.
LOG.info is useful in that it's visible in the server log. The old
behavior did not log this info which makes debugging through server logs harder.
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch,
> ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch,
> ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745007#comment-15745007 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- Github user abhishekrai commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/111#discussion_r92156748 --- Diff: docs/zookeeperAdmin.html --- @@ -1201,8 +1205,10 @@ Configuration Parameters (Java system property: zookeeper.snapCount) ZooKeeper logs transactions to a transaction log. After snapCount transactions are written to a log - file a snapshot is started and a new transaction log - file is created. The default snapCount is + file a snapshot is started. It also influences rollover + of the current transaction log to a new file. However, --- End diff -- Done, thanks. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701055#comment-15701055 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- Github user lvfangmin commented on a diff in the pull request: https://github.com/apache/zookeeper/pull/111#discussion_r89721901 --- Diff: docs/zookeeperAdmin.html --- @@ -1201,8 +1205,10 @@ Configuration Parameters (Java system property: zookeeper.snapCount) ZooKeeper logs transactions to a transaction log. After snapCount transactions are written to a log - file a snapshot is started and a new transaction log - file is created. The default snapCount is + file a snapshot is started. It also influences rollover + of the current transaction log to a new file. However, --- End diff -- keep indention? > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701054#comment-15701054
]
ASF GitHub Bot commented on ZOOKEEPER-2574:
---
Github user lvfangmin commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/111#discussion_r89722055
--- Diff: src/java/main/org/apache/zookeeper/server/PurgeTxnLog.java ---
@@ -108,9 +141,11 @@ public boolean accept(File f){
// remove the old files
for(File f: files)
{
-System.out.println("Removing file: "+
+final String msg = "Removing file: "+
DateFormat.getDateTimeInstance().format(f.lastModified())+
-"\t"+f.getPath());
+"\t"+f.getPath();
+LOG.info(msg);
+System.out.println(msg);
--- End diff --
do we need to keep both system.out and log4j logging?
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch,
> ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch,
> ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701053#comment-15701053 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- Github user lvfangmin commented on the issue: https://github.com/apache/zookeeper/pull/111 minor comments, others look good to me. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679498#comment-15679498 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. GitHub Pull Request Build +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 20 new Findbugs (version 3.0.1) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/82//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/82//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/82//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679486#comment-15679486 ] Abhishek Rai commented on ZOOKEEPER-2574: - [~hanm] and [~rakeshr], thanks for finding the relation to ZOOKEEPER-2420 and thanks for your guidance. I've created a pull request as per your suggestion with the following changes: (1) Patch previously uploaded containing fix and tests. (2) Tests from ZOOKEEPER-2420 and enabling code. (3) Documentation fixes. [~rakeshr] great call on documentation review, as I went through it I found multiple inconsistencies about the snapshot-log dependency. I've fixed all that I could find in the docs/ directory. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679478#comment-15679478 ] ASF GitHub Bot commented on ZOOKEEPER-2574: --- GitHub user abhishekrai opened a pull request: https://github.com/apache/zookeeper/pull/111 ZOOKEEPER-2574: PurgeTxnLog can inadvertently delete required txn log files … files This fix includes patch from Ed Rowe for ZOOKEEPER-2420, which is the same issue as ZOOKEEPER-2574. You can merge this pull request into a Git repository by running: $ git pull https://github.com/abhishekrai/zookeeper ZOOKEEPER-2574 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zookeeper/pull/111.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #111 commit 4bc4a77800c25ab5bcdaf1149c28b1912d29064f Author: Abhishek Rai Date: 2016-11-18T18:42:51Z ZOOKEEPER-2574: PurgeTxnLog can inadvertently delete required txn log files This fix includes patch from Ed Rowe for ZOOKEEPER-2420, which is the same issue as ZOOKEEPER-2574. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675825#comment-15675825 ] Rakesh R commented on ZOOKEEPER-2574: - Yes, I've identified few occurrences in the doc. One more [comment|https://issues.apache.org/jira/browse/ZOOKEEPER-2574?focusedCommentId=15643171&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15643171] is open and has to be fixed. Probably, we could do more detailed reviews and all such cases has to be re-phrased as there was a behavioral change due to the [Learner snapshotting logic|https://github.com/apache/zookeeper/blob/branch-3.4/src/java/main/org/apache/zookeeper/server/quorum/Learner.java#L439]. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675807#comment-15675807 ] Rakesh R commented on ZOOKEEPER-2574: - How about keeping only one jira open and close the other one. That way, we will try to converge the discussions into one place. I'm OK to close any one of these two jira issues:-) Could you please look at [my previous comment|https://issues.apache.org/jira/browse/ZOOKEEPER-2574?focusedCommentId=15662873&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15662873] and do necessary src & test code merge from ZK-2420. Appreciate creating a pull request for the same. Thanks! > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668508#comment-15668508 ] Michael Han commented on ZOOKEEPER-2574: I am also changing priority of this from Blocker to Major per discussions in ZOOKEEPER-2420. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668505#comment-15668505 ] Michael Han commented on ZOOKEEPER-2574: Hi [~rakeshr]: bq. we could club these two patches together to make a good solution. Sounds good! Also, a late LGTM on the patch attached to this JIRA after incorporating tests from ZOOKEEPER-2420. bq. I think, this would be a serious concern in the view of disaster recovery scripts. If someone has written a script blindly following that "A new log file is started each time a snapshot is begun". Would the doc fix proposed in this patch address this concern? With the new doc it is clearly stated that the prior one transaction log should be retained / required. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15662873#comment-15662873 ] Rakesh R commented on ZOOKEEPER-2574: - Good catch [~hanm]. Yes, both are addressing same problem. Thanks a lot [~EdRowe] for the good number of unit test cases in your patch and working on this major issue. After looking at ZOOKEEPER-2420 patch, it looks to me that the proposed fix in ZOOKEEPER-2574 jira is simple and with less code complexity. Just a suggestion from me, we could club these two patches together to make a good solution. (1) Merge the test cases and (2) FileSnap#findNRecentSnapshots() changes from ZOOKEEPER-2420 patch to this jira proposal patch. Does this makes sense to you? Also, I'd like to re-iterate some of the discussions regarding ZK docs made earlier in this jira [comment1|https://issues.apache.org/jira/browse/ZOOKEEPER-2574?focusedCommentId=15483048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15483048], [comment2|https://issues.apache.org/jira/browse/ZOOKEEPER-2574?focusedCommentId=15643171&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15643171]. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655700#comment-15655700 ] Michael Han commented on ZOOKEEPER-2574: This JIRA sounds like a duplicate of ZOOKEEPER-2420. [~abhishekrai]: Could you please take a look at ZOOKEEPER-2420 and confirm if it's solving the same problem as this JIRA? If so, we probably should remove this issue from release blockers as the consensus so far in ZOOKEEPER-2420 is that the issue is better to be fixed but we will be fine live with it for a while, given no real issues being reported from prod (with reasons stated in ZOOKEEPER-2420) so far. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643171#comment-15643171
]
Rakesh R commented on ZOOKEEPER-2574:
-
Thank you [~abhishekrai]. I could see there are few more corrections required
in the
[zookeeperAdmin.html#The+Log+Directory|https://zookeeper.apache.org/doc/r3.4.9/zookeeperAdmin.html#The+Log+Directory],
right?. I'd suggest you to read the {{ZooKeeper transaction log and snapshot}}
related sections in the ZK docs and do necessary changes. Appreciate your time
& efforts.
{code}
The Log Directory:
A new log file is started each time a snapshot is begun. The log file's suffix
is the first zxid written to that log
{code}
Could you please create a pull request for the proposed patch, that will be
used for code reviews and commits.
Hi [~fpj], [~phunt], [~rgs]. As part of this jira we have came across a
situation {{"where snapshotting has happened multiple times without
accompanying log rollover"}}, this is contradicting with the ZooKeeper docs. I
think, this would be a serious concern in the view of disaster recovery
scripts. If someone has written a script blindly following that "A new log file
is started each time a snapshot is begun". It would be really helpful if you
could pitch in and give your thoughts on this problem. Thanks!
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch,
> ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch,
> ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570304#comment-15570304 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12833002/ZOOKEEPER-2574.6.patch against trunk revision f78061aafb19b102c37cb6d744ec6258d5f5b66e. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3488//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3488//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3488//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.6.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568287#comment-15568287
]
Rakesh R commented on ZOOKEEPER-2574:
-
Should we update the [zk
docs|https://zookeeper.apache.org/doc/r3.4.9/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup]
part, {code}As changes are made to the znodes these changes are appended to a
transaction log, occasionally, when a log grows large, a snapshot of the
current state of all znodes will be written to the filesystem. This snapshot
supercedes all previous logs{code} It looks like conveying wrong behavior,
right?
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch,
> ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505152#comment-15505152 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12829284/ZOOKEEPER-2574.5.patch against trunk revision ec20c5434cc8a334b3fd25e27d26dccf4793c8f3. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3443//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3443//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3443//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.5.patch, ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504796#comment-15504796
]
Abraham Fine commented on ZOOKEEPER-2574:
-
> I switched logsToPurge from a List to an ArrayList so I can
> simply use remove(0) to remove the first element in the list on line 239
I think I must be missing something as all of the lists are ArrayLists. For
example, this still passes:
{code:java}
List snapsToPurge = new ArrayList();
List snaps = new ArrayList();
List logs = new ArrayList();
List snapsAboveRecentFiles = new ArrayList();
List logsAboveRecentFiles = new ArrayList();
List logsToPurge = new ArrayList();
{code}
> However, I agree that the logic would be simpler if all filtering logic is in
> one place, in MyFileFilter.accept(). I've moved it there now.
Agreed
> Is there a way to achieve both goals, logging and console output (preferably
> stdout) without any duplication.
I'm not sure, perhaps system.err?
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch,
> ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15501456#comment-15501456 ] Hadoop QA commented on ZOOKEEPER-2574: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12829119/ZOOKEEPER-2574.4.patch against trunk revision b2a484cfe743116d2531fe5d1e1d78b3960c511e. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3441//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3441//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3441//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.4.patch, ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494661#comment-15494661 ] Abraham Fine commented on ZOOKEEPER-2574: - A couple minor questions: PurgeTxnTest.java:224 Can we change ArrayList logsToPurge back to List logsToPurge? PurgeTxnLog.java:138 Do we need to use the FileFilter here since we do "filtering" on line 142? PurgeTxnLog.java:148 We do logging and System.out.println for the same String, do we need both? > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486135#comment-15486135 ] Hadoop QA commented on ZOOKEEPER-2574: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827931/ZOOKEEPER-2574.3.patch against trunk revision b2a484cfe743116d2531fe5d1e1d78b3960c511e. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3422//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3422//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3422//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485706#comment-15485706 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827931/ZOOKEEPER-2574.3.patch against trunk revision 8c4082647f89b0a92fa00a2af8de84b3c7314e23. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3419//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3419//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3419//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485618#comment-15485618 ] Hadoop QA commented on ZOOKEEPER-2574: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827931/ZOOKEEPER-2574.3.patch against trunk revision 8c4082647f89b0a92fa00a2af8de84b3c7314e23. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3416//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3416//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3416//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483721#comment-15483721 ] Abhishek Rai commented on ZOOKEEPER-2574: - Thanks for the references [~rakeshr]. The learner writes the snapshot in response to the NEWLEADER message received from the leader. Based on my understanding, this is because the leader could be ahead of the learner - meaning that the learner is missing some transactions that the leader has. So receiving a snapshot and committing it locally is a valid option for the learner to catch up and join the quorum. However, going forward it will receive subsequent transactions, which as you mentioned get appended to the existing txn log file. It seems a log rollover could have been done before snapshotting in the learner, but perhaps changing behaviors at this point is not worth it given the need to support old behavior too? > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483048#comment-15483048
]
Rakesh R commented on ZOOKEEPER-2574:
-
Good catch, [~abhishekrai]. Thanks for the proposed patch with good unit
testing.
I agree the following scenario will occur in learner side, where snapshotting
has happened multiple times without accompanying log rollover, [refer source
code|https://github.com/apache/zookeeper/blob/branch-3.4/src/java/main/org/apache/zookeeper/server/quorum/Learner.java#L439].
I think there is mismatch in the zookeeper documentation, it says "This
snapshot supercedes all previous logs", [please refer zk
doc|https://zookeeper.apache.org/doc/r3.4.9/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup].
But in this particular case the latest transactions after snapshotting is gong
to the existing transaction log file, which is behind if we look at the
naming(zxid part). I'm not sure whether this is intentionally implemented?.
Again, iiuc, recovery will use a snapshot file + delta taken from the
transaction log file name after snapshotted zxid, but in this case admins need
to consider the log file just before, the snapshot file. Should we need to make
the snapshot file and log file consistent?
[~phunt], any thoughts?
{code}
2. Following files exist:
log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
snapshot.110 - snapshot as of zxid=110
snapshot.120 - snapshot as of zxid=120
snapshot.130 - snapshot as of zxid=130
{code}
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch,
> ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482179#comment-15482179 ] Arshad Mohammad commented on ZOOKEEPER-2574: Latest patch LGTM +1(non-binding) > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482129#comment-15482129 ] Hadoop QA commented on ZOOKEEPER-2574: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827931/ZOOKEEPER-2574.3.patch against trunk revision 1759917. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3414//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3414//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3414//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.3.patch, > ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481700#comment-15481700
]
Arshad Mohammad commented on ZOOKEEPER-2574:
Thanks [~abhishekrai] for working on this issue.
The solution looks good to me. I can see FileTxnLog.getLogFiles has the logic
to retain highest transaction log which is less than the leastZxidToBeRetain.
Few comments
1) can we rename exclude to retainedTxnLogs or some thing similar.
{code}
final Set exclude = new HashSet();
{code}
2) FileTxnSnapLog.getSnapshotLogs(long) javadoc does not have complete
information.
can you update it to indicate that it also returns "one snapshot log which is
highest but less than the given zxid"
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481246#comment-15481246 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827921/ZOOKEEPER-2574.2.patch against trunk revision 1759917. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3411//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3411//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3411//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481196#comment-15481196 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827920/ZOOKEEPER-2574.2.patch against trunk revision 1759917. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3410//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3410//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3410//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.2.patch, ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481163#comment-15481163 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827918/ZOOKEEPER-2574.patch against trunk revision 1759917. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3409//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3409//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3409//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481132#comment-15481132 ] Hadoop QA commented on ZOOKEEPER-2574: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12827917/ZOOKEEPER-2574.patch against trunk revision 1759917. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/3408//console This message is automatically generated. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Assignee: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > Attachments: ZOOKEEPER-2574.patch > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481129#comment-15481129
]
Abhishek Rai commented on ZOOKEEPER-2574:
-
Thanks [~phunt], I've uploaded a fix and unittest. Without the fix, the
unittest fails in the assertion below, thanks.
{noformat}
/**
* Verify that the last znode that was created above exists. This
znode's creation was
* captured by the transaction log which was created before any of the
above
* SNAP_RETAIN_COUNT snapshots were created, but it's not captured in
any of these
* snapshots. So for it it exist, the (only) existing log file should
not have been purged.
*/
final String lastZnode = "/snap-" + (unique - 1);
final Stat stat = zk.exists(lastZnode, false);
Assert.assertNotNull("Last znode does not exist: " + lastZnode, stat);
{noformat}
> PurgeTxnLog can inadvertently delete required txn log files
> ---
>
> Key: ZOOKEEPER-2574
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
> Environment: Zookeeper 3.4.8, standalone, and 3-server quorum
>Reporter: Abhishek Rai
>Assignee: Abhishek Rai
>Priority: Blocker
> Fix For: 3.4.10, 3.5.3
>
> Attachments: ZOOKEEPER-2574.patch
>
>
> As part of the fix for ZOOKEEPER-1797, the call to
> FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a
> result, some old-looking but required txn log files can be deleted, resulting
> in data corruption or loss.
> For example, consider the following:
> 1. Configuration:
> autopurge.snapRetainCount=3
> 2. Following files exist:
> log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
> snapshot.110 - snapshot as of zxid=110
> snapshot.120 - snapshot as of zxid=120
> snapshot.130 - snapshot as of zxid=130
> Above scenario is possible when snapshotting has happened multiple times but
> without accompanying log rollover, which is possible if the server was
> running as a learner.
> 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is
> older than the zxid of the oldest snapshot (110). This results in loss of
> transactions in the range 131-140.
> Before the fix for ZOOKEEPER-1797, this was avoided by the call to
> FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log
> file with starting zxid < oldest retained snapshot's highest zxid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15480686#comment-15480686 ] Patrick Hunt commented on ZOOKEEPER-2574: - FYI [~michim] and [~rakeshr] > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ZOOKEEPER-2574) PurgeTxnLog can inadvertently delete required txn log files
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15480684#comment-15480684 ] Patrick Hunt commented on ZOOKEEPER-2574: - [~abhishekrai] I've upped the priority to blocker. Any chance you could submit a test (or test and fix) that reproduces this issue? Thanks. > PurgeTxnLog can inadvertently delete required txn log files > --- > > Key: ZOOKEEPER-2574 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2574 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2 > Environment: Zookeeper 3.4.8, standalone, and 3-server quorum >Reporter: Abhishek Rai >Priority: Blocker > Fix For: 3.4.10, 3.5.3 > > > As part of the fix for ZOOKEEPER-1797, the call to > FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a > result, some old-looking but required txn log files can be deleted, resulting > in data corruption or loss. > For example, consider the following: > 1. Configuration: > autopurge.snapRetainCount=3 > 2. Following files exist: > log.100 spans transactions from zxid=100 till zxid=140 (inclusive) > snapshot.110 - snapshot as of zxid=110 > snapshot.120 - snapshot as of zxid=120 > snapshot.130 - snapshot as of zxid=130 > Above scenario is possible when snapshotting has happened multiple times but > without accompanying log rollover, which is possible if the server was > running as a learner. > 3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is > older than the zxid of the oldest snapshot (110). This results in loss of > transactions in the range 131-140. > Before the fix for ZOOKEEPER-1797, this was avoided by the call to > FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log > file with starting zxid < oldest retained snapshot's highest zxid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
