Re: [jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-27 Thread Manos Kapritsos
Hi guys, I am working on implementing a Byzantine Fault Tolerant module 
on top of Zookeeper. As part of our work, we run experiments on the 
unmodified version of Zookeeper. We noticed yesterday that the 
PurgeTxnLog does not work properly, and were surprised to find that you 
noticed it at the same time! :) I tried to apply the latest patch that 
Mahadev posted, but it does not work properly. Most probably, I am 
missing something from the development. I applied the patch over version 
3.0.1 and get the following compile errors:


compile-main:
   [javac] Compiling 3 source files to 
/home/manos/zookeeper-3.0.1_orig/build/classes
   [javac] 
/home/manos/zookeeper-3.0.1_orig/src/java/main/org/apache/zookeeper/server/PurgeTxnLog.java:98: 
cannot find symbol

   [javac] symbol  : method getDataDir()
   [javac] location: class 
org.apache.zookeeper.server.persistence.FileTxnSnapLog
   [javac] 
Arrays.asList(txnLog.getDataDir().listFiles(new MyFileFilter(log.;

   [javac] ^
   [javac] 
/home/manos/zookeeper-3.0.1_orig/src/java/main/org/apache/zookeeper/server/PurgeTxnLog.java:100: 
cannot find symbol

   [javac] symbol  : method getSnapDir()
   [javac] location: class 
org.apache.zookeeper.server.persistence.FileTxnSnapLog
   [javac] 
files.addAll(Arrays.asList(txnLog.getSnapDir().listFiles(new 
MyFileFilter(snapshot.;

   [javac]  ^
   [javac] 2 errors

I checked and the txnLog (a FileTxnSnapLog object) does not have a 
getDataDir() method. The only getDataDir method is in ServerConfig, 
returning the static instance.dataDir string.
The patch refers to revision 737951, so I guess these methods are 
already there in that revision. Is there any patch available that can be 
applied to 3.0.1 directly? If not, I can try and use the dataDir 
directly (but I guess there was a reason why it was changed).


Thank you,
Manos

Mahadev konar (JIRA) wrote:

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Attachment: ZOOKEEPER-252.patch

had a bug in the last patch.

  

PurgeTxnLog is not handling the new dataDir directory structure
---

Key: ZOOKEEPER-252
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
Project: Zookeeper
 Issue Type: Bug
 Components: server
   Affects Versions: 3.0.0, 3.0.1
   Reporter: Patrick Hunt
   Assignee: Mahadev konar
   Priority: Critical
Fix For: 3.1.0

Attachments: ZOOKEEPER-252.patch


org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle the new 
directory structure imposed by the upgrade from v2 to v3 of ZooKeeper. In particular the 
dataDir now has a version-2 subdirectory that stores all of the 
snaps/transactionallogs for version2 of the persistence layer.
I also note that the documentation of this class is particularly poor. I'm 
working on ZOOKEEPER-229 and would like to point to the API docs for this class 
regarding usage but they api docs are nonexistent
Also - I think it's important for the user to be able to specify the number of backup 
snaps and logs that should be kept -- right now it seems we delete all but the current 
snaps/txlogs. Either by count or by date -- ie remove anything 5 days or older, with a minum 
of 3 most recents snaps (and accompanying txlogs) seems like a pretty common user case 
(assuming the operator is doing system backups every X days, etc...)
in general this class needs some tlc - the formatting should also be cleaned up.
Also - the API docs for this and LogFormatter are not included in the build.xml 
javadoc target. These are user utilities so javadoc for these two classes 
should be included. I will fix this issue as part of ZOOKEEPER-229. I'm also updateing 
the forrest documention in 229 so don't worry about that either.



  




Re: [jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-27 Thread Mahadev Konar
Hi Manos,

 I think I might have added the getDataDir and getSNapdir later than 3.0.1.
So the patch would only apply to the trunk. Also the getdatadir is different
than the datadir that's used in purgetxnlog. Its a versioned directory
inside datadir (datadir/version-2 more specifically).


mahadev



On 1/27/09 12:36 PM, Manos Kapritsos ma...@cs.utexas.edu wrote:

 Hi guys, I am working on implementing a Byzantine Fault Tolerant module
 on top of Zookeeper. As part of our work, we run experiments on the
 unmodified version of Zookeeper. We noticed yesterday that the
 PurgeTxnLog does not work properly, and were surprised to find that you
 noticed it at the same time! :) I tried to apply the latest patch that
 Mahadev posted, but it does not work properly. Most probably, I am
 missing something from the development. I applied the patch over version
 3.0.1 and get the following compile errors:
 
 compile-main:
 [javac] Compiling 3 source files to
 /home/manos/zookeeper-3.0.1_orig/build/classes
 [javac] 
 /home/manos/zookeeper-3.0.1_orig/src/java/main/org/apache/zookeeper/server/Pur
 geTxnLog.java:98:
 cannot find symbol
 [javac] symbol  : method getDataDir()
 [javac] location: class
 org.apache.zookeeper.server.persistence.FileTxnSnapLog
 [javac]  
 Arrays.asList(txnLog.getDataDir().listFiles(new MyFileFilter(log.;
 [javac] ^
 [javac] 
 /home/manos/zookeeper-3.0.1_orig/src/java/main/org/apache/zookeeper/server/Pur
 geTxnLog.java:100:
 cannot find symbol
 [javac] symbol  : method getSnapDir()
 [javac] location: class
 org.apache.zookeeper.server.persistence.FileTxnSnapLog
 [javac]  
 files.addAll(Arrays.asList(txnLog.getSnapDir().listFiles(new
 MyFileFilter(snapshot.;
 [javac]  ^
 [javac] 2 errors
 
 I checked and the txnLog (a FileTxnSnapLog object) does not have a
 getDataDir() method. The only getDataDir method is in ServerConfig,
 returning the static instance.dataDir string.
 The patch refers to revision 737951, so I guess these methods are
 already there in that revision. Is there any patch available that can be
 applied to 3.0.1 directly? If not, I can try and use the dataDir
 directly (but I guess there was a reason why it was changed).
 
 Thank you,
 Manos
 
 Mahadev konar (JIRA) wrote:
  [ 
 https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.p
 lugin.system.issuetabpanels:all-tabpanel ]
 
 Mahadev konar updated ZOOKEEPER-252:
 
 
 Attachment: ZOOKEEPER-252.patch
 
 had a bug in the last patch.
 
   
 PurgeTxnLog is not handling the new dataDir directory structure
 ---
 
 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.0, 3.0.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0
 
 Attachments: ZOOKEEPER-252.patch
 
 
 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle
 the new directory structure imposed by the upgrade from v2 to v3 of
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that
 stores all of the snaps/transactionallogs for version2 of the persistence
 layer.
 I also note that the documentation of this class is particularly poor. I'm
 working on ZOOKEEPER-229 and would like to point to the API docs for this
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number
 of backup snaps and logs that should be kept -- right now it seems we
 delete all but the current snaps/txlogs. Either by count or by date -- ie
 remove anything 5 days or older, with a minum of 3 most recents snaps (and
 accompanying txlogs) seems like a pretty common user case (assuming the
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned
 up.
 Also - the API docs for this and LogFormatter are not included in the
 build.xml javadoc target. These are user utilities so javadoc for these
 two classes should be included. I will fix this issue as part of
 ZOOKEEPER-229. I'm also updateing the forrest documention in 229 so don't
 worry about that either.
 
 
   
 



[jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-27 Thread Mahadev konar (JIRA)

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Attachment: ZOOKEEPER-252.patch

this patch adds a test and checks to see we have exactly a given number of 
snapshots in the dir... 


 PurgeTxnLog is not handling the new dataDir directory structure
 ---

 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.0, 3.0.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0

 Attachments: ZOOKEEPER-252.patch, ZOOKEEPER-252.patch


 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle 
 the new directory structure imposed by the upgrade from v2 to v3 of 
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that 
 stores all of the snaps/transactionallogs for version2 of the persistence 
 layer.
 I also note that the documentation of this class is particularly poor. I'm 
 working on ZOOKEEPER-229 and would like to point to the API docs for this 
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number 
 of backup snaps and logs that should be kept -- right now it seems we 
 delete all but the current snaps/txlogs. Either by count or by date -- ie 
 remove anything 5 days or older, with a minum of 3 most recents snaps (and 
 accompanying txlogs) seems like a pretty common user case (assuming the 
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned 
 up.
 Also - the API docs for this and LogFormatter are not included in the 
 build.xml javadoc target. These are user utilities so javadoc for these two 
 classes should be included. I will fix this issue as part of ZOOKEEPER-229. 
 I'm also updateing the forrest documention in 229 so don't worry about that 
 either.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-27 Thread Mahadev konar (JIRA)

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Status: Patch Available  (was: Open)

 PurgeTxnLog is not handling the new dataDir directory structure
 ---

 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.1, 3.0.0
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0

 Attachments: ZOOKEEPER-252.patch, ZOOKEEPER-252.patch, 
 ZOOKEEPER-252.patch


 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle 
 the new directory structure imposed by the upgrade from v2 to v3 of 
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that 
 stores all of the snaps/transactionallogs for version2 of the persistence 
 layer.
 I also note that the documentation of this class is particularly poor. I'm 
 working on ZOOKEEPER-229 and would like to point to the API docs for this 
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number 
 of backup snaps and logs that should be kept -- right now it seems we 
 delete all but the current snaps/txlogs. Either by count or by date -- ie 
 remove anything 5 days or older, with a minum of 3 most recents snaps (and 
 accompanying txlogs) seems like a pretty common user case (assuming the 
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned 
 up.
 Also - the API docs for this and LogFormatter are not included in the 
 build.xml javadoc target. These are user utilities so javadoc for these two 
 classes should be included. I will fix this issue as part of ZOOKEEPER-229. 
 I'm also updateing the forrest documention in 229 so don't worry about that 
 either.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-27 Thread Mahadev konar (JIRA)

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Attachment: ZOOKEEPER-252.patch

fixed a minor bug.

 PurgeTxnLog is not handling the new dataDir directory structure
 ---

 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.0, 3.0.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0

 Attachments: ZOOKEEPER-252.patch, ZOOKEEPER-252.patch, 
 ZOOKEEPER-252.patch


 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle 
 the new directory structure imposed by the upgrade from v2 to v3 of 
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that 
 stores all of the snaps/transactionallogs for version2 of the persistence 
 layer.
 I also note that the documentation of this class is particularly poor. I'm 
 working on ZOOKEEPER-229 and would like to point to the API docs for this 
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number 
 of backup snaps and logs that should be kept -- right now it seems we 
 delete all but the current snaps/txlogs. Either by count or by date -- ie 
 remove anything 5 days or older, with a minum of 3 most recents snaps (and 
 accompanying txlogs) seems like a pretty common user case (assuming the 
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned 
 up.
 Also - the API docs for this and LogFormatter are not included in the 
 build.xml javadoc target. These are user utilities so javadoc for these two 
 classes should be included. I will fix this issue as part of ZOOKEEPER-229. 
 I'm also updateing the forrest documention in 229 so don't worry about that 
 either.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-27 Thread Mahadev konar (JIRA)

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Attachment: ZOOKEEPER-252.patch

missed the test.

 PurgeTxnLog is not handling the new dataDir directory structure
 ---

 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.0, 3.0.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0

 Attachments: ZOOKEEPER-252.patch, ZOOKEEPER-252.patch, 
 ZOOKEEPER-252.patch, ZOOKEEPER-252.patch


 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle 
 the new directory structure imposed by the upgrade from v2 to v3 of 
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that 
 stores all of the snaps/transactionallogs for version2 of the persistence 
 layer.
 I also note that the documentation of this class is particularly poor. I'm 
 working on ZOOKEEPER-229 and would like to point to the API docs for this 
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number 
 of backup snaps and logs that should be kept -- right now it seems we 
 delete all but the current snaps/txlogs. Either by count or by date -- ie 
 remove anything 5 days or older, with a minum of 3 most recents snaps (and 
 accompanying txlogs) seems like a pretty common user case (assuming the 
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned 
 up.
 Also - the API docs for this and LogFormatter are not included in the 
 build.xml javadoc target. These are user utilities so javadoc for these two 
 classes should be included. I will fix this issue as part of ZOOKEEPER-229. 
 I'm also updateing the forrest documention in 229 so don't worry about that 
 either.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-26 Thread Mahadev konar (JIRA)

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Attachment: ZOOKEEPER-252.patch

this patch fixes the issue.
It allows purgetxn to be called with -n num (which is to keep history for num 
snapshots).


 PurgeTxnLog is not handling the new dataDir directory structure
 ---

 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.0, 3.0.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0

 Attachments: ZOOKEEPER-252.patch


 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle 
 the new directory structure imposed by the upgrade from v2 to v3 of 
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that 
 stores all of the snaps/transactionallogs for version2 of the persistence 
 layer.
 I also note that the documentation of this class is particularly poor. I'm 
 working on ZOOKEEPER-229 and would like to point to the API docs for this 
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number 
 of backup snaps and logs that should be kept -- right now it seems we 
 delete all but the current snaps/txlogs. Either by count or by date -- ie 
 remove anything 5 days or older, with a minum of 3 most recents snaps (and 
 accompanying txlogs) seems like a pretty common user case (assuming the 
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned 
 up.
 Also - the API docs for this and LogFormatter are not included in the 
 build.xml javadoc target. These are user utilities so javadoc for these two 
 classes should be included. I will fix this issue as part of ZOOKEEPER-229. 
 I'm also updateing the forrest documention in 229 so don't worry about that 
 either.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-26 Thread Mahadev konar (JIRA)

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Attachment: (was: ZOOKEEPER-252.patch)

 PurgeTxnLog is not handling the new dataDir directory structure
 ---

 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.0, 3.0.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0


 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle 
 the new directory structure imposed by the upgrade from v2 to v3 of 
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that 
 stores all of the snaps/transactionallogs for version2 of the persistence 
 layer.
 I also note that the documentation of this class is particularly poor. I'm 
 working on ZOOKEEPER-229 and would like to point to the API docs for this 
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number 
 of backup snaps and logs that should be kept -- right now it seems we 
 delete all but the current snaps/txlogs. Either by count or by date -- ie 
 remove anything 5 days or older, with a minum of 3 most recents snaps (and 
 accompanying txlogs) seems like a pretty common user case (assuming the 
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned 
 up.
 Also - the API docs for this and LogFormatter are not included in the 
 build.xml javadoc target. These are user utilities so javadoc for these two 
 classes should be included. I will fix this issue as part of ZOOKEEPER-229. 
 I'm also updateing the forrest documention in 229 so don't worry about that 
 either.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (ZOOKEEPER-252) PurgeTxnLog is not handling the new dataDir directory structure

2009-01-26 Thread Mahadev konar (JIRA)

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-252:


Attachment: ZOOKEEPER-252.patch

had a bug in the last patch.

 PurgeTxnLog is not handling the new dataDir directory structure
 ---

 Key: ZOOKEEPER-252
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-252
 Project: Zookeeper
  Issue Type: Bug
  Components: server
Affects Versions: 3.0.0, 3.0.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Critical
 Fix For: 3.1.0

 Attachments: ZOOKEEPER-252.patch


 org.apache.zookeeper.server.PurgeTxnLog class has not been updated to handle 
 the new directory structure imposed by the upgrade from v2 to v3 of 
 ZooKeeper. In particular the dataDir now has a version-2 subdirectory that 
 stores all of the snaps/transactionallogs for version2 of the persistence 
 layer.
 I also note that the documentation of this class is particularly poor. I'm 
 working on ZOOKEEPER-229 and would like to point to the API docs for this 
 class regarding usage but they api docs are nonexistent
 Also - I think it's important for the user to be able to specify the number 
 of backup snaps and logs that should be kept -- right now it seems we 
 delete all but the current snaps/txlogs. Either by count or by date -- ie 
 remove anything 5 days or older, with a minum of 3 most recents snaps (and 
 accompanying txlogs) seems like a pretty common user case (assuming the 
 operator is doing system backups every X days, etc...)
 in general this class needs some tlc - the formatting should also be cleaned 
 up.
 Also - the API docs for this and LogFormatter are not included in the 
 build.xml javadoc target. These are user utilities so javadoc for these two 
 classes should be included. I will fix this issue as part of ZOOKEEPER-229. 
 I'm also updateing the forrest documention in 229 so don't worry about that 
 either.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.